Although my own weekly podcast appears on the ITConversations channel of the Gigavox network, lately I find myself listening more often to our sister channel, Social Innovation Conversations. And I’ve started to wonder: Why are these two different channels, for two different audiences? Increasingly I wish I could mash them together. Dean Kamen’s recent appearance on Tim Zak’s Globeshakers series on SIConversations gives me a sense of what that would be like. Here his pitch:

Given the enormous rate at which technology is moving forward, almost all the ‘Can this be done?’ questions have essentially been answered by ‘Yes.’ The much tougher question right now isn’t ‘What can we do with technology?’ — it’s ‘What should we do with technology?’ That’s a much harder question involving practical issues, moral issues…the haves and the have-nots, in technology, education, and health care, are diverging.

People who can develop new technologies ought to start thinking, more than they have in the last few decades, about where it’s appropriate to deploy the energy and passion to develop the next level of technology. There are just so many video games that we need, and just so many luxury leisure-time products that we need.

If societies start to recognize that we really do get what we celebrate, and we start celebrating the right things, we’ll see a much more effective use of our available technologies and a much more appropriate and focused set of developments of our future technologies. Instead of focusing on what we can do with technology, we should focus on how to be responsible to each other, to the environment, to the future of this delicate little planet.

I’ve transcribed that quote here for two reasons. First, because I know that relatively few people have the time or inclination to listen to as many podcasts as I lately find myself wanting to do. Second, because I know that people who self-identify as technogeeks are more likely to subscribe to ITConversations than to SIConversations.

As I write this entry I’m enroute from one technogeek paradise, the Microsoft campus, to another, the O’Reilly campus. In doing so I’m crossing a bridge between two cultures that are, in some ways, very different. On the Microsoft campus, for example, Windows laptops are ubiquitous and Macintosh laptops are scarce. On the O’Reilly campus it’s the reverse.

In other ways these cultures are very alike. In both places, you’ll routinely see people whizzing around on the invention for which Dean Kamen is best known: the Segway. Geeks of all persuasions are early adopters and everyday users of this machine which, for most people, remains an exotic curiosity.

And yet, there wasn’t a single mention of the Segway in Tim Zak’s 45-minute interview with Dean Kamen. What’s top of mind, for Kamen, is US FIRST — the acronym expands to For Inspiration and Recognition of Science and Technology. The goal is to reframe the idea of success which, he says, too many teenagers define unrealistically in terms of sports and entertainment. He wants them to know that success in science and engineering is, for the vast majority, both more achievable and more socially productive. A Wired article in 2000 called the idea far-fetched, but FIRST’s robotics competitions have grown steadily since 1992, and in 2007, Kamen says, the final event packed 70,000 people into Atlanta’s Georgia Dome.

A key ingredient of the program is the mentoring that’s provided by scientists and engineers on loan from corporate sponsors:

These kids really weren’t building robots. They were building relationships with serious adults, they were building an understanding of what’s possible if you put your energy and passion to things that matter.

I’d love to see a mashup of ITConversations and SIConversations that would produce more shows like that.

It’s been an interesting couple of weeks for folks who care about RESTful web services. Dare Obasanjo kicked things off with a couple of items about the Atom Publishing Protocol (APP) and Google’s use of it for its GData project. Tim Bray bristled at Dare’s characterization of APP, and it looked like we were headed for another summer syndication flamefest. (Why do those always happen in June?) When the Web3S protocol — not the Atom Publishing Protocol — was revealed as the proposed mechanism for granting read/write access to the half-billion Hotmail contacts, Messenger buddies, and Spaces friends that comprise the Live Contacts database, I was sure it’d turn into a flamefest.

But it didn’t. And now that things have settled down a bit, I’d like to note two points that may interest the majority of folks who don’t follow the saga of RESTful web services.

First, there’s the role of the blogosphere. I’ve often talked about how the interplay of voices in the technical blogosophere models a style of professional collaboration that I expect will someday prevail more broadly. We see that happening here. Sam Ruby usefully asks whether another protocol proposed by Microsoft, SSE, might play a role in contact synchronization. Tim Bray usefully analyzes the Web3S spec and offers some excellent advice, in particular:

Get yourself a test suite! APP has already been helped by the existence of code from Joe Gregorio, me, and others. Test suites matter way more than specs, in the big picture.

Along with the technical back-and-forth you can see a social process at work as the always-tricky interplay between competition and cooperation gets sorted out. APP folks: “If you had concerns about APP’s capabilities, why didn’t you voice them sooner?” Microsoft folks: “Well, we were worried about seeming to interfere, but yeah, in retrospect, we should have.” Although some of us have started to take this interplay for granted, it’s still quite novel for most people, and it’s a remarkable thing.

The second point is that, technical back-and-forth notwithstanding, the purpose of Web3S is to open up walled-garden social networks. That’s been another — and more broadly inclusive — conversation in the blogosphere recently. Facebook’s ignorance of web reputation is part of the story. Here’s another, the Facebook friend finder:

In order to find people you may know on one or another of the popular webmail systems, you’re invited to lend Facebook your credentials so it can probe your address book on one of those systems. I understand why this happens, but it’s totally the wrong message about security and digital identity to be sending to a large community of young people.

From that perspective, Web3S is just a small part of a big story: opening up Live Contacts so there’s no need for this kind of impersonation. In his talk at MIX¹ (MP3 here²), Yaron Goland lays out two scenarios. For one-off exchanges, there’s the contacts control which a third-party site can embed in one of its pages so people can selectively relay Live Contacts data into the page. For longer-term relationships with trusted services, people can grant the permission to read and update their Live Contacts directly, so that social activities elsewhere will be reflected in their own address books.

In both scenarios, you retain control. You never allow a service to use your name and password to impersonate you to another service. That’s a good thing for Facebook, which would rather not have to impersonate people. It’s a good thing for people who (whether they realize it or not) don’t want to be impersonated. It’s a good thing for Microsoft because the enabling platform services will become a business. It’s a good thing for RESTful web services³ because the API is RESTful. And it’s a good thing for everyone who believes that ultimately the Internet is our social network.

---

¹ The RESTful instinct isn’t yet fully developed. Although the API discussed in the talk is RESTful, I had to extricate this URL from the MIX RSS feed because the navigational apparatus at http://sessions.visitmix.com/ doesn’t disclose it on the URL-line or in a permalink. Note to team: Let’s please make it easier for people to point to these things.

² I had to extract the soundtrack from the video and then republish it. Note to team: Let’s please make it easier for people to hear these things.

³ I’m wondering, though, like Tim Bray, about the introduction of a new HTTP verb. The session blurb was: “Data wants to be free! So come to this technical deep dive to learn how you can POST/GET/PUT/DELETE your way into Windows Live.” There was no mention of UPDATE. Of course the spec was published in order to solicit feedback, so I’ll be interested to see what comes of that.

The audio accompaniment for yesterday’s exercise hour was Tom Raftery’s interview with Brad Abrams, group program manager for Silverlight. I mention it for three reasons.

First, it’s a nice comprehensive overview of the history and mission of the Silverlight project. Now that the flurry of MIX announcements is over, this is a good time to step back and reflect on the big picture. As someone who’s been working on the .NET Common Language Runtime since its inception, Brad’s in a good position to paint that picture.

Second, it reminds me of an obvious strategy for podcasts that I’ve somehow managed to ignore: solicit questions ahead of time! Tom Raftery does that routinely. In this case people asked a bunch of great questions, Brad Abrams engaged straightforwardly with them, and the resulting show was much richer and deeper than it otherwise would have been. Given that I was an avid practitioner of this method in my journalism days, it’s crazy that I haven’t carried it forward into my podcasting. Gotta fix that.

Third, one particular segment of the interview really grabbed me. Referring to his talk at MIX (WMV, MP4), Brad discusses a strategy for exposing videos to search engines. The ingredients of the strategy are:

1. A feature of the ASP.NET “Futures” release — DynamicDataSearchSiteMapProvider — that helps developers dynamically generate sitemaps that provide the breadcrumb trails otherwise unavailable to search engines when they visit dynamically-generated sites.
2. An data source from which the sitemap provider can extract titles and timecodes for chapters within a video.
3. A SMIL wrapper that provides closed captioning both to the video and, indirectly, to the web pages that the sitemap points crawlers to.
4. A streaming server.

As an industry we’ve gone back and forth on that last point. In the beginning there was Real which primarily relied on streaming servers rather than standard web servers. The downside was that these were specialized and non-ubiquitous. One of the upsides was that they enabled random access. But then, hardly anybody took advantage of that opportunity. As you can see here, although it’s quite feasible to form URLs that point into Real streams, the details are just geeky enough to deter almost everyone.

Then things shifted. Increasingly the media encoders and players conspired to support progressive downloading. In this mode, you only need a standard webserver, serving up static files. The encoders tuck enough extra information into the files so that players can begin playing right away, after only a short buffering delay. It looks like streaming to most people, and a lot of applications and services even call it streaming rather than progressive downloading.

The upside here was that no specialized servers were needed. Any regular webserver would work, so this approach is very blog-friendly. Got an audio or video file? Just upload it to your blog, and bingo, you’re podcasting or videoblogging.

This radically democratized media publishing, and continues to do so. But, although few recognized the tradeoff, there was one. You couldn’t randomly access a static media file.

Or so most of us thought.

As it turned out, that’s not strictly true, at least not for MP3 files. I realized that some players were able to randomly access parts of statically-served MP3 files, found out how, and prototyped a gateway that would enable anyone to form a URL to a timecoded segment from an MP3 file hosted on a remote webserver.

This was an interesting result, but it was even clunkier than the methods already supported by the Real servers and players — and as we’ve seen, hardly anybody ever discovered or used their random-access features. What’s more, my method only worked for MP3 files by virtue of a special property of that format: frames are (usually) independent of one another, so you can reach blindly into the middle of a file, shove bytes at a player, and expect it to find the next frame boundary and start playing. I’m mostly ignorant of the details of video formats but, so far as I can tell, they don’t tend to work that way.

Now I wonder if we’re heading back to the future. Flash (with FLV) and Silverlight (with WMV) don’t require streaming servers on the back end, they can do progressive downloading as well. But in the services era, you’re less likely to worry about deploying your own streaming server and more likely to use an instance of one that runs in the cloud. That instance can react to requests for timecoded segments in a more intelligent way than by seeking to byte offsets.

It’s true that we failed, the first time around, to make the formation of those requests easy and obvious to people using media players. But a new generation of players — again, both Flash-based and now Silverlight-based — can be friendlier to that kind of innovation.

An example of what we should expect appears at 59:50 in Brad Abrams’ MIX talk (WMV, MP4). You search, find some title or caption text (thanks to a sitemap), click the link, and begin playing a segment at a timecode.

The hardest part, of course, is the data preparation. On my trip to the UK in January I mentioned the Open University’s FlashMeeting system which does a great job of segmenting captured video on the fly, then making it randomly accessible.

There are already too many triple-S acronyms so I probably shouldn’t do this, but the formula I’m looking for is: Sitemaps + Segmentation + Streaming.

While I’m back on the topic of screencasting, I’ve been meaning to mention another important use of the medium. Recently a colleague reported severe trouble trying to present demos that rely on a live connection to the Internet. My solution is a variation of the old joke:

Patient: It hurts when I do that.

Doctor: Don’t do that!

To avoid the pain I use screencasts instead of live demos. There are a variety of reasons for doing so. An obvious one is that it makes you immune to network glitches.

A subtler reason is that it’s hard to show software in use without wasting effort and motion. You reach for the wrong menu item, you fumble while typing. These are perfectly normal and natural behaviors, but they only add dead time to your presentation and therefore, by definition, they detract from it. When you edit out the wasted motion and false starts you create an effect that isn’t quite real — it’s hyperreal — but that’s exactly the effect that you want (or anyway, I want) a presentation to achieve.

Another subtler reason is that video playback gives you more control over timing. It can be hard or even impossible to replay a piece of a demo in response to an audience question. Likewise, it can be hard or impossible to fast-forward a demo if you’re running short on time, or if you’re losing the audience. When you’ve canned your demos as screencasts, you have a lot more flexibility.

Finally, there’s just the peace of mind that comes with only having to keep track of one single media file, as opposed to lots of moving parts. When you are speaking and showing demos, the fewer moving parts, the better.

I’ll be in Redmond for part of next week and, although my first effort to have a Crossroads Mall meetup got sidetracked, I’d like to try again. To keep things simple: I’ll just be there this Thursday night, the 21st, at around 6:30, near the bookstore. (I look kind of like this but grayer.) At some point I’ll gravitate toward the noodle soup.

As part of my re-exploration of the walled-garden social networks, I’ve accepted the entire batch of LinkedIn invitations that had queued up in my dormant account. One of them was a request from Beth Kanter for advice on screencasting. From my point of view, LinkedIn was superfluous in this case because the same request had already been made (implicitly) in this blog post in which Beth summarizes what she had figured out for herself, and then invites feedback.

Although we should probably not yet simply assume that linking to a blog post will draw the attention of the author of that post, the blogosphere does in fact propagate awareness in that way, and does so with remarkable speed and reliability. So in this case I’d seen Beth’s item before I began receiving requests from her via LinkedIn intermediaries. Because I was boycotting walled-garden social networks at the time, I thought this was a good opportunity to show how, in a case like this, the open Net can obviate the need for a closed network. So I replied to Beth’s blog item in a comment. Or rather, I thought I did. But although I wrote the reply it seems I never managed to post it. Oops. I’m sorry about that, Beth, and I’ll try to make up for it here.

But first, I want to note that your item is a textbook example of how to construct an online query for information. By summarizing what you’ve already learned, you’re helping bring other folks up to speed. At the same time, you’re helping me understand where and how I can add value. This custom is just good common sense, of course, but one that’s honored more often in breach than in the observance. If I were teaching this kind of thing in grade school, I’d use del.icio.us to keep lists of good examples of netiquette, and I’d put this example on one of those lists.

Now, in the context of the genre that I’ve called the conversational demo, here are your questions and my answers.

Q: How much scripting does he do prior to the interview?

A: None.

Q: Does he “rehearse” with his guest?

A: No. I do, of course, choose topics in which I’m interested, and to which I bring plenty of domain knowledge.

Q: Or does he capture everything and edit?

A: Yes. As with my podcasts, I lean heavily on editing when making these conversational screencasts. The editing happens on two levels: macro and micro. On the macro level, because we (interviewer and interviewee) know that whole scenes can be cut, we don’t need to worry about the performance. If something doesn’t work we can just call it a bad take and try again. We can also plan, on the fly, where to go next, again knowing that such discussion is effectively out of band and will be deleted.

On the micro level, there’s internal editing. The term comes from the audio domain and it applies in the same way here. If I can eliminate ums and you knows and false starts without compromising the video, I do.

Q: What tools does he use to capture these interviews?

A: For video I mostly use Camtasia in conjunction with one or another of the many screen-sharing tools. Since most software demos don’t require a high frame rate and don’t push lots of screen bits, that’s usually OK. However in this screencast about tagging in Photo Gallery — which, by the way, was edited down from 35 minutes to 14 minutes — the screen-sharing setup couldn’t keep pace with all the images. So I had Scott Dart record locally using Windows Media Encoder, and then ship me the resulting WMV file. I was able to follow along in screen-sharing well enough to carry on the conversation, even though what was displayed on my screen would have been useless for production. This is a variation of a technique that’s really useful for podcasters who are struggling with expensive and/or poor-quality phone lines. If you can get both parties to record high-quality audio locally, you can use a marginal VoIP setup to converse and then join up the high-quality audio later. I did that here and I’d love to do more shows that way.

For audio I also mostly use Camtasia. Originally I didn’t, I used Audacity, because I hadn’t figured out how to get Camtasia to record the two channels (caller, callee) from my Telos as a stereo track. Eventually I found that setting. It’s in Tools Options -> Streams -> Audio Setup.

Because the conversational screencast is a superset of a podcast, you’re dealing with all of the same audio production issues as in podcasting. For me, working remotely, that’s been an ongoing challenge. Telephone recording is just plain hard. Although I’ve been using a Telos for a while, for example, I only recently discovered that I’d been using it incorrectly. On the other hand, VoIP recording is hard too.

Granted, I wasn’t born with an audio chromosome, but then neither were most folks. So, remote audio is going to be a problem for most of us — a problem that, I reckon, somebody is going to make money by solving. At this point I can muddle through fairly well. But if I hadn’t already invested in the Telos I’d be looking really hard at the technique of recording locally on both ends and then joining the results in post-production. It’s not particularly hard to do that, and it’s really nice to simply abolish all the problems associated with the voice channel — whether it’s the public telephone system or the Internet.

Because it rarely applies to me, I haven’t mentioned the scenario in which both parties are together in the same place looking at the same computer. In that case I’d use whatever capture software was convenient. If the software were on the interviewee’s computer, I’d ask the interviewee to install the free Windows Media Encoder and capture video that way. And I’d probably use a standalone digital audio recorder with a handheld microphone to separately capture audio.

One final point from my recent conversation with Doug Kaye: a lot of people who think they don’t have digital audio recorders overlook the fact that they have camcorders which can perform that function. A related point: if you use a camcorder, it’s tempting to let it do the whole job — that is, screen capture as well as audio capture. Although my Channel 9 colleagues do that all the time, I don’t recommend it. You’d much rather use perfect screen capture than fuzzy camcorder capture. And ideally you’d like to be able to do that without installing any software on the target computer, using a direct-capture device. I’ve never seen one of those, but next week in Redmond I’ll be visiting our new production studio where I’m told we have such a beast. I’m curious to see it in action.

Q: Does he edit in Camtasia?

A: Yes, I do. I’d honestly rather edit in iMovie instead, because I find it to be more elegant and more capable, but it’s a huge hassle to get stuff in and out of iMovie so I usually take the path of least resistance and edit in Camtasia. If you want to do micro-edits in Camtasia, one important tip is to record at a higher frame rate than you will ultimately produce. A screencast is legible at 5 or even fewer frames per second. But if you only capture at that rate, you’ll find that you can’t make intra-frame audio micro-edits. So record at 15 or more frames per second, then produce at a lower rate.

Q: What are some best practices in terms of production and editing?

A: It’s tempting to jump in and start editing right away, and to be honest I often do. But I think it’s better to just watch the raw recording all the way through, setting markers along the way to annotate the segments that you want to include, discard, or perhaps rearrange. Ultimately you’re trying to tell a story, and those markers will help you visualize the outline of the story.

Sorry this took so long, Beth. I hope it helps.

This week’s show on ITConversations explores what Jeannette Wing means by computational thinking. As I noted here, she has coined that evocative phrase to suggest how the intellectual tools of computer science — including abstraction, naming, composition, state machines, refactoring, and separation of concerns — can add up to “a universally applicable attitude and skill set that everyone, not just computer scientists, would be eager to learn and use.”

At Carnegie Mellon, where Dr. Wing is head of the computer science department, this way of thinking pervades many other academic disciplines. But in her view, it’s really as fundamental as reading, writing, and arithmetic, and like those skills it should be taught in grade school. Since that’s not likely to happen anytime soon, I wonder if computer games — which already teach kids certain aspects of computational thinking — could help advance this agenda in a more deliberate way.

Like other social applications, Facebook wants to know how you’re connected to people. So it asks: “How do you know this person?” and presents these choices:

The choice I usually want — “Through the Web” — isn’t available. One friend coerced “Met randomly” by adding “The web as a conversation engine” — but that’s an unsatisfactory workaround. There was nothing random about how we met. Given our shared interests and our online expression of them, it was inevitable that we would come into contact.

“Through the Web” should be a first-class answer for “How do you know this person?”

In preparation for a panel at the MIT Enterprise Forum I summarized my thoughts about walled-garden social networks. It came back around the other day in conversation with Avi Bryant, who — like a great many people — is appreciating the refinements Facebook brings to various protocols that work less smoothly (if at all) on today’s open Net. He reflected our exchange into a blog post where he also asks: “How do we Facebookize the open Web?”

As Avi notes in his posting, I think Gary McGraw is right when he says: “People keep asking me to join the LinkedIn network, but I’m already part of a network, it’s called the Internet.” It’s also obviously true that the walled gardens are petri dishes growing cultures that will merge with or (some believe) recreate the open Web. So although I don’t spend a lot of time in the walled gardens, I do visit them in order to learn about the cultures growing there.

Last night, I noticed that I’m one of four 1974 graduates of my high school who appear in Facebook. That’s fewer than 1% of the 600 grads. This year, 302 grads appear in Facebook. That’s 50% of the graduating class. Here’s a nice picture of the bimodal distribution over time:

In 2012, what percentage of that year’s class do we think will be participating in Facebook?

While we ponder our answers, I’d like to digress for a moment on the data supporting the chart, and on the chart itself.

Although I thought there were about 600 students in my graduating class, I couldn’t remember for sure. And I had no idea how the student population might have changed over the years. How can you check these facts? Here’s the Google query I ran:

“cheltenham high school” “student population” 100..5000

The third term exploits a powerful but little-known and rarely-used feature called numeric range search. In this case I picked a lower limit that I was sure was smaller than the population of any one grade, and an upper limit that I was sure was higher than the population of the whole school. The query finds pages with both of the exact phrases (the quotation of which is another underutilized feature of search) plus a number in the given range. From the result set, I culled two documents.

First was the curriculum vitae of Joseph Cifelli, which confirmed that in my era “the high school student population was divided into three houses of about 600 students each.” That’s 1800 total, and there were three grades — 10, 11, 12 — so 600 per grade sounds right.

Second was a page on a real estate information site which, though frustratingly undated, I presume to be reasonably current. It gives the student population as 1706. That’s close enough to 1800 for my purposes.

The chart itself is the first I’ve attempted using Excel 2007. The process feels very different from earlier versions, but I’m no expert in this area so I can’t make a detailed comparison. I do have a question for Excel wizards, though. In following Edward Tufte’s recommendation to subtract ink wherever possible, I was able to get Excel to remove almost all of the unnecessary cruft: grid lines, tick marks. But there were still a bunch of zeros that were unnecessarily reported out on the long tail of the data labels. I wound up taking a picture of the chart and then using Paint to remove those, in order to achieve what I think is an admirably clean and spare result. Is there a way to get Excel to do that directly?

Anyway, back to our question: In 2011 will Facebook’s penetration among CHS grads approach totality? Beats me. That’ll depend on the Facebookization of the Web, but also on the Webification of Facebook.

Avi’s post has examples of the former:

• “a smart feed, which aggregates and filters all of my subscriptions in a holistic way”
• “an API which allows access both to your blogroll data and to your smart feed…so that if I post and tag a photo of A on my blog, anyone subscribed to A’s blog – even if they have no idea who I am and aren’t subscribed to me – will get an item in their feed about it.”

Here’s an example of the latter. Facebook invites me to manage streams of photos and events. But I have other ways to manage streams of photos (e.g., Flickr) and events (e.g., Eventful). Why not enable me to hook into them?

The process of diffusion can flow in both directions, I guess, and I hope that it will.

I recently spent a long day waiting for and visiting with someone who had surgery at the Exeter Hospital in Exeter, NH. It’s a wonderful facility that’s got all sorts of things right: pleasant decor, free valet parking, an excellent and inexpensive cafeteria. But for me, it was the public WiFi that made my day. Everywhere I flipped open my laptop — in a physician’s office, in a surgical waiting room, in a patient’s room — there was always a strong signal, and it always Just Worked.

If any of the IT staff at Exeter Hospital are within earshot of this blog: Thanks! That made a huge difference for me. Twelve hours of disconnectedness would have compounded the stress of being there. Instead I had twelve hours of connectedness, I got a lot done while waiting, and was spared the tyranny of Fox News.

Of course I realize, as you folks do, that providing that experience for me is more than a courtesy. It’s a smart business decision. If I had to choose between hospitals, and if yours were only an hour away instead of two, your robust WiFi setup — as opposed to the always-spotty and now apparently nonexistent setup at my local hospital — would weigh heavily in the decision. I notice that you don’t advertise your WiFi capability. You probably should!

My wish for Channel 9 to augment the trademark videos with downloadable audio files has been granted. As a result, I was able to listen to Charles Torre’s interview with Brian Beckman during my exercise hour yesterday. Even if you’re not a gamer (I’m not), there’s a good chance you’ll enjoy Brian’s lively explanation of the physics that govern the simulation of planes and cars. I guess I’d been vaguely aware that PC flight simulators predated PC racecar simulators by many years, but I’d never thought about why.

To do a credible simulation of a plane, Brian says, you only need to account for two coordinate systems (the earth’s and the plane’s) and four forces (lift, drag, thrust, gravity). Because you can do table look-up for the values of variables such as lift and drag, and because those tables are small, you don’t need a “fast and fat” computer (fast CPU, big memory) to do the job. That’s why Flight Simulator was possible on the earliest PCs.

Since only some of us fly, but most of us drive, you might have expected automotive simulators to have arrived sooner than the mid-1990s. And they would have, were it not the case that car physics is, counter-intuitively, way more complicated than airplane physics:

If you analogize a tire to a wing — that is, the thing that generates the force — you’ve got four of them. And they’re connected to the car by these complicated linkages. Maybe they’re McPherson struts, maybe they’re independent suspensions, maybe they’re leaf springs. All of these things act differently, so each different car is going to have not just different shades of physics, but completely different kinds of physics, completely different equations.

It goes on from there: more coordinate frames to account for, complex dynamics of steering, acceleration, and braking. It’s fascinating to learn why accurate simulation of cars only recently became possible on PCs. Along the way, you’ll be reminded about the properties of the safety envelope that ordinary drivers seldom push but that race drivers always do.

The interview concludes with another counter-intuitive observation. Although the ability to crunch complex nonlinear equations at 30 or 60 frames per second is what makes these simulations possible today, Brian thinks this radically different approach is the way of the future. The Rigs and Rods system shown in that video, created by Pierre-Michel Ricordel, uses a technique that Brian calls “sticks and stones”:

Stones, which have mass, and sticks between them which are little flexible things we can model with very simple physics: harmonic oscillators. That’s all you need, just one physics model, a damped harmonic oscillator connecting pairs of stones, and, by gum, you can simulate really amazing things.
…
The computer is now fast enough to be able to simulate hundreds or thousands of these independent systems simultaneously, every step. We can now dispense with the really hard mathematics. If you’d asked me to bet money that this were even possible, I’d have said no. But that’s because I had this long experience where you just didn’t think about doing thousands of particles iterated in a system like this. Here’s a guy who took a fresh approach, he said let me see what I can do, and sure enough, this is a magnificent system.

We’ve heard it before, we’ll hear it again: a network of many simple parts trumps one big complex monolith. It’s a story that keeps on surprising us, but probably shouldn’t.

So, you might wonder how a video with as much visual content as this one — equation-filled slides, Mathematica screens, YouTube videos — fares in audio format. In this case, surprisingly well. It’s amazing how much information voice alone can carry. Later I did review parts of the video, and having listened I knew just which parts I wanted to see. But this exercise confirmed my belief that downshifting from video to audio is a really useful way to give people access to long-form material they might not have time to watch.

From time to time, Phil Windley and I marvel at the vast reservoirs of tacit knowledge that we bring to the table when working with computers and software. And we ask one another: “What do normal folks do?”

Case in point is this debugging exercise in which Phil discovers, after a round of installation and reconfiguration on his Mac, that “the display was washed out.” Here was his 7-step recovery program:

1. I noticed that the washed out effect didn’t happen until after the desktop picture had been painted.
2. I created a clean user account and logged into it. The problem didn’t appear leading me to believe it was a user problem, not a system problem.
3. I looked at console and system logs to see if there were any error messages. Not much information there on this problem.
4. I removed all the start-up items. Still no joy.
5. I compared the activity monitor processes for the account with the problem to those in the account without the problem. No significant differences.
6. I now suspected something in the preferences. So, I moved the preferences directory to another place. The problem went away confirming that there was something messed up in the preferences.
7. I used a binary process to put preferences back, logging in and out each time until I found the culprit.

The culprit, Phil reports, was the Universal Access area in Preferences, where a contrast slider had been pushed to the wrong setting.

Wow. It’s been more 20 years since my old BYTE pal, Tom Thompson, first showed me that binary search process. Have we made no progress since then?

Actually, we have. In both OS X and Vista, system settings are reasonably well corralled into System Preferences and the Control Panel. What’s more, you can search within those areas. So, as is always true with search, if you know the magic word you can find the right answer. The magic word, in this case, was contrast. Click this image to see a brief screencast of what Phil would have seen if he’d searched System Preferences for that word:

Style points to OS X for the graduated halos around the icons.

Click this image to see a brief screencast of the comparable feature in Vista:

This contextual (and incremental) search is really helpful! It might even be more helpful to those of us who don’t have twenty years of experience doing the binary process of elimination. But because this search feature is relatively new, I suspect that many — both old hands and relative novices alike — have yet to discover it, or develop the habit of using it.

Even so, search depends crucially on the right vocabulary. There’s always that Catch-22: if you could name the thing you’re looking for, you’d already have your answer. What complementary techniques can help?

Yesterday’s item about Koala suggests one. Although that project focuses on recording, playing back, and sharing sequences of actions in the browser, the principle could apply much more broadly. When you’re debugging a system configuration, the key question is ultimately: What changed?

Although Phil (unlike most folks) knew enough to check his logs, they didn’t tell him which system and application settings had recently changed. Logging those changes, and surfacing them in ways that we who make the changes can easily review, would be really helpful.

But we’re in the Internet era now, and our desktop systems are connected. Why not exploit that? Imagine a service that trickles the changes you make to your system and application settings up to the cloud. When you go online to research a problem, that data connects you to other people who have made similar changes. There’s already a tremendous amount of user-to-user support happening on the Net. Think how much more effectively it might happen in the context of pooled data.

Even if you don’t want to share your data with anybody else, it can be helpful just to you. Recently, for example, I moved a copy of Adobe Audition from XP to Vista. The settings didn’t come along for the ride, and it was a pain to reconstruct them. If systems and applications routinely reflected settings to a private space in the cloud, migration wouldn’t be such a headache.

Today’s New York Times has a story on regional variation in the availability and cost of health care. The story is accompanied by a “multimedia interactive graphic” — that is, a Flash visualization that charts the following variables on a U.S. map:

• Reimbursements
• Total
• Acute care
• Outpatient
• Surgical
• Surgery Rates
• All
• Heart bypass
• Knee replacement
• Mastectomy
• Enrollees

For each mapped variable, mousing over the displayed hospital referral regions yields the local, state, and national values for that variable.

It’s nicely done. There’s no question that, as of mid-2007, this is cutting-edge data interactivity for the mainstream. But times are changing fast. The Times sourced this data from the Dartmouth Atlas of Health Care. It took me five minutes to download the surgical data, upload it to Dabble DB, and publish a similar map along with a complete tabular dump.

Of course I cheated by aggregating only to the state level, which Dabble DB can do easily, rather than to the level of hospital referral regions. And I left out the national averages. But still, it’s striking to see what can be accomplished in a few minutes with no programming.

It’s even more striking to see what you can do directly on the Dartmouth site. Suppose you were thinking about having a knee replacement done in Keene, NH, and you wondered how many of these procedures are done at the Keene hospital, at other hospitals around the state, and in Boston and New York. Here’s the answer:

Knee Replacement per 1,000 Medicare Enrollees HSA Level Rates (2003)
Area	Population	Rates	Ratio to Benchmark	Surplus/ Deficit
*Keene , NH	8,047	5.07	–	–
Laconia , NH	8,116	8.60	1.70	29
Concord , NH	13,428	8.49	1.68	46
National Average	28,767,985	6.88	1.36	52,049
Lebanon , NH	9,483	6.73	1.33	16
State: New Hampshire	148,431	6.54	1.29	218
Manchester , NH	20,748	6.32	1.25	26
Rochester , NH	5,806	6.25	1.23	7
Nashua , NH	16,706	6.16	1.22	18
Dover , NH	8,462	5.16	1.02	1
Exeter , NH	10,624	4.96	0.98	-1
Boston , MA	67,651	4.90	0.97	-11
Manhattan , NY	166,112	3.32	0.66	-290

Wow! Hats off to the team at the Dartmouth Atlas of Health Care. This is the sort of thing that will change expectations about what interactive data ought to mean.

For this week’s ITConversations show I talked with Tessa Lau about Project Koala, a “a system for recording, automating, and sharing business processes performed in a web browser.” I’ve been interested in that idea for a long time, and mentioned it most recently in this item on pooling citizens’ collective knowledge about the services of government websites, and about how to make effective use of those services. In a comment on that item, Koranteng Ofosu-Amaah mentioned Project Koala and suggested that I speak with Tessa about it, so I did.

Of course we’ve had macro recorders since the dawn of computing, and Koala is yet another of those. What’s different? Crucially, the ability not only to capture and replay, but also to share, performances of tasks. The descriptions of those tasks are shared on a Wiki, and they’re written in an English-like syntax that’s very close to what you’d write if you were narrating instructions, e.g.: “Enter 94301 into the Search By Zipcode textbox, then click the Continue button.” These instructions can be edited, tagged, searched, and indexed by the URLs embedded in them. In theory that will enable us to pool our experiential knowledge of web applications. In practice, we’ll see. Koala has yet to emerge from IBM’s research lab. But you’ve got to love the idea.

For a while now I’ve been uncomfortable with the words user and content, and with the phrase user-generated content. But although produced or created are almost certainly better generic terms than generated, I’ll admit that I’ve failed to come up with a generic alternative to user or content (a bullshit word as Doc Searls rightly notes).

The commentary attached to Jimmy Guterman’s recent plea — Don’t call me a user! — has convinced me that there may not be superior generic alternatives. But several of the comments there reach the same conclusion that I have. Use the generic terms when necessary. But wherever possible, be more specific. The word user connotes a role, but so does member or contributor or participant and, even more specifically, writer or photographer or indexer or webjay.

The latter is a complete neologism, of course, but note the effect that it had at webjay.org. The reflex would have been to say things like “OddioKatya is my favorite Webjay user.” But because the term webjay was so active and so evocative — a DJ for the web — it became natural to simply say “OddioKatya is my favorite webjay.”

Likewise content. Because a more distinctive and evocative term was available — playlist — you’d never think of saying “I love OddioKatya’s content.” Instead you’d say “I love OddioKatya’s playlists.”

While we’re at it, webjays are not generators of playlists, they are curators of them. This isn’t just pedantry. Language governs thought, and when we enrich our language we enrich our individual and shared mental lives. With evocative and precise vocabulary, we can imagine more and accomplish better.

When Ryan Sholin’s manifesto on the future of newspapers appeared the other day, the blogosphere cheered loudly. “Great summary,” said one commenter, “Too bad they’re not listening.”

“They” are the newspaper writers, editors, and journalists — and the J-school teachers — whose attitudes and skills require a major overhaul:

Get over the whole bloggers vs. journalists thing…

…you and Mr. Notebook need to make some new friends, like Mr. Microphone and Mr. Point & Shoot.

Although everything on Ryan’s 10-point list is devastatingly true, it’s important to consider all the reasons why “they’re not listening.” In an item linked from his manifesto, Ryan expounds on the new skillset. The list of desiderata includes:

1. Can you code a Flash stage for chaptered Soundslides?
2. Can you edit audio, photos, and video into a compelling multimedia presentation?
3. Can you manage a community of users?
4. Can you moderate comments and forums and reader-contributed stories and photos and video?
5. Can you build a maps mashup that feeds itself with data scraped from public records?
6. Can you design interactive graphics in Flash?

Most of these things I’m able to do by virtue of long experience in a variety of disciplines and much intensive self-training. A few (items 1 and 6) I’ve never done. Sure, the world would be a better place if more journalists — and indeed more professionals in all fields — possessed these and related skills. It’s precisely for that reason that those of us who do possess them ought to think carefully about how to help others come up to speed. Hitting them over the head with a bewildering list of new skills foreign to them seems more likely to alienate than to inspire. Perhaps we could use our vaunted new media storytelling ability to tell stories that demystify what we’ve learned to do. What we see as stubborn foot-dragging is, in many cases, just fear. It’s a scary chasm to cross. Among our new skills, we should include the ability to build conceptual bridges that help people cross it.

One approach that’s helpful in all situations, but perhaps especially so here, is to teach by analogy. I understand why we focus on the newness of new media. There’s a lot that really is new, and it’s natural to celebrate that. But much remains the same, and it’s important to point that out too.

In my case, for example, Mr. Microphone and Mr. Camcorder were new modes of expression that I adopted only recently. As I discussed in this interview for the IDG, it was (and still is) a challenge to become proficient with these tools. But I probably haven’t said enough about how comfortably my writing and editing skills transfer into these domains. Editing audio and video feels different in some ways, but very familiar in other ways. The literary experience I brought with me into the audiovisual realm counts for a lot.

It’s the same story with interactive and data-rich information display. True, I bring some programming chops to the table that many others would lack, but the core skills are common-sense numeracy and a good sense of composition.

In my interview with Bill Buxton I asked Bill how you can help people adopt unfamiliar user interfaces. He said the same thing I’m saying here: You rely on analogy. By way of example, he proposed a force-sensitive mouse button. The harder you press, the faster it makes things scroll. Nobody’s ever used one of those, but he suggests (and I agree) that everyone could, because everyone drives a car with an accelerator pedal.

If we want newspapers to reinvent themselves, we might want to tone down the paradigm-shift rhetoric and focus more on how the important skills can and will transfer.

The audio interview mentioned in my review of the new Leonard Richard and Sam Ruby book, RESTful Web Services, is now available at ITConversations. This one presented a bit of an audio challenge because Leonard speaks loudly and Sam speaks very softly. In the course of working on this show, ITConversations’ ace audio engineer Paul Figgiani reviewed my recording setup and we discovered that what I thought was a mix-minus configuration actually wasn’t. So thanks to Paul it is now, and things should be sounding better from now on.

This week’s show is timely, given the recent remarks about REST by the Burton Group’s Anne Thomas Manes. Predictably, that led to another round of WS-* naysaying. For what it’s worth, I stand by what I’ve been saying all along, in a variety of places including this InfoWorld cover story: there’s WS-Heavy, there’s WS-Lite, and for every situation there’s a WS-JustRight that may rely on elements of one, the other, or both. The Richardson/Ruby book brings much-needed clarity to the WS-Lite end of the tolerance continuum, and that’s a great thing. But when we celebrate one end of the continuum, why must we deprecate the other? As Mike Champion recently noted:

The WS technologies are taking hold, deep down in the infrastructure, doing the mundane but mission critical work for which they were designed.

There are plenty of nails to be hammered and, thankfully, we’ve got more than one hammer.

ITConversations, the channel where my weekly podcast appears, has a sister channel called Social Innovation Conversations, which is an initiative of Stanford’s Center for Social Innovation. Over the weekend, while working around the house and yard, I caught up on my SIConversations listening. One compelling episode was a conversation with Michael Pollan, author of The Omnivore’s Dilemma, and John Mackey, CEO of Whole Foods. The two met initially after Mackey responded to criticism, in Pollan’s book, that Whole Foods is fostering the industrialization of the organic movement. Here’s a review of their joint appearance at Berkeley, which helpfully includes links to the back-and-forth correspondence published prior to the event.

I should probably take it for granted, by now, that issues raised in that book (which I haven’t yet read) are being discussed by the author and one of his subjects. And that those ongoing discussions are available to everybody in written form as well as in audio and video. And that the reactions to those ongoing discussions are similarly available to everybody. But somehow I can’t yet take any of this for granted. It still amazes me every day.

In any event, one of the points Mackey makes in that conversation is that although there are certainly problems associated with large-scale commercial production of organic food, it’s not inherently a bad idea to operate such a business at scale. On the contrary, sustaining a business at scale is part of what will make it possible to sustain a healthier food network.

That’s a common refrain on SIConversations. Most speakers believe, and argue, and in many cases have proved, that the changes they seek — in the realms of food, energy, health, and the environment — not only represent sound business opportunities and, in fact, but can only be accomplished with the help of sound business practices. So I was a bit surprised by the following comment from Nic Frances in another episode of SIConversations featuring three social entrepreneurs who “discuss what is takes to unleash the power of business to make the world a better place.” He said:

I looked at Microsoft and Gates, and thought, this man has changed AIDS like nobody else has on the planet. He has brought more money than has ever been brought to the issue. He’s brought the focus of somebody who knows how to grow a business. And he said ‘We’re going to change it.’ But actually if you really wanted to change AIDS or poverty in the world, what you would do is give away Microsoft free as an open platform for people to share information.

That was an oddly discordant note. Why would a social innovator who sees business opportunity everywhere else see none in software? Now of course, software is a very different kind of commodity from food, energy, or healthcare. But it’s also a fundamental enabler of the reorganization of the networks that deliver those commodities. Free software will play a key role in that reorganization, and so will commercial software. I’d love to hear an SIConversations show that explores the business opportunity for commercial software as an enabler of social innovation.

In the latest episode of the Long Now lecture series, Steven Johnson talks about the process of thinking on multiple scales simultaneously, both in time and space. Stewart Brand has a nice summary of the talk, which is full of interesting stuff, but there’s one piece of it that I want to highlight here because it relates to one of my current themes about citizen use of government data.

If you’re an Edward Tufte fan, like me, you’ll know the story of the 1854 cholera outbreak in London, and of John Snow’s map which showed deaths clustered around the Broad Street pump and which proved that the cause was bad water, not bad air (miasma). That story plays a central in Steven’s current book, and in his talk he points out that Snow was part of a larger cast of characters. One important but neglected figure was Henry Whitehead, a local vicar who collaborated with Snow. Another was William Farr, a government statistician. Although he initially favored the incorrect miasma theory, Farr had the good sense to publish the data that enabled others to find the right answer. Steven says:

The government could have said “OK, we’re going to compile all these statistics about who’s dying of what, where,” but Farr had this great idea that we’re going to make it available to everyone because someone maybe was going to find something of interest. And without that open access to the government statistics, the case wouldn’t have been made convincingly enough.

Rereading the comments on this entry, it’s humbling to realize that, 150 years later, that kind of access still seems like a novelty.

And in cases where we do have access to the data, it’s frustrating to see how little effective use we are able to make of it, despite all the bandwidth and computational resources available to us. Arranging pushpins on maps is a fine thing to do, but we should expect, and be able to do, more.

Greg Whisenant says “I’d be interested in features that people would like to see in a ‘basic analytics’ framework.” Rather than answer in general terms, I’ll talk about a specific situation. Here in idyllic smalltown Keene, New Hampshire, we’re seeing a very disturbing increase in crime. It’s nothing that would raise anyone’s blood pressure in Chicago or San Francisco, but there have been a lot more burglaries lately, and a handful of violent assaults of the sort we almost never see here.

By way of background, the police station recently moved from downtown to a peripheral location. One hypothesis is that our crime wave correlates with that move: less police presence downtown, more crime. One proposed response is to beef up that presence by creating and manning staffing a downtown substation. How do we test this hypothesis and evaluate that proposal?

I’ve tried (so far unsuccessfully) to get hold of the raw data: crimes, by type and date and location, as far back as our electronic records go. Let’s assume I can get that data. What then? For starters, I’d like to count incidents before and after the move. But I’d also like to count incidents within circles of varying radii centered on the police station, over time, and see an animated visualization of that information. Or rather, since I’m somebody with the programming chops to make something like that happen, I’d like to see a toolkit that would enable an ordinary citizen with data and a hypothesis to make that happen.

In the latest episode of my Microsoft Conversations series I got together with Bill Buxton to talk about the design philosophy set forth in his new book Sketching User Experiences. Nowadays Bill is a principal researcher with Microsoft Research, and before that he was chief scientist at Alias/Wavefront, but his involvement in the design of software and hardware user interfaces goes all the way back to Xerox PARC. Along the way he’s accumulated a fund of wisdom about what he calls design thinking — a way of producing, illustrating, and winnowing ideas about how products could work.

I haven’t yet received my copy of his book, but my background for this conversation was a talk given last November at BostonCHI, the Boston chapter of the ACM’s special interest group on computer-human interaction. In that talk (which summarizes key themes from the book), and also in this conversation, Bill lays down core principles for designing effective user experiences.

He proceeds from the assumption that sketching is fundamental to all design activity, and explores what it means to sketch a variety of possible user experiences. His approach is aggressively low-tech and eclectic. He argues that although you can use software tools to create fully-realized interactive mockups, you generally shouldn’t. Those things aren’t sketches, they’re prototypes, and as such they eat up more time, effort, and money than is warranted in the early stages of design. What you want to do instead is produce sketches that are quick, cheap, and disposable.

How would you apply that strategy to the design of, say, the Office ribbon? When Bill talks about sketching, he means it literally:

You’d start with paper prototyping — quickly hand-rendered versions, and for the pulldown menus and other objects you’d have Post-It notes. So when somebody comes with a pencil and pretends it’s their stylus and they click on something, you’ve anticipated the things they’ll do, and you stick down a Post-It note.

What matters here isn’t the interaction between the test subject and the prototype, because it isn’t really a prototype, it’s a sketch. Rather, what matters is the interaction between the test subject and the designer. The sketch need do no more than facilitate that interaction.

Continuing with the same example, here’s how an eclectic strategy keeps things simple and cheap:

Now that will give you the flow and the sequence of actions, but it will not give you the dynamics in terms of response time. To show that, I’d use exactly the same things, photograph them, and then make a rough pencil-test video so I could play back what I think the timing has to be to show it in realtime. It’s a combination of techniques, where none is sufficient on its own.

Later in the conversation, he challenges some of my favorite themes. Bill’s skeptical about the notion (popularized by Eric von Hippel) that lead users can be co-designers of products. And he doesn’t think that logging interaction data is as useful as I think it is. But he agrees with me that a key weakness of paper prototypes is their inability to incorporate the actual data that animates our experiences of products and services. One of his examples: MP3 players think in terms of songs, not movements, so if you load one with classical music you’ll find a bunch of duplicate songs called Adagio. In such a case, Bill admits, you’d like to have used a more fully-realized prototype that could have absorbed real data and flushed out these kinds of problems. His point isn’t that you should never deploy heavier design artillery, but rather that you should reserve it for when it’s absolutely necessary. Much of the time, he believes, sketching is faster, cheaper, and more productive.

Several months ago my bank implemented an anti-phishing scheme called Site ID, and now my mortgage company has gone to a similar scheme called PassMark. Both required an enrollment procedure in which I had to choose private questions and give answers (e.g., mother’s maiden name) and then choose (and label) an image. The question-and-answer protocol mainly beefs up name/password security, and secondarily deters phishing — because I’d notice if a site I believed to be my bank or mortgage company suddenly didn’t use that protocol. The primary anti-phishing feature is the named image. The idea is that now I’ll be suspicious if one of these sites doesn’t show me the image and label that I chose.

When you’re talking about a single site, this idea arguably makes sense. But it starts to break down when applied across sites. In my case, there’s dissonance created by different variants of the protocol: PassMark versus Site ID. Then there’s the fact that these aren’t my images, they’re generic clip art with no personal significance to me. Another variant of this approach, the Yahoo! Sign-In Seal, does allow me to choose a personally meaningful image — but only to verify Yahoo! sites.

These fragmentary approaches can’t provide the grounded and consistent experience that we so desperately need. One subtle aspect of that consistency, highlighted in Richard Turner’s CardSpace screencast, is the visual gestalt that’s created by the set of cards you hold. In the CardSpace identity selector, the images you see always appear together and form a pattern. Presumably the same will be true in the Higgins-based identity selector, though I haven’t seen that yet.

I can’t say for sure, because none of us is yet having this experience with our banks and mortgage companies, but the use of that pattern across interactions with many sites should provide that grounded and consistent experience. Note that the images forming that pattern can be personalized, as Kevin Hammond discusses in this item (via Kim Cameron) about adding a handmade image to a self-issued card. Can you do something similar with a managed card issued by an identity provider? I imagine it’s possible, but I’m not sure, maybe somebody on the CardSpace team can answer that.

In any event, the general problem isn’t just that PassMark or Site ID or Sign-In Seal are different schemes. Even if one of those were suddenly to become the standard used everywhere, the subjective feeling would still be that each site manages a piece of your identity but that nothing brings it all together under your control. We must have, and I’m increasingly hopeful that we will have, diverse and interoperable identity selectors, identity providers, relying parties, and trust protocols. But every participant in the identity metasystem must also have a set of core properties that are invariant. One of the key invariant properties is that it must bring your experience of online identity together and place it under your control.

Because of travel, last week’s ITConversations show was a rerun of my conversation with Lou Rosenfeld about a cluster of topics including information architecture, search analytics, print and online publishing, designing for usability, tagging, and microformats. This week’s show is a conversation with ITConversations founder Doug Kaye about his new project, PodCorps, which aims to connect producers of spoken-word events with stringers who can help get those events audio- or video-recorded and then published on the Web.

Here’s a fun fact I uncovered. Doug hadn’t heard of one of my favorite things lately, the LibriVox project. When I mentioned that Hugh McGuire cites AKMA’s collaborative recording of Larry Lessig’s Free Culture as a primary inspiration, Doug pointed out that he was the reader for the first chapter of that project. Small world!

RESTful Web Services, by Leonard Richardson and Sam Ruby, was published this month. I interviewed the authors yesterday for an upcoming ITConversations show, but I also want to spell out here why I think it’s such an important book.

In the realm of IT you could hardly pick a more controversial topic. Or, in a way, a more unlikely one, given that the REST (Representational State Transfer) architectural style has its roots in what would normally have been an obscure Ph.D. thesis. Roy Fielding, the author of that thesis, told me in an interview that he was surprised by its breakout popularity. But he probably shouldn’t have been. There are not many technologies as foundational as the Hypertext Transfer Protocol (HTTP), whose principles that thesis defines.

But Fielding’s thesis is a thesis, not a practical guide. The effort to bridge from theory to practice has produced a considerable amount of folklore. “We’re writing a book,” the authors say in their web introduction, “to codify the folklore, define what’s been left undefined, and try to move past the theological arguments.” The mission is clearly defined in the first chapter:

My goal in this book is not to make the programmable web bigger. That’s almost impossible: the programmable web already encompasses nearly everything with an HTTP interface. My goal is to help make the programmable web better: more uniform, better structured, and using the features of HTTP to greatest advantage.

The book opens by usefully distinguishing between a set of architectural styles (REST, RPC [remote procedure call], REST-RPC hybrid) and suite of technologies (HTTP, XML-RPC, WS-*, SOAP). We tend to conflate architectures with technologies because they usually go together, but that’s not necessarily the case. The authors cite Google’s SOAP API (and other “read-only SOAP and XML-RPC services” as being “technically REST architecture” but nevertheless “bad architectures for web services, because they “look nothing like the Web.”

This book asserts that most services can, and should, “look like the Web,” and it spells out what that means. Among the key principles:

• Data are organized as sets of resources
• Resources are addressable
• An application presents a broad surface area of addressable resources
• Representations of resources are densely interconnected

To illustrate these principles, the authors work through a series of examples from which they distill gems of practical advice. When designing URIs, for example, they recommend that you use forward slashes to encode hierarchy (/parent/child), commas to encode ordered siblings (/parent/child1,child2), and semicolons to encode unordered siblings (/parent/red;green). Pedantic? Yes. And bring it on. Lacking a Strunk and White Elements of Style for URI namespace, we’ve made a mess of it. It’s long past time to grow up and recognize the serious importance of principled design in this infinitely large namespace.

Here’s another key principle: “When in doubt, model it as a resource.” To illustrate that principle in a dramatic way, the authors apply it to a problem that RESTful web services are normally thought incapable of solving: transactions. By modeling the transaction itself as a resource, they arrive at the following:

First I create a transaction by sending a POST to a transaction factory resource:

POST /transactions/account-transfer HTTP/1.1
Host: example.com

The response gives me the URI of my newly created transaction resource:

201 Created
Location: /transactions/account-transfer/11a5

I PUT the first part of my transaction: the new, reduced balance of
the checking account.

PUT /transactions/account-transfer/11a5/accounts/checking/11 HTTP/1.1
Host: example.com

balance=150

I PUT the second part of my transaction: the new, increased balance of
the savings account.

PUT /transactions/account-transfer/11a5/accounts/savings/55 HTTP/1.1
Host: example.com

balance=250

At any point up to this I can DELETE the transaction resource to roll
back the transaction. Instead I’m going to commit the transaction:

PUT /transactions/account-transfer/11a5 HTTP/1.1
Host: example.com

committed=true

I don’t think my bank’s going to be adopting this technique any time soon, but it’s a fascinating thought experiment which suggests that what the authors call resource-oriented architecture (ROA) is a young and in many ways still relatively unexplored discipline.

On the question of ROA versus SOA (service-oriented architecture), the authors say that for certain kinds of enterprisey problems — including advanced security protocols and complex coordinated workflows — only SOA meets the need. They recommend it for these purposes, when the need arises. But in the many situations where the need does not arise, they recommend starting with ROA.

I’m inclined to agree, but I’d feel better about that recommendation if the glide path from ROA to SOA were smoother. It isn’t. Toward the end of our interview I asked Sam Ruby, who has been a long and forceful advocate for a smooth glide path, whether he thinks we’ll achieve it. He doesn’t. That worries me, but I haven’t given up hope. I’ve always seen ROA and SOA as points along what Frank Martinez calls a tolerance continuum. Among its other accomplishments, this excellent book advances that important point of view.

As an early supporter and ongoing fan of WebJay, I’m sorry to hear that the service will close in June and curious to know why.

Today my digital assets are spread out all over the place. Some are on various websites that I control, and a lot more that I don’t. Others are on various local hard disks that I control, and a lot more that I don’t. It’s become really clear to me that I’d be willing to pay for the service of consolidating all this stuff, syndicating it to wherever it’s needed, and guaranteeing its availability throughout — and indeed beyond — my lifetime.

The scenario, as I’ve been painting it in conversations with friends and associates, begins at childbirth. In addition to a social security number, everyone gets a handle to a chunk of managed storage. How that’s coordinated by public- and private-sector entities is an open question, but here’s how it plays out from the individual’s point of view.

Grade 3

Your teacher assigns a report that will be published in your e-portfolio, which is a website managed by the school. Your parents tell you to write the report, and publish it into your space. Then they release it to the school’s content management system. A couple of years later the school switches to a new system and breaks all the old URLs. But the original version remains accessible throughout your parents’ lives, and yours, and even your kids’.

Grade 8

On the class trip to Washington, DC, you take a batch of digital photos. You want to share them on MySpace, so you do, but not directly, because MySpace isn’t really your space. So you upload the photos to the place that really is your space, where they’ll be permanently and reliably available, then you syndicate them into MySpace for the social effects that happen there.

Grade 11

You’re applying to colleges. You publish your essay into your space, then syndicate it to the common application service. The essay points to supporting evidence — your e-portfolio, recommendations — which are also (to a reasonable degree of assurance) permanently recorded in your space.

College sophomore

You visit the clinic and are diagnosed with mononucleosis. You’ve authorized the clinic to store your medical records in your space. This comes in handy a couple of years later, when you’ve transferred to another school, and their clinic needs to refer to your health history.

Working professional

You use your blog to narrate the key events and accomplishments in your professional life, and to articulate your public agenda. All this is, of course, published in your space where you are confident (to the level of assurance you can reasonably afford) that it will be reliably available for your whole life, and even beyond.

Although this notion of a hosted lifebits service seems inevitable in the long run, it’s not at all clear how we’ll get there. The need is not yet apparent to most people, though it will increasingly become apparent. The technical aspects are somewhat challenging, but the social and business aspects are even more challenging.

In social terms, I think it’ll be hard to get people to decouple the idea of storage as a service from the idea of value-added services wrapped around storage.

On the business side, my conversations with Tony Hammond and Geoffrey Bilder have given me a glimpse of how these issues are being approached in the world of scholarly and professional publishing. But it’s not yet apparent that the specialized concerns driving these efforts will, in fact, generalize in important ways to almost everybody.

Parents nowadays face tough questions about whether to monitor or (try to) control their kids’ use of the Internet, and if so, how. Although my personal opinion is that trying to restrict access is a losing battle, I understand why the idea is appealing. You’d like your kids to have some maturity and some perspective under their belts before encountering some of what the Internet so readily brings to their attention. When my kids were younger, the Internet was younger too. I guess if they were still that young I’d be wishing I could create a sandbox for them, even though I don’t think you can. But they’re teenagers now, and they have their own computers. For two reasons, activating the parental controls on those computers isn’t the strategy I want to pursue.

The first reason is that I don’t think filtering the Internet is feasible. Even if we could agree on a definition of what may be harmful, which we never could, people will find ways to route around censorship. Meanwhile we’ll inevitably censor things we never meant to — like, for example, my InfoWorld blog.

The second reason is that I don’t want to incent my kids to route around controls I might try to impose. Nor do I want to force them to go elsewhere to experience an uncensored Internet. The reality of the Internet, like the reality of the world, is something they’ll be dealing with for the rest of their lives. I’d rather they engage with that reality at home where I can more easily keep track of their activities.

If you want to be able to monitor without imposing explicit controls — in other words, trust but verify — then it’s worth knowing about the feature of Windows Vista that supports that preference. It’s in Control Panel -> User Accounts and Family Safety -> Parental Controls. There are two On/Off choices. The first, Parental Controls, enforces any of the controls you elsewhere define. These include restrictions about which websites are accessible, when the computer may be used, and which games or other programs may be used.

My strategy is to leave Parental Controls off, but switch on the second On/Off choice: Activity Reporting. That produces a detailed report about which websites were visited, which applications were used when, which games were played, messages sent/received and contacts added (if the kids use Outlook and Windows Messager, which mine don’t), and more.

The crucial item here, for me, is websites visited. That’s recently become an issue that I want to keep an eye on. With this setup I can, in a way that’s browser-independent, persistent across flushes of the browser cache, and very unlikely to be disabled.

Windows XP and Mac OS X don’t offer the same capability out of the box, but there are of course lots of third-party add-ons. Not ever having tried them myself, I’d be interested to hear how effectively they can be used to implement a “trust but verify” policy. And more generally, I’d be interested to hear about how other parents of teenagers are dealing with the difficult tradeoffs involved in this thorny issue.

More than 25 years ago, Allen Wirfs-Brock created one of the early implementations of Smalltalk. He was working at Tektronix at the time, as was Ward Cunningham who became the first user of Tektronix Smalltalk. Allen later served as chief scientist of Digitalk-ParcPlace and CTO of Instantiations, then joined Microsoft four years ago. His original charter was to work on future strategies for Visual Studio, but recently — in light of growing interest in dynamic languages at Microsot — he’s returning to his roots.

In the latest installment of my Microsoft Conversations series we review the history of Smalltalk, and trace the evolution of the techniques that it (and Lisp) pioneered, from the early implementations to such modern descendants as Python and Ruby.

I’m always looking for ways to explain why dynamic programming techniques are so important, and a great explanation emerged from this conversation. A Smalltalk system is, among other things, a population of continuously evolving objects that communicate by passing messages. That same description applies to another kind of system: the Internet. I suggested — and Allen agreed — that this congruence is driving renewed appreciation for dynamic languages.

Matt McAlister heard “crackling firearms” in his San Francisco neighborhood and wrote a wonderful essay on a theme that was central to my keynote talk last week at the GOVIS conference: how citizens can and will work with governments to diagnose social problems and develop solutions. When the District of Columbia’s DCStat program rolled out last summer, I was delighted by the forward thinking involved. Publishing the city’s operational data directly to the web, for everyone to see and analyze, with the explicit goal of making the delivery of government services transparent and accountable, was and is an astonishingly bold move. And as Matt found when investigating crime in his neighborhood, it’s still part of the unevenly distributed future:

I then found the official San Francisco Police Department Crime Map. Of course, the data is wrapped in their own heavy-handed user interface and unavailable in common shareable web data formats.

Access to data is good, and access to data in useful formats is better, but these are only the first steps. We need to make interpretations of the data, compare and discuss those interpretations, and use them to inform policy advocacy. The mashups that Matt reviews are a glimpse of what’s to come, but these interactive visualizations have a long way to go.

Here’s another glimpse of what’s to come: I took a snapshot of the DC crime data, uploaded it to Dabble DB, built a view of burglary by district and neighborhood, and published it at this public URL. There are two key points here. First, discussion can attach to (and will be discoverable in relation to) that URL. Second, the data behind the view is also available at that URL, in a variety of useful formats, so alternate views can be produced, pointed to, and discussed.

Still, these are only views of data. There’s no analysis and interpretation, no statistical rigor. Since most ordinary citizens lack the expertise to engage at that level, are governments that publish raw data simply asking for trouble? Will bogus interpretations by unqualified observers wind up doing more harm than good?

That’s a legitimate concern, and while the issue hasn’t yet arisen, because public access to this level of data is a very new phenomenon, it certainly will. To address that concern I’ll reiterate part of another item in which I mentioned John Willinsky’s amazing talk on the future of education:

Willinsky talks about how he, as a reading specialist, would never have predicted what has now become routine. Patients with no ability to read specialized medical literature are, nonetheless, doing so, and then arriving in their doctors’ offices asking well-informed questions. Willinsky (only semi-jokingly) says the Canadian Medical Association decided this shouldn’t be called “patient intimidation” but, rather, “shared decision-making.”

How can level 8 readers absorb level 14 material? There are only two factors that govern reading success, Willinsky says: motivation, and context. When you’re sick, or when a loved one is sick, your motivation is a given. As for context:

They don’t have a context? They build a context. The first time they get a medical article, duh, I don’t know what’s going on here, I can’t read the title. But what happened when I did that search? I got 20 other articles on the same topic. And of those 20, one of them, I got a start on. It was from the New York Times, or the Globe and Mail, and when I take that explanation back to the medical research, I’ve got a context. And then when I go into the doctor’s office…and actually, one of the interesting things…is that a study showed that 65% of the doctors who had had this experience of patient intimidation shared decision-making said the research was new to them, and they were kind of grateful, because they don’t have time to check every new development.

When your loved one is sick, you’re motivated to engage with primary medical literature, and you’ll build yourself a context in which to do that. Similarly, when your neighborhood is sick, you’ll be motivated to engage with government data, and you’ll build yourself a context for that.

The quest for context could, among other things, lead to a renewed appreciation for a tool that’s widely available but radically underutilized: Excel. Most people don’t earn a living as quants, so Excel, for most people, winds up being a tool for summing columns of numbers and arranging text in tabular format. That may change as more public data surfaces, and as more people realize they want to be able to interpret it. In which case Chris Gemignani and the rest of the Juice Analytics team will emerge as leading resources available to motivated citizens wanting to learn how to make better use of Excel.

In my talk on Friday at the GOVIS (government information systems) conference in Wellington, I wasn’t the only one to suggest that web 2.0 attitudes will change the relationship between governments and citizens. That notion now seems to be pretty firmly established, and the question is not whether citizens will collaborate with their governments, but rather how.

Among other developments, I think we’ll soon see a refreshing new approach to the consumption of government services. A couple of weeks ago at Berkeley’s school of information I met Anna Kartavenko, one of Bob Glushko’s graduate students, She’s working on ways to make the byzantine California regulatory apparatus more accessible. If you’re starting a business in that state, it’s really hard to figure out which licenses you need to apply for, as well as how (and in what order) to apply for them.

The problem is universal, of course, and folks at GOVIS were wrestling with it too. When you’re providing the information systems that both document and implement government services, you certainly want to do everything right in terms of system and information architecture. But I suspect there’s about to be a new force in the world that will work toward the same ends — easy discovery and effective use of services — by very different means. That force is shared experiential knowledge.

Yes, search should give the right answer, and the systems that search points you to should work well. No, these things don’t always happen. But even if they do, you’d still like to plug into somebody who’s been down the same path you are traveling. A formal description of a procedure is never enough. If possible, we’d always like to hear from somebody who’s been there, done that, knows the drill, and can point out the pitfalls.

What we loosely call social media are beginning to create that possibility. For a variety of reasons, people are beginning to document and share what they know. If you write it down, you’ll be able to remember it yourself in case you have to replay the steps. And writing it down in a shared information system in the cloud is becoming a more reliable way to assure your own future access to this documentation than writing it down locally.

To the extent your knowledge is a source of competitive advantage, you’ll want to be cautious about how much of it you publish. But then again, the reputation you establish by publishing some of your knowledge may lead to new opportunities to use that knowledge for your own gain.

Along with these incentives, which I classify as examples of enlightened self interest, there are also purely altruistic motives, and I don’t discount those. But let’s just stick with enlightened self interest for now. Given those incentives to share knowledge, how can we lower the activation threshold for sharing?

I think one answer will emerge from the intersection of social bookmarking and clickstream logging. Suppose that instead of bookmarking and tagging a single URL, you could bookmark and tag a sequence of page-visiting and form-filling events. The sequence corresponds to some complex multi-step task. The performance of the task crosses several (or many) online jurisdictions. The outcome might be successful or not: “Yes, I got the license,” or “No I didn’t.” But in either case, it would be qualified by an anecdotal report: “Yes, I got the license, but I found out that if you’re in my category you need an import license and you have to meet the following insurance requirement.”

You couldn’t reasonably expect very many people to reflect on their encounters with online bureaucracy and take time to write reports like that. But what if it were a much more lightweight activity, like the difference between writing a blog entry and tossing off a del.icio.us bookmark or a Twitter message? Then participation becomes much more likely.

The key ingredient here is identifying a sequence of events in the browser (or rich client), and enabling people to visualize and then categorize and describe that sequence. And that seems eminently doable.

Jeremy Zawodny describes his method for preparing talks and asks:

If you end up speaking in front of audiences on a semi-regular basis, is your preparation experience anything like mine?

My process used to be what Jeremy describes — composing slides — but now it’s turned into something completely different and quite surprising to me. As I discussed here, I’ve finally trained myself to use dictation effectively. I’ll go out for a long walk, like two or three hours, and dictate a rough draft of the talk. I’m not able to do that continuously, I have to stop and think and start again, but I turn the recorder off during the think time so when I’m done I’ve got something approximating what the talk will be. Then I go for another long walk and listen to what I recorded, making notes about what slides to use. For last week’s talk I didn’t take those notes in audio form, I scribbled them down while walking, but next time I’m going to go back to audio capture. If you distill the long narrative down to short titles or phrases, it’s quick and easy to listen to a spoken distillation and write down the titles which become the armature for the slides.

The obvious reason why this works is that speaking out loud is good practice for speaking out loud. One of the subtler reasons is that exercise and fresh air really help. Another is that when I’m away from my office and can’t fiddle with a computer or look things up on the web, I have to literally think on my feet.

As I acknowledged here, I’m indebted to John Mitchell for suggesting this technique to me. According to him, it dates back to “the BBC WW2 radio correspondents, and then Edward R. Murrow.” Thanks again for the tip, John, it’s been really helpful.