In an essay last week about Yahoo Pipes, Tim O’Reilly said he was inspired, back in 1997, by a talk at the first Perl conference in which I had “expressed a vision of web sites as data sources that could be re-used, and of a new programming paradigm that took the whole internet as its platform.” Someone asked in the comments whether that idea hadn’t instead been put forward in Andrew Schulman’s talk. It turns out that neither Tim nor I can remember exactly what Andrew and I said, but I hope we both touched on this idea because it’s a big one that underlies the whole web services movement and much else besides.
Later on in that comment thread, Tim cites an email message from me in which I try to reconstruct what may have happened. One of the artifacts I dug up was this 1996 BYTE column (cleaner version here). That’s when the lightbulb clicked on for me, and I saw very clearly that the web was collection of components that I’d be able to wire together.
Of course all I was doing was drawing attention to what the creators of the web had intended and made possible. In my recent interview with Roy Fielding, for example, we talked about his early work on libwww-perl, the library that made websites into playthings for Perl programmers. Wiring the web was very much part of the original vision. The idea just needed some champions to broaden its appeal. That’s the role that I, among others, have played.
From that perspective, then, what of Yahoo Pipes? It delights me! Much more importantly, I think it could ultimately appeal to non-technical folks, but there are some conceptual barriers to overcome. The concept of “wiring the web” is one of those, but not the first one. The dominant way in which most people will “program” the web is by writing metadata, not code, and we’ll need an interface as friendly and powerful as Pipes to help them do that.
That last sentence won’t make any sense to the average non-technical person, but the example I gave yesterday might. A by-product of this presidential election cycle will be massive quantities of online video. We should expect to be able to reach into the various repositories and assemble coherent views by issue and by candidate, and Yahoo Pipes would be a great way to do that. But not until and unless the video has been sliced and diced and tagged appropriately so as to yield to structured search.
It’s the slicing and dicing and tagging, not the rewiring, that’s the real bottleneck. I talked last week about factoring group formation out of the various social networks into a common infrastructure. We need to do the same for tagging. How do I know whether to tag my contribution as HillaryClinton and NewHampshire and manufacturing or Hillary Clinton and NH and manufacturing? Where’s the immediate feedback that shows me, across tag-oriented services including YouTube and Blip, how my contribution does or doesn’t align with others, and how I might adjust my tag vocabulary to improve that alignment?
When I tag a video snippet with the name of a politician (“Hillary Clinton”) and a topic (“manufacturing”) I clearly envision a future query in which these slots are filled with the same values or different ones (“Barack Obama”, “energy”). And I clearly envision the kinds of richly-annotated topical remixes that such queries will enable. But such outcomes are not obvious to most people. We need to figure out how to make them obvious.