In an essay last week about Yahoo Pipes, Tim O’Reilly said he was inspired, back in 1997, by a talk at the first Perl conference in which I had “expressed a vision of web sites as data sources that could be re-used, and of a new programming paradigm that took the whole internet as its platform.” Someone asked in the comments whether that idea hadn’t instead been put forward in Andrew Schulman’s talk. It turns out that neither Tim nor I can remember exactly what Andrew and I said, but I hope we both touched on this idea because it’s a big one that underlies the whole web services movement and much else besides.
Later on in that comment thread, Tim cites an email message from me in which I try to reconstruct what may have happened. One of the artifacts I dug up was this 1996 BYTE column (cleaner version here). That’s when the lightbulb clicked on for me, and I saw very clearly that the web was collection of components that I’d be able to wire together.
Of course all I was doing was drawing attention to what the creators of the web had intended and made possible. In my recent interview with Roy Fielding, for example, we talked about his early work on libwww-perl, the library that made websites into playthings for Perl programmers. Wiring the web was very much part of the original vision. The idea just needed some champions to broaden its appeal. That’s the role that I, among others, have played.
From that perspective, then, what of Yahoo Pipes? It delights me! Much more importantly, I think it could ultimately appeal to non-technical folks, but there are some conceptual barriers to overcome. The concept of “wiring the web” is one of those, but not the first one. The dominant way in which most people will “program” the web is by writing metadata, not code, and we’ll need an interface as friendly and powerful as Pipes to help them do that.
That last sentence won’t make any sense to the average non-technical person, but the example I gave yesterday might. A by-product of this presidential election cycle will be massive quantities of online video. We should expect to be able to reach into the various repositories and assemble coherent views by issue and by candidate, and Yahoo Pipes would be a great way to do that. But not until and unless the video has been sliced and diced and tagged appropriately so as to yield to structured search.
It’s the slicing and dicing and tagging, not the rewiring, that’s the real bottleneck. I talked last week about factoring group formation out of the various social networks into a common infrastructure. We need to do the same for tagging. How do I know whether to tag my contribution as HillaryClinton and NewHampshire and manufacturing or Hillary Clinton and NH and manufacturing? Where’s the immediate feedback that shows me, across tag-oriented services including YouTube and Blip, how my contribution does or doesn’t align with others, and how I might adjust my tag vocabulary to improve that alignment?
When I tag a video snippet with the name of a politician (“Hillary Clinton”) and a topic (“manufacturing”) I clearly envision a future query in which these slots are filled with the same values or different ones (“Barack Obama”, “energy”). And I clearly envision the kinds of richly-annotated topical remixes that such queries will enable. But such outcomes are not obvious to most people. We need to figure out how to make them obvious.
You know you’re a Jon Udell disciple *grin* when you realize you know what 1996 Byte magazine he’s talking about before you click the link! (I did, anyway, to doublecheck…and I was right. I think I still have that issue, too!). With that article, you helped me reach the same epiphany, and I will appreciate it forever.
I agree that the challenge of metadata is formidable; I have confidence, though. The good news is that it is something, like blogs, web pages, and even the videos themselves, that can be done by individual people, a bit at a time, building on each other.
Your tag-comparison “meta-taxonomy” idea is interesting, though…grab a few of the taglists and compare them to what you’re doing…good idea!
Part of the answer seems to be just a matter of user interface. One of the things I always loved, in principle, about Facebook was how it cross-referenced *every* database field describing people. The key, then, became how to get people to more consistently populate these fields so a machine could discover similarities without having solved the AI problem. Their first-order answer, which has worked quite well in practice, was just to introduce javascript auto-complete into many of their fields to be make it easy and obvious for people to type things consistently with how most others were typing them.
Of course, getting different services to share a common taxonomy to drive each instance of such a UI is the next question, but as a first-order solution I’ve been quite impressed at the effectiveness of this very simple form of persuasive technology — a very light hammer going a long way in what seemed like a heavy technological problem.
Seems like a google or web 2.0 company could do the job of surveying all tags on the Internet and drawing up similar ones together with the counts so that the top most common variations e.g. “New Hampshire” vs NewHampshire” and perhaps through association “NH” could be offered via a webservice API that sites could use to provide drop-down lists or suggestions. Then again, I guess there is no revenue model in a behind the scenes public service web API like that. I am thinking of how a year or two ago google demonstrated an AJAX drop-down list of the top searches on every keystroke. So, in the case of tags, as you’re typing tags it could be listing the Internet-wide most common ones.
Hi
by driving or constaining the tagging aren’t you killing the folksonomy?
Wasn’t the idea there that everyone tags as they wish and we’ll get a set of most popular tags?
Instead I’d address it from the other way around…
When you search for “Hillary Clinton” you should be offered with similar results i.e. those for “HillaryClinton”
Speaking of Pipes or something similar this would mean that instead of Equality operator we would have something like a Soundex. An advanced operator that extrapolates from the filtering criteria.
regards
Sic.
Yes. The searech engine should be smart enough to identify Hillary Clinton and HillaryClinton. Computers can do that a lot better than they can prescribe a meta language to everyone in the world who might want to do tagging.
“The search engine should be smart enough to identify Hillary Clinton and HillaryClinton.”
And before things even get that far, the tagging interface should help. For example, by providing realtime feedback like:
Tag Source “Hillary Clinton” “HillaryClinton”
A 33% 66%
B 50% 50%
C 66% 33%
Now admittedly that’s overkill for spellings of the same name that vary only slightly. It matters much more when you’re trying to decide between alternate vocabulary terms. Del.icio.us does a great job of prompting with choices, but of course only within the domain of its tag vocabulary. A prompter that worked across multiple tag-oriented services — and that gave realtime numeric feedback about the popularity of alternate terms among various constituencies — seems interesting.
Hi Jon and others.
Yahoo Pipes looks awesome! It too embodies what I really want to be able to do with the web and with software in general.
To me the real problem is pretty fundamental to the current way the web works. It’s really entrenched in a document paradigm. In order to extract one piece of info that could be “piped” to another component, either you have to fish around and find it within a web page (screen scrape it), or the publisher has to also provide a seperate XML document or Web-service API for that info — and each of these XML formats and APIs are often unique to each company/service. (Even if a stardard like RDF is used, it’s so wide open that you sometimes need to also apply special knowlege of application-specific semantics; and using standards like RSS is hard because it was never really meant for general non-news data publishing.)
You can’t just address a single unit of data (text or otherwise) or re-use in another context.
Recently I’ve been trying to build up some tools on top of a system called VOS (http://interreality.org) so you can build web pages out of more granular objects, which can be reused/filtered/rewired to create other web pages (similar to Yahoo Pipes), or used as inputs to other kinds of software components — even 3D virtual reality.
So far with VOS you can put together web pages out of VOS objects (the interreality.org site runs on this), and you can mix and match these objects in any number of pages. I’m still working on some automated blogging/news generation stuff, and some CGI/form stuff. I’m also looking for ideas for ways to take “live” data streams from external sources and display them in web pages and other media using this system; let me know if you think of any. There’s also a real lack of user-friendly tools for working with VOS object networks online.
Anyway, I invite anyone who’s interested to contact me to talk more about this. Or visit the web page and mailing list.
Best,
Reed
Jon- As you must know, you have the pieces to a solution in hand; combining faceted navigation with bulk user tagging of result sets will provide the immediate feedback you’re looking for together with the ability to rewire and align tags.
BTW, thanks for the indirect decade-old shout-out. – cheers, BPA
There is an assumption behind Jon’s piece that is interesting to question. The assumption is that consensus in tagging is both possible and a Good Thing. To stretch the metaphor a bit, think of tag clouds in which particular tags for given concepts attract more and more adherents until they precipitate out as perfectly formed, implicitly-agreed-upon tag raindrops. I wonder whether such tag rain is necessary. We have been playing with the idea of making tag-rules a basic unit of currency. I say tomahto you say tomayto, but so long I know your tomayto is my tomahto I can carry on tomahto-ing to my heart’s content. In this case we have a synoymy tag-rule, but there are a number of others (class-member, part-whole and so on). It is these rules that then become the tokens that we share within and across social networks. If you and I share the tomahto-tomayto synonymy rule we both benefit from the tagging of both tomato communities. This allows the diverse, idiosyncratic idiolects of tagging to be seen as something valuable that should be encouraged rather than something frowned upon that is to be ultimately eradicated.
“In order to extract one piece of info that could be “piped” to another component, either you have to fish around and find it within a web page (screen scrape it), or the publisher has to also provide a seperate XML document or Web-service API for that info”
Or…and I realize this hasn’t caught on yet, but still…the stuff is more sanely available within the web page as an island of structure. In other words: a microformat.
“The assumption is that consensus in tagging is both possible and a Good Thing.”
Depends what you mean by consensus. What I don’t mean is a static definition. What I do mean is a dynamic process that works the way natural languages work. The latter idea was what I tried to convey in a screencast called Language Evolution in Del.icio.us:
http://weblog.infoworld.com/udell/2005/03/14.html
The authority file concept that librarans love itself needs re-invention – but not likely that it’s longstanding exponents will do so.
Instead of “either/or” thinking about how a descriptive element (AKA: a tag) is applied to an item of interest, many rating/classifying/tagging situations could benefit from the ability to express the degree of application of a “tag” term, as well as to indicate the certainty of the tagger that the term applies. For example, instead of making a tag/no-tag decision (which is what the current authority control regime enforces), one acutally possesses the more nuanced ability to indicate that (1.) a given tag is 60% applicable to everything the item of interest is about; and (2.) the tagger is 90% certain that the tag applies to the item.
Given enough tags assigned this way – this no longer a problem with web-accessible items – techniques from psychophysics and/or marketing can then be brought to bear behind the scenes to relate the tagged item to other tagged items, while at the same time make it possible to tease out the tagger’s individual or group-related characteristics. Seeking consensus in tag assignation serves to squeeze out both “fractional applicability” plus individual and group differences in apprehending and/or appreciating an item’s characteristics.
Indeed, the “programmable web” is still forming. I recently found a nice little blog post titled “Pipes + SQL = Structured Web Query Language” that offers a great angle on the web-as-database concept.
The main thing that Yahoo Pipes has done is that it’s made the current “programmable web” more accessible to mainstream users. Just as so many programming languages make assembly language more accessible. However, there is still quite a learning curve for the average end-user. I have created some video tutorials to help lessen this curve. With your permission, here is the link: http://usefulvideo.blogspot.com/2007/02/yahoo-pipes-tutorials.html
Wonderful and informative web site.I used information from that site its great.b