If you consider that (a) markup is inherently data and (b) the markup is in a known format (i.e. it hasn’t changed since you last looked at it) then that data can be extracted.

The example I saw was automating the process of going to a SERP, extracting the title/URL/Description of each of the “10 blue links” on the first 5 results pages and returning those 50 rows in a 3-column dataset. There’s no HTML pages in a SERP but Kapow can still loop over them because of the nature of HTML.
Plus it also has worflow (i.e. visit each of the first 5 SERPs in turn and union the results together)

-Jamie

By extracting it from HTML tables?

To that end I’ve just seena d emo of something called Kapow (don’t worry I’m not selling anything here. I have no vested interest, I just thinks its a cool technology) which is effectively a cross between a screen scraper and a ETL tool. It harvests data from a DOM and presents it in a RESTful data service – really fascinating stuff. I blogged about it in case you’re interested: http://blogs.conchango.com/jamiethomson/archive/2009/07/08/kapow-etl-for-html.aspx

Keep up the great work – especially the evangelism of calendar subscription.

cheers
Jamie

That’d be an interesting quid pro quo!

Of course it needn’t be either/or. There can be APIs and downloadable files. Arguably there should be, with a preference for the latter when data quantity is modest, and the former when it is vast.

Hopefully growing a useful resource will breed more ideas and then changes in the resource. Interestingly it’s probably a little easier for an unofficial skunkworks project to do this at least to begin with than the UN itself – since expectations on day one would be much higher.

The SQL example is a nice one – it would actually be interesting to add different query languages that people find useful – security and load issues we’d have to look at (plus the API runs on the Google AppEngine – i.e. Big Table, not MySQL).

Perhaps there should be one proviso though – asking for a feature means a commitment to use it :). That way we can fulfill Jon’s idea of co-evolution of the data and the Apps!

We’d be very happy to have feedback / comments / suggestions here in this thread or over at http://www.undata-api.org/

I always wonder whether the right answer is to get an API to a query interface to a database, or whether you’re better off with a snapshot and dump of a collection of raw data.

The API route sounds better, but it means that whatever development you do to pull things from a site starts with protocols and interfaces and software and queries and all sorts of things that are appealing to programmers but difficult for most everyone else.

The alternative, a simple data dump in some easy to parse file format, lets you figure out how the query structure looks like based on yor own data needs and gives you the opportunity either to restructure things for better efficiency or to apply much more primitive tools to do ad hoc queries.

My experience with this so far has been on a street tree database (nicknamed “EveryTree”) in Ann Arbor – there’s now a CSV file with about 50000 geocoded and identified trees, and I was able to get something useful out of it with tools as simple as “grep” in a small amount of time to get a sense for what was possible.

thanks

Ed

annarbor.com

Very interesting point. Historically that was unthinkable because of the fear that unthrottled query would impact services. But as databases move to the cloud there is renewed incentive to manage such access. I hope what you envision will come to pass.

Figures don’t lie, but liars figure.

Data portals often fail for the same reason that chart wizards fail — because they are constrained by pre-built casts, which data must be poured into.

But we already have a widely-understood data query language that is interactive and expressive: it’s called SQL.

If these data portals would merely allow end-users read-only SQL access to their underlying databases — they would be amazed at what innovative uses might emerge.