Providence,RI, population: 183,500
Boston, population: 4,467,000

Demo spreadsheet: http://spreadsheets.google.com/ccc?key=pq1fjlmNTLRo9yuyQAWZ_eg

You can choose what answer you want anyway – see http://googlesystem.blogspot.com/2009/03/alternative-answers-for-googlelookup.html

These queries still require exogenous knowledge that relates not to the data but to the way it is accessed. Python libraries are exogenous knowledge; they are simply there as a means to an end. The goal should be to absolutely minimize the need for additional such knowledge. Better syntax gets us part of the way there, but it still is incomplete. You still need to know which library and function. Python makes it almost as simple as possible, but it is as simple as possible within the domain of having code that is executed rather than question that is asked.

Computer literacy is not just about learning the limitations of the data, but also the seemingly arbitrary conventions of accessing it. We certainly need to learn better how to understand the limitations of various data. However, I don’t think the end game is teaching everyone basic scripting. It is a problem to me if everyone needs to know how to code in order to ask computers for things would not require code if you asked a knowledgeable human being.

That is different than asking computers for things that only computer are good at, such as lists of answers, sorted and filtered. But between sites like Yahoo pipes and open.dapper.com we are getting closer to data feed nirvana.

Yes. In this case the answer for these two queries:

=GoogleLookup(“Providence,RI”,”population”)
=GoogleLookup(“Boston”,”population”)

is the same value: 2000.

The source is pages like:

http://quickfacts.census.gov/qfd/states/25/2507000.html

where GoogleLookup finds lines of text like:

Population, 2000

Some examples:

To insert the number of Internet users in Paraguay:
=GoogleLookup(“Paraguay”; “internet users”)
To insert the Earned Run Average of Roger Clemens:
=GoogleLookup(“Roger Clemens”; “earned run average”)

See http://docs.google.com/support/bin/answer.py?answer=54199

Our URLs each point to a single time-series – they represent the value of a given quantity varying across time (plus any associated metadata). One of those pieces of metadata is the source (provenance) of the data – which for the case you highlighted is a spreadsheet from the European Central Bank containing the history of thirty-odd exchange rates.

So what we’re doing is taking a document and transforming it into several addressable resources, putting a documented API onto the resources thus created, and building tools around that.

Joe writes:
> I hope that those tools expose their
> innards so that if they turn out to not
> have captured the right standardizations,
> we can adapt them.

Our APIs are public and the data’s downloadable, so although we by necessity impose a particular form on the data in Timetric, we’d hope that ultimately it’s no more siloed than it was up on the European Central Bank’s website. Which form is more useful will more than likely depend on what one’s trying to do.

Jon wrote:
> Yes but time is a scarce resource, so a
> lot of questions go unasked/unanswered
> even by those who could do the fiddling.

Hopefully, the things services like ours can do – primarily through the API, in this context – help out in writing programs which give you the answers you’re after, and provide facilities which let non-programmers get results they couldn’t easily get otherwise. And as Joe writes, the boundaries between programmer and non-programmer are fuzzy; I used to be a computational physicist, and that’s a field full of people who write programs but mostly wouldn’t think of themselves as programmers. It’s something they (sometimes) do, rather than something they are.

This is, by the way, the very motivation and rationale that’s driving Oslo. It takes a hybrid approach to domain-specific languages, positing that people could actually create these for themselves, and then use them textually or graphically.

I’m basically talking about what Jon (and Jeannette Wing) have been framing around the idea of “computational literacy.” One of the reasons I’m enthusiastic about Python is that I think it is within reach of the “computationally literate” even if they don’t think of themselves as programmers.

I definitely laud the idea of building tools that standardize and guide people into doing things the right way — but I hope that those tools expose their innards so that if they turn out to not have captured the right standardizations, we can adapt them. I boggle at the kinds of things “non-programmers” have achieved using MS Excel and cell formulas, and I have to believe that more people would be able to learn to do those things if they could use a more humane language to do them.

Jon wrote:
> it does make me very curious to see what will happen when
> a substantial core of curated data winds up in the Wolfram
> silo with a very robust set of tools and services
> wrapped around it.

Good point. Perhaps Wolfram will end up as the Adobe of data tools and de facto standards…

We can only wait and see what the API will eventually enable. But initially, accepting input is a non-goal. Very counter-web-2.0, to be sure.

> Timetric

I had read about that here: http://seanmcgrath.blogspot.com/2009/03/take-look-at-timetric.html

My first thought was: Hmm. GeoCommons wants me to upload upload data so we can understand its spatial dimension. Timetric wants me to upload data so we can work with the temporal dimension. In many cases it’ll be the same data, but now analysis and curation will be happening in silos.

There’s no easy answer to this. But it does make me very curious to see what will happen when a substantial core of curated data winds up in the Wolfram silo with a very robust set of tools and services wrapped around it.

> permanent URLs for given bits of data

At what granularity? For example, is this URL:

http://timetric.com/series/K63mz40KTzy3TjGdyXEoVA/

your alias for the source:

http://www.ecb.int/stats/eurofxref/eurofxref-hist.zip

Or for your transformation of the source?

What is the atomic/versionable unit of reference? The datum? The row or column? The matrix? The collection?

> as programmers, we can solve most of
> the problems we think about, even if
> it’s annoying and fiddly

Yes but time is a scarce resource, so a lot of questions go unasked/unanswered even by those who could do the fiddling.

anyway, what I was saying: focussing on access neglects the possibility of making the data usful to as many people as possible once it’s there. That, too, needs work in a couple of different directions, I reckon: in the systems we use to help people find the data, and in the systems we use to let people transform and analyse it.

So I’m intrigued by Wolfram Alpha because it merges both these things into a natural-language-like interface: “find these facts, then do this bit of computation on them, and tell me the answer”. It’s the world’s weirdest DSL, if you want to look at it that way. But the downside is that it looks like it needs a lot of the knowledge “baked in” – how would you add your own facts?

The startup I’m working on, http://timetric.com/, takes a bit of a different approach. On the face of it, it’s less ambitious than what Wolfram and co. are aiming to do, because we’ve restricted the domain of facts we deal with we do numbers and their variation over time – time series, though you can always express constants as values which never change – and built an infrastructure with three important bits: permanent URLs for given bits of data, tools for finding and viewing data and uploading new series, and a calculation engine for deriving new time series from data already there.

One important thing we share with Wolfram Alpha, though, and which maybe speaks to your idea of data friction, is the idea that it’s about expressing the computations people want to do in a way they find immediately familiar. Achieving that means leveraging a language people already know. Wolfram Alpha’s using natural language; we’re using a formula language which looks like the one people know from spreadsheets. Both of these are accessible to non-programmers – any answer which starts with “learn Python” has to be the wrong one somehow. Take the geopy example above: as programmers, we can solve most of the problems we think about, even if it’s annoying and fiddly and involves writing altogether too many little parsers. But there are many more people who don’t program at all, and making data much more useful is going to mean writing tools for them.

What most people miss, though, is that there are two variables there, and one of them is who “you” is.
The implication of that is that there are two ways to make data more useful. Most of us have focussed on access – making the data more available, which is all well and good, but