Jon Udell

Having waxed theoretical about the Open Data Protocol (OData), it’s time to make things more concrete. I’ve been adding instrumentation to monitor the health and performance of my elmcity service. Now I’m using OData to feed the telemetry into Excel. It makes a nice end-to-end example, so let’s unpack it.

Data capture

The web and worker roles in my Azure service take periodic snapshots of a set of Windows performance counters, and store those to an Azure table. Although I could be using the recently-released Azure diagnostics API, I’d already come up with my own approach. I keep a list of the counters I want to measure in another Azure table, shown here in Cerebrata’s viewer/editor:

When you query an Azure table like this one, the records come back packaged as content elements within Atom entries:

<entry m:etag="W/"datetime'2010-02-09T00%3A00%3A53.7164253Z'"">
<id>http://elmcity.table.core.windows.net/monitor(PartitionKey='ProcessMonitor',
  RowKey='634012704503641218')</id>
<content type="application/xml">
<m:properties>
<d:PartitionKey>ProcessMonitor</d:PartitionKey>
<d:RowKey>634012704503641218</d:RowKey>
<d:HostName>RD00155D317B3F</d:HostName>
<d:ProcName>WaWorkerHost</d:ProcName>
<d:mem_available_mbytes m:type="Edm.Double">1320</d:mem_available_mbytes>
...snip...
<d:tcp_connections_established m:type="Edm.Double">24</d:tcp_connections_established>
</m:properties>
</content>
</entry>

This isn’t immediately obvious if you use the storage client libary that comes with the Azure SDK, which wraps an ADO.NET Data Services abstraction around the Azure table service. But if you peek under the covers using a tool like Eric Lawrence’s astonishingly capable Fiddler, you’ll see nothing but Atom entries. In order to get direct access to them, I don’t actually use the storage client library in the SDK, but instead use an alternate interface that exposes the underlying HTTP/REST machinery.

Exposing data services

If the Azure table service did not require special authentication, it would itself be an OData service that you could point any OData-aware client at. To fetch recent entries from my table of snapshots, for example, you could use this URL in any browser:

GET http://elmcity.table.core.windows.net/monitor?$filter=Timestamp+gt+datetime’2010-02-08′

(A table named ‘monitor’ is where the telemetry data are stored.)

The table service does require authentication, though, so in order to export data feeds I’m creating wrappers around selected queries. Until recently, I’ve always packaged the query response as a .NET List of Dictionaries. A record in an Azure table maps nicely to a Dictionary. Both are flexible bags of name/value pairs, and a Dictionary is easily consumed from both C# and IronPython.

To enable OData services I just added an alternate method that returns the raw response from an Azure table query. Then I extended the public namespace of my service, adding a /odata mapping that accepts URL parameters for the name of a table, and for the text of a query. I’m doing this in ASP.NET MVC, but there’s nothing special about the technique. If you were working in, say, Rails or Django, it would be just the same. You’d map out a piece of public namespace, and wire it to a parameterized service that returns Atom feeds.

Discovering data services

An OData-aware client can use an Atom service document to find out what feeds are available from a provider. The one I’m using looks kind of like this:

<?xml version='1.0' encoding='utf-8' standalone='yes'?>
<service xmlns:atom='http://www.w3.org/2005/Atom'
  xmlns:app='http://www.w3.org/2007/app' xmlns='http://www.w3.org/2007/app'>
  <workspace>
    <atom:title>elmcity odata feeds</atom:title>
    <collection href='http://elmcity.cloudapp.net/odata?table=monitor&hours_ago=48'>
      <atom:title>recent monitor data (web and worker roles)</atom:title>
    </collection>
    <collection href="http://elmcity.cloudapp.net/odata?table=monitor&hours_ago=48&
      query=ProcName eq 'WaWebHost'">
      <atom:title>recent monitor data (web roles)</atom:title>
    </collection>
    <collection href="http://elmcity.cloudapp.net/odata?table=monitor&hours_ago=48&
         query=ProcName eq 'WaWorkerHost'">
      <atom:title>recent monitor data (worker roles)</atom:title>
    </collection>
    <collection href="http://elmcity.cloudapp.net/odata?table=counters">
      <atom:title>peformance counters</atom:title>
    </collection>
  </workspace>
</service>

PowerPivot is an Excel add-in that knows about this stuff. Here’s a picture of PowerPivot discovering those feeds:

It’s straightforward for any application or service, written in any language, running in any environment, to enable this kind of discovery.

Using data services

In my case, PowerPivot — which is an add-in that brings some nice business intelligence capability to Excel — makes a good consumer of my data services. Here are some charts that slice my service’s request execution times in a couple of different ways:

Again, it’s straightforward for any application or service, written in any language, running in any environment, to do this kind of thing. It’s all just Atom feeds with data-describing payloads. There’s nothing special about it, which is the whole point. If things pan out as I hope, we’ll have a cornucopia of OData feeds — from our banks, from our Internet service providers, from our governments, and from every other source that currently publishes data on paper, or in less useful electronic formats like PDF and HTML. And we’ll have a variety of OData clients, on mobile devices and on our desktops and in the cloud, that enable us to work with those data feeds.

Linda Stone, coiner of the marvelous phrase continuous partial attention, has lately been exploring another modern pathology she calls email apnea, which means failure to breathe while checking email. In retrospect, we shouldn’t be surprised. Look:

• The new 25-payline special edition of Wheel of Wealth will have you holding your breath in excitement…

• Play Online Slot Machine Game. Coin in – spin – hold your breath……Watch those symbols…..Will it or won’t it?

• After the first two hits you’re holding your breath for the third reel…

We don’t talk about slot-machine apnea but it’s the same syndrome, produced by the same cause: an intermittent, or variable-interval, schedule of reinforcement. Any activity that exhibits this pattern will be powerfully addictive. A dog begging for scraps of food at the table, rewarded only once in a thousand times, will always beg. Likewise a human begging for scraps of attention.

The link between variable-interval reinforcement and email addiction is well known. Less studied is how this plays out in other modes of electronic discourse. The architecture of those modes introduces another key variable: attention payoff. In a group-structured system, like email or Facebook, the payoff is bounded by group size. It’s true that email messages can escape and go viral, but when that happens the attention payoff is never the kind you want.

But in open pub/sub systems, like blogs or Twitter, the payoff is unlimited. Any item that you post could attract worldwide attention, boost your reputation, land you a job, or make a key personal or professional connection. However there’s no guarantee that you’ll get any reinforcement at all. So some fall by the wayside, others become addicted.

“Technology is here to stay,” Linda says. “Can our relationship to it change?”

It must, it can, and it will. But we’ll need to develop some intuitions about global scale and connectedness for which evolution did not prepare us. And then we’ll need to translate them back down to the human scale. Evolution has taught us how to be social. Technology amplifies our ability to give and receive attention, but it doesn’t change the rules of the game. There’s a time to listen, a time to talk, a time to breathe. We’ll remember, and we’ll figure it out.

My guest for this week’s Innovators show is Sal Khan. He’s the creator of http://khanacademy.org, a catalog of more than 1000 YouTube video lessons in math, physics, biology, chemistry, and economics. All of these videos are made by Sal himself, in an engagingly personal style, using simple screencasting tools.

When I first got interested in screencasting, I envisioned the medium not only as a way to demonstrate software, but also as a way to share knowledge at Internet scale. Sal’s work fulfills that vision, and points the way toward a profound and much-needed disruption of our educational system.

At its core, Sal’s project isn’t about YouTube screencasts. It’s about intuition.

I always got frustrated by what went on in the classroom. You see otherwise intelligent peers memorizing facts and not really caring about the actual intuition. And because they didn’t care about the intution in their junior year, when that same idea pops up in senior year, it’s like they’ve never seen it before. It boggled my mind. You’re just relabeling the same concept over and over.

Sal cares about the intuition, and he wants others to care about the intution too. The first beneficiary of that desire was his cousin Nadia, whom he tutored remotely. Then followed other cousins and family friends. Then it dawned on him that there were no limits. The project could scale out. He could become a superempowered individual, reaching anyone who finds value in his method.

One of the key ingredients of that method is improvisation. These videos aren’t carefully planned, and they aren’t edited. As a viewer, you find yourself looking over the shoulder of a smart and broadly knowledgeable person who is solving problems by thinking on his feet. You watch a practitioner at work: engaged with his medium, wrestling with his tools, correcting false starts.

It was Chris Gemignani who first showed me the value of this approach, in a screencast that teaches how to do unexpectedly powerful and elegant Excel charting. He did it in one take. I’d have been tempted to edit out the false starts. But Chris knew better. Learning how a practitioner really thinks about solving a problem is even more valuable than learning the solution to the problem.

One thing that Sal’s lessons can’t be, of course, is interactive. Nor does he pretend that these videos will make teachers obsolete. But he does suggest, and I violently agree, that teachers can and should become curators of online assets like the ones Sal is creating, and should know when and how to weave those assets into their classes.

Teachers should also become connectors. Sal won’t be the only game in town. Other superempowered tutors will emerge. Each will have a unique style. For a given student, a given subject, and a given problem, one or another of those styles may be right. The best teachers will know their own strengths and limitations, will know which online tutors complement their strengths in a variety of ways, and will connect their students with those tutors.

Sal Khan is on fire. He burns with a passion to share his intuitions with anyone and everyone. It is a beautiful thing to see. He has abandoned a lucrative career in finance to do this fulltime, and I am quite sure he will find a way to keep doing it.

PS: The title of this piece refers to Richard Ankrom’s Los Angeles freeway project. At a busy intersection, millions of motorists have been directed to North 5 by a sign that Caltrans omitted. Ankrom created and installed that missing sign.

PPS: I wrote to my son’s math teacher about Sal Khan. She replied: “Thanks for that link to the Khan Academy. I was overwhelmed by how many video lessons he has! He does seem like an inspiring man. Unfortunately, You Tube is blocked here at the high school.”

OData, the Open Data Protocol, is described at odata.org:

The Open Data Protocol (OData) is a web protocol for querying and updating data. OData applies web technologies such as HTTP, Atom Publishing Protocol (AtomPub) and JSON to provide access to information from a variety of applications, services, and stores.

The other day, Pablo Castro wrote an excellent post explaining how developers can implement aspects of the modular OData spec, and outlining some benefits that accrue from each. One of the aspects is query, and Pablo gives this example:

http://ogdi.cloudapp.net/v1/dc/BankLocations?$filter=zipcode eq 20007

One benefit for exposing query to developers, Pablo says, is:

Developers using the Data Services client for .NET would be able to use LINQ against your service, at least for the operators that map to the query options you implemented.

I’d like to suggest that there’s a huge benefit for users as well. Consider Pablo’s example, based on some Washington, DC datasets published using the Open Government Data Initiative toolkit. Let’s look at one of those datasets, BankLocations, through the lens of Excel 2010’s PowerPivot.

PowerPivot adds heavy-duty business analytics to Excel in ways I’m not really qualified to discuss, but for my purposes here that’s beside the point. I’m just using it to show what it can be like, from a user’s perspective, to point an OData-aware client, which could be any desktop or web application, at an OData source, which could be provided by any backend service.

In this case, I pointed PowerPivot at the following URL:

http://ogdi.cloudapp.net/v1/dc/BankLocations

I previewed the Atom feed, selected a subset of the columns, and imported them into a pivot table. I used slicers to help visualize the zipcodes associated with each bank. And I wound up with a view which reports that there are three branches of WashingtonFirst Bank in DC, at three addresses, in two zipcodes.

If I were to name this worksheet, I’d call it WashingonFirst Bank branches in DC. But it has another kind of name, one that’s independent of the user who makes such a view, and of the application used to make it. Here is that other name:

http://ogdi.cloudapp.net/v1/dc/BankLocations?$filter=name eq ‘WashingtonFirst Bank’

If you and I want to have a conversation about banks in Washington, DC, and if we agree that this dataset is an authoritative list of them, then we — and anyone else who cares about this stuff — can converse using a language in which phrases like ‘WashingtonFirst Bank branches in DC’ or ‘banks in zipcode 20007′ are well defined.

If we incorporate this kind of fully articulated web namespace into public online discourse, then others can engage with it too. Suppose, to take just one small example, I find what I think is an error in the dataset. Maybe I think one of the branch addresses is wrong. Or maybe I want to associate some extra information with the address. Today, the way things usually work, I’d visit the source website and look for some kind of feedback mechanism. If there is one, and if I’m willing to provide my feedback in a form it will accept, and if my feedback is accepted, then my effort to engage with that dataset will be successful. But that’s a lot of ifs.

When public datasets provide fully articulated web namespaces, though, things can happen in a more loosely coupled way. I can post my feedback anywhere — for example, right here on this blog. If I have something to say about the WashingtonFirst branch at 1500 K Street, NW, I can refer to it using an URL: 1500 K Street, NW.

That URL is, in effect, a trackback that points to one record in the dataset.¹ The service that hosts the dataset could scan the web for these inbound links and, if desired, reflect them back to its users. Or any other service could do the same. Discourse about the dataset can grow online in a decentralized way. The publisher need not explicitly support, maintain, or be liable for that discourse. But it can be discovered and aggregated by any interested party.

The open data movement, in government and elsewhere, aims to help people engage with and participate in processes represented by the data. When you publish data in a fully articulated way, you build a framework for engagement, a trellis for participation. This is a huge opportunity, and it’s what most excites me about OData.

¹ PowerPivot doesn’t currently expose that URL, but it could, and so could any other OData-aware application.

I’m listening to the audio version of a very cool talk given by astronaut-turned-artist Alan Bean. (Skip the hokey intro, though, and jump in at minute 7 when he starts.)

He tells great stories about the space program, but also offers wider perspectives on life, art, and human potential.

Along the way, he tells an amusing anecdote about the famous picture of Neil Armstrong planting an American flag onto the moon’s surface. Armstrong told Bean it had been a scary moment, and Bean asked why. Armstrong said (as paraphrased by Bean):

Well, I couldn’t get that flag into the ground, like in training. Up there, those particles in the dirt aren’t rounded like regular sand. On Earth I would just do like that, and it would go in. But up there I did like that and it didn’t go in.

I imagined that when I let go, it would fall into the dirt, and people all over the world would see the American flag fall into the dirt. So I tipped it back until the center of gravity was over the hole. Then I put a little dirt around it. I knew that if I could get it balanced, and get away from it, that without any wind it would stay balanced. So that’s what we did. We got away from it, and we never got close to it again.

Bean adds: “It probably blew over when they launched, but it didn’t make any difference. That’s an engineer’s solution!”

What a great hack!

Today on a conference call I was reminded of another. A few years ago, in an airport, I saw a guy with a cellphone in one hand and a payphone in the other. His ear, brain, and mouth were trying to bridge two phone networks together, it wasn’t working well, and he was visibly frustrated. Finally he removed his head from between the two phones, stuck them together, and reversed them earphone-to-microphone, so the two parties were talking directly to each other.

My conference call today presented a different version of that scenario. It was scheduled as a VOIP call, then was switched to a POTS call, but not everybody got the memo. So I made the POTS call. And since I have a podcast rig that lets me do POTS calls through my computer, using the same headset I use for VOIP, I made the call that way.

Then people started to show up on both the POTS side and the VOIP side. I realized that, unexpectedly, I was hearing both sides and they were hearing me. Both were being conveyed through my computer’s audio subsystem. I was just like the guy with the cellphone on one ear and the payphone on the other.

It would have been cool to do the same kind of earphone-to-microphone hack. But before I got the chance to try, the VOIP folks hung up and dialed back in on the POTS side.

Oh well, maybe next time.

The sound track for yesterday’s run was a talk by primatologist Richard Wrangham¹, author of Catching fire: how cooking made us human. Cooking, he says, has long been thought to be an optional cultural practice, like wearing jewelry. But really, he argues, cooking was the essential technological innovation that enabled us to produce the metabolic energy we needed to become human.

How? Cooked food is more digestible than raw food. And not just by a little, but by a lot. Learn how to control fire, use it to cook your food, and you free up extra energy — plus time that would otherwise be spent masticating. Spend that time hunting, and your metabolic equation gets even better.

Wrangham has fascinating things to say about how this surplus time and energy explains such cultural universals (or former universals) as marriage, sexual division of labor, and the family dinner. Whether you agree or disagree with this analysis, though, it’s supported by an attention-grabbing claim. Everything we thought we knew about absorption of energy from food is wrong.

To this day, Wrangham says, the USDA website² publishes tables that make no distinction between the nutritional value of cooked and raw food. On this page, for example, the energy content of one large raw egg is given as 75 kcal. The value for one large hard-boiled egg is almost the same: 78 kcal.

This is wrong, Wrangham says. A cooked egg delivers way more energy than a raw egg. How could this be? And how could we not know it?³

Here’s the explanation. We have traditionally measured the energy content of food by comparing input (the food we eat) and output (the feces we excrete). Burn both in a calorimeter, subtract, and the difference is the energy that was extracted from the food.

Yes, but extracted by whom? Or rather, by what? The energy that we humans take from our food has almost all been extracted by the time it reaches the end of the small intestine. But it has a long way to go yet. It must also pass through the large intestine, where dwell a myriad of gut flora. And they, Wrangham says, are hungry. If you eat a raw banana you only get some of its energy, and they get most of the remainder. If you eat a cooked banana, though, you get a lot more of its energy and leave less for them. The end result looks the same, but the internal distribution is quite different.

So you need to compare the energy in food entering the mouth to the energy remaining in the digestive products leaving the small intestine.⁴ Only then does the dramatic difference between the energies we get from raw versus cooked food become evident.

This is a great parable about instrumentation, measurement, knowledge, and epistemology. What other profound errors of basic understanding arise from misplaced instrumentation? And what might we learn by making simple — and in retrospect obvious — adjustments?

¹ Yet another podcast from KUOW’s Speakers’ Forum, which has become one of my most reliable sources of audio brain food.

² A sad reminder that government website and chamber of horrors are still, too often, synonymous.

³ The error, if it is indeed an error, propagates to WolframAlpha, which sources the USDA data. Compare 100 g of raw egg to 100 g of cooked egg.

⁴ How do you tap in at that point? Recruit people who have had ileostomies.

If you’re interested in the use of computers and networks to support collaboration, you’ll have heard of PLATO. It was an early courseware system, and by early I mean circa 1960, running on vacuum tubes. But it was also a petri dish in which much of what we now know as online culture first evolved.

I’ve long known that PLATO inspired many other systems, including VAX Notes and Lotus Notes. But I never heard the backstory. So when I found out that Brian Dear is completing a history of PLATO, and planning a conference to commemorate its 50th anniversary, I invited him onto my weekly show to find out more about it. PLATO matters, Brian says, because

it challenges our assumptions of how the online world evolved. It rewrites the history. It’s as if we discovered Wilbur and Orville Wright were not the first to fly a powered plane — that it’d been done faster and longer with a jet aircraft 30 years earlier.

Of couse the same can be said of other early technologies, notably Smalltalk, which introduced ideas and methods that are only now hitting the mainstream. It’s fun to wax nostalgic, but I’d rather explore how these systems arose, why they flourished, and what accounts for the propagation of their memes but not their genes.

From that perspective Brian reminds us, first, that PLATO was expensive. Few universities were willing or able to invest millions in a Control Data mainframe and a fleet of gas-plasma flat-panel bitmapped touch-screen display terminals. Those terminals enabled some extraordinary things, like the interactive music software that captivated Brian as a University of Delaware undergrad. They also enabled a now-extinct species of emoticons, which relied on the bitmapped graphics. But since much of what became PLATO’s essential DNA required only character-mapped graphics, those expensive bitmapped screens became an evolutionary bottleneck.

Another feature that didn’t pass through that bottleneck was PLATO’s ability to make sense of natural language input. Many thousands of programmer hours were invested in enabling PLATO to recognize a variety of human utterances. That in turn enabled courseware authors to create lessons that responded intelligently — and, Brian says, in ways that are sadly still not typical of modern courseware.

Today we can attack that problem by creating open source libraries, by reusing them, and by extending them. That’s a great way to create DNA that can propagate. But it’s useful to consider why it might not. We still, for the most part, create dependencies on specific programming languages, and on the environments in which they run.

As we move into an era of services, though, we can start to imagine a more fluid environment in which capabilities persist across language and system boundaries. Consider this exhibit from an antique PLATO library:

This is a screenshot from the live PLATO system running (in emulation) at cyber1.org. It’s a page from the catalog of functions in PLATO’s CYBIS library. Shown here are some of the methods available to process responses to questions.

Some of those methods might still be useful. And if they’d been packaged in a language- and system-independent way, some might conceivably still be in use.

PLATO programmers didn’t have the option to package their work in a such a way. Now we’re on the cusp of an era in which these kinds of library services can also be language- and system-independent web services. Will we exploit this new possibility? Will some of today’s core services still be delivering value decades from now, freeing developers to add value farther up the stack? It’s worth pondering.

A while back I reviewed the reading machine that my mom, who suffers from macular degeneration, now depends on. I gave it a thumbs up, but also noted that she was having some problems.

On my last visit I came up with a method that will help, if she can get the hang of it. The method is non-obvious, and isn’t documented anywhere I’ve been able to find, so I made a short movie to illustrate it.

The key insights are:

Use the left margin screw to set a left margin somewhere

It almost doesn’t matter where, you just need a guide for carriage returns.

Position the book and the tray

Getting this right makes a huge difference. My mom was constantly fiddling with the position of the book on the tray. This frustrated her, and seriously impaired her ability to read fluidly.

But if you position the tray correctly, and the book relative to the tray, then you can easily read the whole page without touching or moving the book at all. Here’s how:

Align the bottom left corner of the book with the bottom left corner of the screen.

This is counter-intuitive. The natural expectation is to start at the top of the page. And you do want to start reading there. But I found that establishing a bottom margin is a crucial first maneuver, and it involves three steps:

1 Push tray all the way forward and rightward

2. Place book on tray

3. Move book to align bottom left corner of page with bottom left corner of screen

With the tray still as far forward and as far right as it will go, you have defined both a left margin and a bottom margin for the page. Now read the whole page without touching the book again. Here’s how:

Find the top of the page.

To do that you pull the tray out (forward, towards yourself) until the top margin of the page lines up with the top of the screen.

Read as many lines vertically as the screen can display.

Use only a two-stroke left/right motion of the tray. The sequence is:

1. Slide tray left to reveal ends of lines

2. Slide tray right for carriage return

My mom had been advancing the tray (by pushing it in) once per line. This wastes effort and disrupts context. If the left margin screw is set, a carriage return always goes to the same place. So it was easy — at least for her — to make a visual connection from the end of the previous line to the beginning of the next one.

I realize this part may not work for everyone, and maybe not even for her as her vision worsens. Right now, at her magnification, her screen can display 8 or 10 lines. At higher magnification, when only a few are visible, there will be less context to help make that connection. Then it may become necessary to scroll vertically once per line. But the longer that can be avoided, the better.

Why was this necessary?

Shouldn’t multi-thousand-dollar gizmos like this come with training materials that help people figure this stuff out? Yes, but I’ve given up being shocked that they don’t.

If you’ve got a friend or relative in the same boat, let me know if this writeup — and/or the accompanying video — makes sense.

A note on making the movie

The video combines slides with a side-by-side animation of the tray and the screen. I wound up using PowerPoint, which conveniently handles the three ingredients: text, bitmap graphics, and vector graphics.

Rather than use PowerPoint’s animation features, though, I made a sequence of frames, nudging objects by small increments from frame to frame. This turned out to be a surprisingly easy and approachable technique.

Then I turned on a screen recorder — I used Camtasia, but it could have been any other — and stepped through the frames.

On this week’s podcast, Greg Wilson tells the story of a university course he created, and has taught for many years, called Software Carpentry. I have known Greg for a long time. We are kindred spirits in several ways. Most notably, we like to mine veins of knowledge, experience, and technique that some practitioners take for granted, but that many others haven’t yet discovered — or don’t yet use as well as they could.

I, for example, wonder why we don’t teach everyone basic principles of structured information, namespace design, and syndication. Greg, similarly, wonders why student programmers — and student scientists whose careers increasingly depend on computational methods — are not taught basic principles of version control, debugging, and refactoring. And why we don’t read great software in the same way we read great literature or study landmark scientific experiments. And why the controlled reproducibility of commercial software development isn’t typical of computational science.

If you care about these issues, there are two ways you can help. First, take a look at the reboot of the Software Carpentry course that Greg’s experience has led him to propose. Second, help him find the funding to keep doing this work.

On FiveThirtyEight.com the other day, Andrew Gelman posted this chart illustrating the high cost of US health care:

He did so to correct a “somewhat misleading (in my opinion) presentation of these numbers [that] has been floating around on the web recently.” The misleading graph, which appeared on a National Geographic blog, was — I agree — a confusing way to show information better represented in a scatterplot.

But I’ve seen this data before, and there’s more to the story. Neither the National Geographic nor FiveThirtyEight has anything to say about which numbers they’re charting.

Back in 2005, in a review of John Abramson’s excellent book Overdo$ed America, I noted that he had used a different source to reach a slightly different conclusion.

His chart, based on OECD health-expenditure data (link now 404) and WHO healthy life expectancy data (link still alive), looked like this:

He used it to make the oft-cited point that US healthcare isn’t just wildly expensive, but that it also correlates with worse life expectancy than in many countries that spend less.

I wondered what the chart would look like if based on the same OECD expenditure data but on the OECD’s rather than the WHO’s definition of life expectancy. The result looked like this:

The U.S. is the clear cost outlier on both charts. The first chart, however, places us near the low end of the life expectancy range, justifying Abramson’s assertion that we combine “poor health and high costs.” The second chart places us near the high end of the life expectancy range, suggesting that while value still isn’t proportional to cost, we’re at least buying more value than the first chart indicates.

Although based on older data, this second chart closely resembles the ones recently shown and discussed by the National Geographic and FiveThirtyEight.

My review of Abramson’s book concluded:

Has Abramson spun the data to make his point, just as he accuses the pharmaceutical industry of doing? Of course. Everybody spins the data. What matters is that:

• Everybody can access the source data, as we can in the case of Abramson’s book but cannot (he argues) in the case of much medical research
• The interpretation used to drive policy expresses the values shared by the citizenry

Would we generally agree that we should measure the value of our health care in terms of healthy life expectancy, not raw life expectancy? That the WHO’s way of assessing healthy life expectancy is valid? These are kinds of questions that citizens have not been able to address easily or effectively. Pushing the data and surrounding discussion into the blogosphere is the best way — arguably the only way — to change that.

That was five years ago. The data was, and is, out there. So it’s disheartening to see the same chart pop up again without any further discussion of the sources of its data, or of the definitions underlying those sources.

On this week’s Innovators show, Doug Day joins me to discuss the new iCalendar validator he has recently deployed on Azure.

The project draws inspiration from the pathbreaking RSS/Atom feed validator originally created by Mark Pilgrim and Sam Ruby. The RSS/Atom validator’s test-driven and advice-oriented approach is exemplary, and the iCalendar validator follows in its footsteps.

The tests, in this case, are iCalendar snippets that are, or are not, valid according to the spec. These snippets, packaged into XML files, form a library of examples that does not depend on the programming language used to run the tests. So although Doug’s validator, based on his open source parser, is written in C#, another validator written in Java or Python or Ruby could use the same test suite.

The advice offered is minimal so far, but I hope will expand as the test suite grows. Sam Ruby observes:

Identifying real issues that prevent real feeds from being consumed by real consumers and describing the issue in terms that makes sense to the producer is what most would call value.

In that spirit, I am gathering examples of calendars in the wild and looking for ways to help Doug add value.

In the podcast we discuss a nice example that came up recently in the curators’ room of the elmcity project. A custom-built calendar contained events (VEVENT components, in iCalendar-speak) with no start or end times (DTSTART and DTEND properties). This, it turns out, is not prohibited by the spec. But reporting no error is unhelpful. The author of the calendar — or of the software that produced the calendar — ought to be warned that such a calendar won’t yield a useful or expected result.

Why would anyone produce such a calendar in the first place? This harkens back to the early days of RSS. Many of us found that we could craft simple ad-hoc feeds in order to leverage RSS as a lightweight data exchange. It was liberating to be able to do that. But hand-crafted feeds, or feeds written by hand-crafted software, were valuable only to the extent they would reliably interoperate. Often they would not. The feed validator, by showing what was wrong with these feeds, and explaining why and how to fix them, was a powerful ally for those of us trying to bootstrap a feed ecosystem.

The iCalendar validator has a long way to go yet. But the road ahead is well lit, and I’m grateful to Doug Day for resolving to travel it.

The other day I listened to a Spark (CBC Radio) interview with Larry Lessig about his New Republic essay Against Transparency, which begins:

We are not thinking critically enough about where and when transparency works, and where and when it may lead to confusion, or to worse. And I fear that the inevitable success of this movement–if pursued alone, without any sensitivity to the full complexity of the idea of perfect openness–will inspire not reform, but disgust. The “naked transparency movement,” as I will call it here, is not going to inspire change. It will simply push any faith in our political system over the cliff.

The essay was published in October 2009. In this interview from November, Prof. Lessig reflected on the reactions that it provoked. Although the delicious and bitly feedback now suggests that most people understood the essay to be a thoughtfully nuanced critique, there were evidently some early responders who read it as a retreat from openness and an assault on the Internet.

I’m glad I missed the essay when it first appeared. Reading it along with a cloud of feedback from readers and from the author amplifies one of the key points: We don’t really want naked transparency, we want transparency clothed in context.

The Net can be an engine for context assembly, a wonderful phrase I picked up years ago from Jack Ozzie and echoed in several essays. But it can also be a context destroyer.

In the interview, Lessig notes one example of context destruction. The article, which most people will read online, spans eleven pages, each of which wraps its nugget of “content” in layers of distraction. Some early negative comments, Lessig says, came from people who had clearly not read to the end.

Our increasingly compressed and fragmented attention can also be a context destroyer:

What about when the claims are neither true nor false? Or worse, when the claims actually require more than the 140 characters in a tweet?

This is the problem of attention-span. To understand something–an essay, an argument, a proof of innocence– requires a certain amount of attention. But on many issues, the average, or even rational, amount of attention given to understand many of these correlations, and their defamatory implications, is almost always less than the amount of time required. The result is a systemic misunderstanding–at least if the story is reported in a context, or in a manner, that does not neutralize such misunderstanding. The listing and correlating of data hardly qualifies as such a context. Understanding how and why some stories will be understood, or not understood, provides the key to grasping what is wrong with the tyranny of transparency.

Transparency is a necessary but not a sufficient condition. Recently my town’s crime data and council meetings have appeared online. But this remarkable transparency does not alone enable the sort of collaborative sense-making that we all rightly envision.

In the case of crime data, we require a context that includes historical trends, regional and national comparisons, guidance from government about how its local taxonomy relates to regional and national taxonomies, and reporting by newspapers and citizens.

In the case of city council meetings, we require a context that includes relevant state law and local code, and reporting by stakeholders, by newspapers, and by affected citizens.

To enable context assembly, we’ll need to organize the numeric and narrative data produced by the “naked transparency” movement in ways friendly to linking, aggregation, and discovery.

But these principles will need to be adopted more broadly than by governments alone. Everyone needs to understand the principles of linking, aggregation, and discovery, so that everyone can help create the context we crave.

Borrowing Bruce Schneier’s wonderful term security theater, Rohit Khare has written about privacy theater. Not to be outdone, here’s a letter to my local newspaper about carbon theater.

To: Editors
Re: Carbon challenge in home stretch

We love our sports rivalries, and the classic contest between Keene and Portsmouth has riveted me to my sofa. Let’s recap. Back in April, seacoastonline.com (http://www.seacoastonline.com/articles/20090423-NEWS-904230413) reported:

Municipal employees in Portsmouth and Keene, the state’s two predominant ‘green’ cities, slugged it out over the course of three weeks and, in the end, Keene delivered the knockout punch.

This week, the Sentinel and the Portsmouth Herald advanced the story of this “carbon-busting throwdown” in a joint communique (http://keenesentinel.com/articles/2009/12/29/news/local/free/id_384393.txt):

Garry Dow of Clean Air-Cool Planet, which manages the carbon challenge, said the scales are tipped in Portsmouth’s favor in the second phase, which involves the number of residents in each city to sign up for the challenge.

The challenge? Check it out at http://necarbonchallenge.org/calculator.jsp. There you will find an online form that reminds you to tighten up your house, use compact fluorescent lights, air-dry your dishes, and recycle.

Back in April, more of Keene’s city employees took the survey than Portsmouth’s. But now, in phase two of the carbon-busting throwdown, Portsmouthians are taking the survey at a higher rate than Keeners.

Across New England, according to the Carbon Challenge website, this slugfest has reduced C02 emissions by over 17 million pounds. That’s nothing to sneeze at. It’s two thirds of New York City’s daily waste stream, a third of the mass of the Titanic, a fifth of the C02 produced by the recent Copenhagen conference.

Except, of course, none of the combatants has actually reduced their C02 emissions. They’ve only take an online survey, and pledged to do all sorts of things that might or might not get done.

I’d like to propose a different challenge. Let’s focus on one thing and really get it done. For example, what if every leaky window in Keene were equipped with an interior storm? John Leeke, who runs Historic HomeWorks in Portland, invented this cheap, appropriate, and effective technology. On his website (http://www.historichomeworks.com/forum/viewtopic.php?t=193) he shows how to build interior storms.

I’ve done this, and it’s a vast improvement over the stick-on window kits I’ve used in previous years. Interior storms are just cheap wooden frames with gaskets around the outside and shrink-wrap plastic facing. They press-fit into your window frames from the inside. You get all the benefits of the stick-on kits: zero air infiltration, a second layer of dead air. And there are none of the drawbacks: awkward yearly installation, destructive yearly removal.

“Keene’s down in the standings,” the Sentinel/Herald article says, “but there’s still plenty of time for residents to take the online survey and boost the city’s chance to take home the green prize.”

Well, OK, but I’d like to see Keene define — and then win — a different prize. What if we become the first city to outfit every leaky window with interior storms? And what if we create jobs while doing so? That would be something worth shouting about.

Recently my town has adopted two innovative web services that I’ve featured on my podcast: CrimeReports.com, which does what its name suggests, and Granicus.com, which delivers video of city council meetings along with synchronized documents.

You can see the Keene instance of CrimeReports here, and our Granicus instance here.

I’m delighted to finally become a user of these systems that I’ve advocated for, written about, and podcasted. I’m also eager to move forward. We’re still only scratching the surface of what Net-mediated democracy can and should become.

In the case of CrimeReports, the next step is clear: Publish the data. It’s nice to see pushpins on a map, but when you’re trying to answer questions — like “Are we having a crime wave?” — you need access to the information that drives the map. Greg Whisenant, the founder of CrimeReports.com, says he’d be happy to publish feeds. But so far the cities that hire him to do canned visualizations of crime data aren’t asking him to do so, because most people aren’t yet asking their city governments to provide source data. So a few intrepid hackers, like Ben Caulfield here in Keene, are reverse-engineering PDF files to get at the information. Check out Ben’s remixed police blotter — it’s awesome. Now imagine what Ben might accomplish if he hadn’t needed to move mountains to uncover the data.

In the case of Granicus, I’m reminded of this item from last year: Net-enhanced democracy: Amazing progress, solvable challenges. The gist of that item was that:

• It’s amazing to be able to observe the processes of government.

• It’s still a challenge to make sense of them.

• Tools that we know how to build and use can help us meet that challenge.

Check out, for example, last week’s Keene city council meeting. Scroll down to an item labeled 2. Ordinance O-2009-21. In this clip, the council agrees to amend the city code for residential real estate tax exemptions. I wish I could link you directly to that portion of the video, which begins at 34:11, in the same way that I can link you to the associated document. But more broadly, I wish that a citizen who tunes in could understand — and help establish — the context for this amendment.

Here’s the new language:

Sec. 86-29 Residential real estate tax exemptions and credits

With regard to property tax exemptions, the city hereby adopts the provisions of RSA 72:37 (Blind); RSA 72:37-b (Disabled); RSA 72:38-b (Deaf or Severely Hearing Impaired); RSA 72:39-a (Elderly); RSA 72:62 (Solar); RSA 72:66 (Wind); and RSA 72:70 (Wood).

With regard to property tax credits, the city hereby adopts the provisions of RSA 72:28, II, (Optional Veterans’ tax credit); RSA 72:29-a , II, (Surviving Spouse); and RSA 72:35, I-a, (Optional Tax Credit for Service-Connected total disability).

In this case, I just happen to know a bit of this amendment’s backstory. Earlier this year I found out — only thanks to a serendipitous encounter with a city councilor at a social event — that my wood gasifier qualified me for an exemption. This was the first such exemption, and to my knowledge is still the only one granted.

If I hadn’t gone through that experience, though, the video clip and its associated document would mean nothing to me. There would be no way to make a connection between state law on the one hand, and a documented case study on the other.

On the next turn of the crank, I hope that services like Granicus will enable us to make those connections. Seeing the process of government in action is a great step forward. Now we need to be able to use links and annotations to help one another make sense of that process.

My guest for this week’s Innovators show is my old BYTE pal Howard Eglowstein. Nowadays he’s working for freewatt, a residential micro-CHP (combined heat and power) system, and our conversation revolved partly around that technology.

But I also invited Howard to reflect on the cultural phenomenon that’s celebrated in the pages of Make. Hacking at the intersection of atoms and bits is nothing new for Howard, he’s been doing it his whole career. One of his epic projects was Thumper, a machine he built for the BYTE lab to test the battery life of notebook computers. Thumper used optical sensors to notice when power-saving features kicked in, and robotic fingers to defeat them by pressing keys. (I resurrected this article about Thumper from the (now-abandoned) BYTE archive.)

It makes perfect sense for Howard to be deploying his hybrid skillset in the realm of energy innovation. But why, I’ve wondered lately, did we devalue those skills and inclinations? Why the long lull between the heydays of Popular Mechanics and Make? Here are some of Howard’s observations:

On toys, cars, and patents:

My background is in electronic toys. Toy engineers know how to take a really cool concept and make it cheap. In the 80s — not so much any more — you could get something at the toy store, open it up with a Dremel, and make it do something different. That’s how some kids get their first taste of reverse engineering.

But how many of us can fix our cars anymore? You just can’t. Even car people don’t have the tool and the documentation. A lot of things are done better than everybody else, and they’re secret.

In the toy industry we never patented anything, there was no point. If you patent something you have to tell everybody how it works, and then they have what they need to make an improvement and then steal the idea from you. So you do something amazing and cool, you wow everybody, and by the time they figure out what you’ve done you’ve moved on to something else which is even cooler.

On a friend’s son who is a Make fan:

We’ve really encouraged people to absorb information. But that gets boring after a while. You browse the computer, it’s kind of fun to click on links and see where they go, but it gets old. Meanwhile we’ve got a lot of kids who, let’s face it, probably aren’t going to get together and throw a football around, they’d rather play video games. So in this kid’s case when he gets tired of looking at stuff he goes and builds stuff. I hope that we’re encouraging more people to do that.

After speaking with Howard I was reminded of one project that is providing that encouragement: Natalie Jeremijenko’s feral robotic dogs, which are “upgraded commercially robotic dog toys that have been transformed into activist instruments to find and display urban pollutants.”

So I guess the toy business still is giving some young people their first taste of reverse engineering!

One of the themes I’ve been exploring for the past few years is computational thinking. It’s an evocative phrase that has led me in a few different directions. One is my intentional use of tagging and syndication as key strategies for social information management. Another is my growing interest in the kinds of uses of WolframAlpha outlined in Kill-A-Watt, WolframAlpha, and the itemized electric bill.

A lot of what I’ve read and heard about WolframAlpha seems to focus on its encyclopedic nature. But it aims to be a compendium of computable knowledge, and as such I think its highest and best use will be to enable computational thinking.

Here’s one small but telling example from my Kill-A-Watt essay:

Q: 9 W * (30 * 24 hours)

A: About half the energy released by combustion of one kilogram of gasoline.

Q: ( 1 kilogram / density of gasoline ) / 2

A: Less than a fifth of a gallon.

I was trying to understand what 9 Watts, over the course of a month, means. WA offered the comparison to the amount of energy in gasoline, but reported in kilograms. I still think in gallons. The conversion is:

( 1 kg / .73 kg/L) / 2 = .685L * .264 gallons / L = .18 gallons

If you don’t do that kind of thing on a regular basis, though — as I don’t, and as many of us don’t — it’s hard to get over the activation threshold. Looking up and applying the relevant formulae is a multistep procedure. WA collapses it into a single step:

( 1 kilogram / density of gasoline ) / 2

It knows the density of gasoline, and when you do the computation it reports results in a variety of units, including gallons.

I was feeling a bit guilty about needing this sort of intellectual crutch. But then I heard from a friend who had just read the Kill-A-Watt/WA piece. It reminded him of an Energy Tribune article entitled Understanding E=mc² which concludes:

A 1000-MW coal plant — our standard candle — is fed by a 110-car “unit train” arriving at the plant every 30 hours — 300 times a year. Each individual coal car weighs 100 tons and produces 20 minutes of electricity. We are currently straining the capacity of the railroad system moving all this coal around the country. (In China, it has completely broken down.)

A nuclear reactor, on the other hand, refuels when a fleet of six tractor-trailers arrives at the plant with a load of fuel rods once every eighteen months. The fuel rods are only mildly radioactive and can be handled with gloves. They will sit in the reactor for five years. After those five years, about six ounces of matter will be completely transformed into energy. Yet because of the power of E = mc2, the metamorphosis of six ounces of matter will be enough to power the city of San Francisco for five years.

This is what people finds hard to grasp. It is almost beyond our comprehension. How can we run an entire city for five years on six ounces of matter with almost no environmental impact? It all seems so incomprehensible that we make up problems in order to make things seem normal again. A reactor is a bomb waiting to go off. The waste lasts forever, what will we ever do with it? There is something sinister about drawing power from the nucleus of the atom. The technology is beyond human capabilities.

But the technology is not beyond human capabilities. Nor is there anything sinister about nuclear power. It is just beyond anything we ever imagined before the beginning of the 20th century. In the opening years of the 21st century, it is time to start imagining it.

Six ounces of matter? Really? My friend wrote:

I remember at the time I tried to run simple order of magnitude calculations in my head to verify the number, but it got messy, I got sidetracked, and forgot.

This time I went to Wolfram-Alpha, and the answer was right there, clear as day, in seconds (and yes, it’s really 6 ounces of matter).

I went back to the article, and the only quantity of energy reported for San Francisco was that Hetch Hetchy Dam “provides drinking water and 400 megawatts of electricity to San Francisco.” That alone would come to:

400MW * 5 years = ~700 grams = ~25 ounces

Or, if Wikipedia is right and the dam yields only about 220MW, then:

220MW * 5 years = ~386 grams = ~14 ounces

Of course since San Francisco has other sources of power, the amount of matter would be more. Still, this doesn’t invalidate the author’s point: we’re talking ounces, not tons.

When I mentioned this to my friend, though, he wrote back:

I went the other way around:

http://www.wolframalpha.com/input/?i=6oz * c^e2 in gw hr

It gives 4247 GWhr which is definitely in the ballpark for San Francisco.

Sweet!

I didn’t actually follow up on that result just now, but over 5 years it comes to:

http://www.wolframalpha.com/input/?i=4247GWh / 5 years = ~100MW. That’s a quarter of what the article reports for Hetch Hetchy, it’s half what Wikipedia reports, and I still don’t know how it relates to San Francisco’s total power draw.

Even so, we’re playing in the kind of ballpark we need to be able to play in if we’re going to have any kind of reasoned discussion about future energy mixes like Saul Griffith’s straw-man proposal of:

2TW Solar thermal, 2TW Solar PV, 2TW wind, 2TW geothermal, 3TW nukes, 0.5TW biofuels

What I find most striking about the energy literacy talks that Saul’s been giving lately is his ability to move fluidly between the personal quantities of energy we experience directly, the city-scale quantities we experience indirectly, and the global quantities that most of us can scarcely imagine.

My point here isn’t to revisit the dispute that Stewart Brand and Amory Lovins are having about the future role of nuclear power. Nor to endorse William Tucker, the author of that Energy Tribune article, who is a journalist not a scientist or an engineer, and whose argument fails to address issues of security and waste disposal.

Instead I want to focus on how mental power tools like WolframAlpha, by making computable knowledge easier to access and manipulate, can augment our ability to think computationally. If we’re going to reason democratically about the energy, climate, and economic challenges we face, we’re going to need those power tools to be available broadly and used well.

My guest for this week’s Innovators show, Randy Julian, founded the bioinformatics company Indigo BioSystems to help modernize the process of drug discovery. The challenge — and opportunity — is partly to standardize the data formats used to represent experimental data, and to locate that data in shared spaces where it can be linked and recombined.

There’s also the crucial issue of reproducibility. One requirement, as Victoria Stodden said in my conversation with her, is to publish not just data but also the code that processes the data, ideally in an environment where data-transforming computation can be replayed and verified. One of the ways Indigo’s system does that is by hosting instances of R, the wildly popular statistical programming system, in the cloud.

Another key requirement for reproducing an experiment, Randy Julian says, is a robust and machine-readable representation of the design of the experiment. If I don’t know what you’re trying to prove, and how you’re trying to prove it, your data are just numbers to me. If I do know those things, I may be able to verify your results. And we may be able to automate more of the work using machine intelligence and machine labor — a vision that also inspires Jean-Claude Bradley, Cameron Neylon, and others to pursue open-notebook science.

In January 2009 I wrote a series of entries [1, 2, 3] documenting examples of ill-formed iCalendar files. And I argued that we need an analog, in calendar space, to the incredibly useful RSS/Atom feed validator.

I’m delighted to report that Doug Day has taken up the challenge. The first incarnation of his validator is up and running at http://icalvalid.cloudapp.net. It’s based on Doug’s DDay.iCal, which is the same .NET-based iCalendar class library used by the elmcity aggregator. But, like the RSS/Atom validator, it’s driven by an extensible and language-independent suite of tests.

The validator reports numerical scores for an iCalendar file, and gives advice about how it will be handled by popular calendar applications. Some examples:

The Keene High Varsity Basketball schedule scores a 96.25: “This calendar has minor problems, but will likely work correctly in major calendar applications.”

The Hannah Grimes Center’s calendar, based on Drupal, scores 92.5: “This calendar has moderate problems, but may work correctly in major calendar applications.”

The Keene Chamber of Commerce calendar scores 0: “This calendar has severe problems; very few (if any) applications will accept this calendar.” (DDay.iCal does, in fact, overlook these problems, and does parse events from this calendar.)

I’m hugely grateful to Doug Day for doing this important work. Although calendars seem to be ubiquitous, familiar, and interoperable, the examples I’ve been collecting in the wild show that, even though the standard has been around for over a decade, the iCalendar ecosystem is still very immature. This validator will help that ecosystem evolve.

The validator itself, of course, will also evolve. You can send feedback to Doug at the address given on its home page. If you’re curating a location or a topic using the elmcity service, you can email me about problem calendars or bring them to the curators’ room.

I’ve deeply enjoyed every one of the Long Now seminars, but it wasn’t until this one by Stewart Brand in October that I really got what he’s up to as the convener of this remarkable series of talks. In October he appeared as speaker rather than host/interviewer, and he summarized his new book Whole Earth Discipline. Kevin Kelly calls the book “a short course on how to change your mind intelligently” — in this case, about cities, nuclear power, and genetic and planetary engineering. These are all things that Steward Brand once regarded with suspicion but now sees as crucial tools for a sustainable world.

The book weaves together insights from many of my favorite Long Now talks, including:

• Robert Neuwirth on squatter cities

• Stephen Lansing on the “water temple” system in Bali

• Saul Griffith on energy literacy

• Pamela Ronald and Raoul Adamchak on organic and genetically-engineered food

• Drew Endy and Jim Thomas on synthetic biology

• Craig Venter on synthetic biology

• Gwyneth Cravens and Rip Anderson on nuclear power

• Paul Hawken on the environmental movement as Earth’s immune system

• Philip Tetlock on foxes and hedgehogs

• Spencer Beebe on biomimicry

I guess the Long Now seminars is the long version of a course on changing your mind. I was already on board with genetic and planetary engineering, but now I think very differently about cities and nuclear power. The book joins these to a common principle: concentrate the harmful stuff. High-density populations and casks of nuclear waste do less harm than scattered populations and dispersed coal residue.

Don’t miss the annotations — a website that reproduces every paragraph that includes citations, links to their sources, and adds updates.

I’ve always imagined getting an itemized electric bill. We’re not there yet, but when I saw a Kill-A-Watt at Radio Shack last night I remembered the discussion thread at this 2007 blog post and impulsively bought it.

In a way I’m glad I waited until 2009 because a companion tool is available now that wasn’t then: WolframAlpha. Its fluency with units, conversions, and comparisons is really helpful if, like me, you can’t do that stuff quickly and easily in your head.

So, for example, I’m sitting at my desk with the Kill-A-Watt watching my main power strip. I have a mixer here that I use about an hour a week for podcast recording. There’s no power switch because, well, why bother, just leave it on, it’s a tiny draw. Negligible.

I reach over and unplug it. Now I’m drawing 9 fewer watts. But what does that mean? I consult Wolfram Alpha:

Q: 9 W

A: About half the power expended by the human brain.

On a monthly basis?

Q: 9 W * (30 * 24 hours)

A: About half the energy released by combustion of one kilogram of gasoline.

In gallons?

Q: ( 1 kilogram / density of gasoline ) / 2

A: Less than a fifth of a gallon.

Relative to my electric usage, which was 1291 kWh last month?

Q: 9 W / (1291kwh / ( 30 * 24 hours)) * 100

A: Half a percent.

In dollars?

Q: 9 W / (1291kwh / ( 30 * 24 hours)) * $205.60

A: One dollar.

I find these comparisons really helpful. A dollar a month is a rounding error. But if I think of it as the energy equivalent of driving my car 7.2 miles, that makes me want to reach over and unplug the mixer for the 715 hours per month I’m not using it.

Saul Griffith has internalized these calculations, but most of us need help. A next-gen Kill-A-Watt that did these sorts of conversions and comparisons could be a real behavior changer.

In his luminous essay Information obesity, Ned Gulley illustrates the paradox of choice:

I’m reading about the Mohawk Trail, where the Cold River crashes noisily down the granitic glacier-fractured hillside. Where whispering understory birches are sheltered by towering firs. Now my mouth is watering. I have to go. I am referred to ReserveAmerica, a well-built web site that manages thousands of parks nationwide, and — DAMN! Mohawk Trail State Forest is booked solid. I start researching other nearby campgrounds, and now I’m sucked into the game. Unfortunately, ReserveAmerica lets you pick your campsite from an interactive map, and my book tells you which sites are the very best at each campground. Just when you start to salivate about the perfect spot, your dream is dashed by some early bird camper who’s beaten you to the reservation. You can cycle through this process for hours.

I borrow the phrase paradox of choice from Barry Schwartz, who argues in a compelling TED talk that as we broaden our options in all areas, we ratchet up our expectations about how good those options will be. The result is disappointment.

Less is more — except when it isn’t. My counterexample is a recent quest of mine for a particular kind of double-stick tape I needed for an interior storm window project. Key criteria included width (roughly 5/8″) and type of adhesion (plastic to wood). Web search yielded a bewildering array of choices, from various sources, but no way to filter by my criteria. This isn’t some idle consumer whim. I’m trying to save energy in the most effective way I can. I want to see as many qualifying choices as possible. But I can’t.

In Restructuring expert attention to revive the lost art of personal customer service I described one great soluion to this problem: Kevin, the resident expert at FindTape.com, with whom I discussed SCF-01, DC-4420LB, and eventually settled on 3M-4905.

When there’s a Kevin available, he’ll be my first choice. But there won’t always be a Kevin. The answer in that case is not to artificially constrain my choices. That already happens because web search doesn’t enable me to state my criteria. Instead I want to search more effectively. To do that — as noted by several comments on Barry Schwartz’s TED video — we need to overcome filter failure.

This week’s Innovators show, with Martin Hepp, explores how we can create better filters. It’s a follow-on to an earlier show with Kingsley Idehen on the topics of RDFa, the GoodRelations ontology, and the idea that we can become the masters of our own search indexes.

The conversation mainly revolves around how to express an offer for goods or services by means of RDFa snippets that use the GoodRelations e-commerce vocabulary, that are generated by a form-based tool, and that rely on the web’s venerable traditions of view source and copy/paste.

But the same vocabulary used to describe offers can also express needs. And here Martin makes a really good observation about the current architecture of web search:

You can only search synchronously. You can’t ask a question and say, ‘Work on this for two weeks, improve your results in the background, and then come back with the best answer.’ But think about the potential if we can increase the amount of computational time for returning results. Currently there is only 400 milliseconds, because this is the average patience of web users. But if you can express what you’re looking for, and save it with a name, then the search engine will have two weeks to produce a good list of results.

I was also intrigued by Martin’s comments on intermediaries and affiliates. In his view, a commerce site like Amazon is not the only possible source of filter-enhancing metadata. Affiliates can play too. A travel service, for example, might supply search engines with enhanced views of Amazon relative to certain places and certain areas of expertise.

The paradox of choice is real, and in many cases we may indeed be happier with less. But when we really need or want more options, we shouldn’t have to prematurely foreclose them. Search could be far more effective, and an approach like the one Martin envisions is the way to make it so.

Ever since Peter Wayner introduced me to the idea of a translucent database I’ve been thinking about the implications of this powerful idea. In a nutshell, the data in a translucent database service is opaque to the operator of the service, and visible only to sets of users who establish trust relationships. My 2002 review of Peter’s book summarizes his babysitter example:

Imagine a web service that enables parents to find available babysitters. A compromise would disastrously reveal vulnerable households where parents are absent and teenage girls are present. Translucency, in this case, means encrypting sensitive data (identities of parents, identities and schedules of babysitters) so that it is hidden even from the database itself, while yet enabling the two parties (parents, babysitters) to rendezvous.

Fast forwarding to 2009, here’s a current headline from InfoWorld: Microsoft adds access controls for SQL Azure online database. The article doesn’t say so, but this is database translucency in action.

The 2009 version of the babysitter example appears at 37:45 in this PDC session, where Dave Campbell and Rahul Auradkur discuss, and also show, a translucent pharmaceutical reagent marketplace. Dave Campbell spells out the scenario:

Pharma companies see reagents as being pre-competitive. They don’t compete at that level, and they’re willing to sell these reagents to one another, as long nobody can see what’s being bought and sold. That’s the controlled trust we need to set up.

The trick is accomplished by means of encryption and careful separation of concerns. Access policies are isolated from data storage, capable of federation, and auditable by trusted intermediaries.

This is exciting new territory. Historically, we’ve always assumed that the operator of an online information system has complete access to the data in that service. Translucency turns that assumption on its head, and leads to entirely new service design patterns. To implement those patterns requires more than just a database in the cloud. You also need a coordinated suite of supporting services for identity, access control, auditing, and more. Azure, as it becomes one provider of such services, will help make translucency a practical reality.

Back in 2007 I talked with Pablo Castro about Astoria, which I described as a way of making data readable and writeable by means of a RESTful interface. The technology has continued to move forward, and I’m now a heavy user of one of its implementations: the Azure table store. Yesterday at PDC we announced the proposed standardization of this approach as OData, which InfoQ nicely summarizes here.

I’ll leave detailed analysis of the proposal, and the inevitable comparisons to Google’s GData, to others who are better qualified. Nowadays I’m mainly a developer building a web service, and from that perspective it’s very clear that wide adoption of something like “ODBC for the cloud” is needed. We have no shortage of APIs, all of which yield XML and/or JSON data, but you have to overcome friction to compose with these APIs.

For example, the elmcity service merges event information from sets of iCalendar feeds and also from three different sources — Eventful, Upcoming, and (recently added) Eventbrite. In each of those three cases, I’ve had to create slightly different versions of the same algorithm:

• Query for future events
• Retrieve the count of matching events
• Page through the matching events
• Map events into a common data model

Each service uses a slightly different syntax to query for future events. And each reports the count of matching events differently: page_count vs. total_results vs. resultcount. OData would normalize the queries. And because the spec says:

The count value included in the result MUST be enclosed in an <m:count>

it would also normalize the counting of results.

Open data on the web has enormous potential value, but if we have to overcome too much data friction in order to combine it and make sense of it, we will often fail to realize that value. ODBC in its era was a terrific lubricant. I’m hoping that OData, widely implemented in software, services, and mashup environments like the just-announced Dallas, will be another.

My guest for this week’s Innovators show is Gavin Bell, author of Building Social Web Applications. A lot has changed in the decade since I wrote my own book on this topic. One constant, as we discuss in the podcast, is that we still reach for special terminology like computer-supported collaborative work or groupware or social software. That won’t be true forever. Sooner or later we’ll take for granted that all networked information systems augment us collectively as well as individually. Until then, though, it remains appropriate to speak of social web applications as opposed to simply web applications.

Whatever we call this kind of software, it’s a challenge in this era of tech churn to write about it at book length. This effort succeeds by exploring patterns and principles that will endure no matter which technologies prevail. Yes, it’s an O’Reilly technical book, with the traditional animal picture on the cover — in this case, of spiders. But it’s not code-heavy. Gavin Bell aptly compares it to the polar bear book by Peter Morville and Louis Rosenfeld. Both books draw on a wealth of experience gleaned from building and evolving web applications.

For designers, developers, project managers, and online community managers, Building Social Web Applications addresses questions like:

What are the social objects at the core of our application?

How can relationships form around such objects?

Which search, navigation, access, and notification patterns can best support those relationships?

How do we evolve our application as our users gain experience with these object-mediated relationships?

We’ll be thinking about these kinds of questions from now on. Gavin Bell’s excellent book provides a framework in which to do that thinking.

Over the weekend I was poking around in the recipient-reported data at recovery.gov. I filtered the New Hampshire spreadsheet down to items for my town, Keene, and was a bit surprised to find no descriptions in many cases. Here’s the breakdown:

In this case, the half-dozen largest awards aren’t described:

That got me wondering: Where does the money go? So I built a little app that explores ARRA awards for any city or town: http://elmcity.cloudapp.net/arra. For most places, it seems, the ratio of awards with descriptions to awards without isn’t quite so bad. In the case of Philadelphia, for example, “only” 27% of the dollars awarded ($280 million!) are not described.

But even when the description field is filled in, how much does that tell us about what’s actually being done with the money? We can’t expect to find that information in a spreadsheet at recovery.gov. The knowledge is held collectively by the many people who are involved in the projects funded by these awards.

If we want to materialize a view of that collective knowledge, the ARRA data provides a useful starting point. Every award is identified by an award number. These are, effectively, webscale identifiers — that is, more-or-less unique tags we could use to collate newspaper articles, blog entries, tweets, or any other online chatter about awards.

To promote this idea, the app reports award numbers as search strings. In Keene, for example, the school district got an award for $1.47 million. The award number is S394A090030. If you search for that you’ll find nothing but a link back to a recovery.gov page entitled Where is the Money Going?

Recovery.gov can’t bootstrap itself out of this circular trap. But if we use the tags that it has helpfully provided, we might be able to find out a lot more about where the money is going.

A couple of years ago I was enamored with a clever password manager that pointed the way toward an ideal solution. It was really just a bookmarklet — a small chunk of JavaScript code — that used a simple method to produce a unique and strong password for the website you were visiting. The method was to combine a passphrase that you could remember with the domain name of the site, using a one-way cryptographic hash, in order to produce a strong password that would be unique to the site — and that you’d otherwise never be able to remember.

It wasn’t perfect. Sometimes the passwords it generated wouldn’t meet a site’s requirements. And sometimes the login domain name would vary, which broke the scheme. But it introduced me to two powerful — and related — ideas. JavaScript could turn your browser into a programmable cryptographic engine. And that engine could be used to implement protocols that relied on cryptography but transmitted no secrets over the wire.

To my way of thinking, that’s a killer combination. For years I’ve been using Bruce Schneier’s Password Safe, a Windows program that keeps my passwords in an encrypted store. There are many such programs, another example being 1Password for the Mac. This kind of app lives on your computer and talks to a local data store. That means it’s cumbersome to move the app and your data from one of your machines to another. And you can’t use it online, say from a public machine at the library or a friend’s computer.

Imagine a web application that would encrypt your credentials and store them in the cloud. It would deliver that encrypted store to any browser you happen to be using, along with a JavaScript engine that could decrypt it, display your credentials, and even use them to automatically log you onto any of your password-protected services. You’d trust it because its cryptographic code would be available for security pros to validate.

I’ve wanted this solution for a long time. Now I have it: Clipperz. My guest for this week’s Innovators show is Marco Barulli, founder and CEO of Clipperz, which he describes as a zero-knowledge web application. What Clipperz has zero knowledge of is you and your data. It just connects you with your data, on terms that you control, in a way that reminds me of Peter Wayner’s concept of translucent databases.

Clipperz is immediately useful to all of us who struggle to manage our growing collections of online credentials, But it’s also a great example of an important design principle. We reflexively build services that identity users and retain all kinds of information about them. Often we need such knowledge, but it’s a liability for the operators of services that store it, and a risk for users of those services. If it’s feasible not to know, we can embrace that constraint and achieve powerful effects.

You will probably never need to know about the Olson database, also known as the Zoneinfo or tz database. And were it not for my elmcity project I never would have looked into it. I knew roughly that this bedrock database is a compendium of definitions of the world’s timezones, plus rules for daylight savings transitions (DST), used by many operating systems and programming languages.

I presumed that it was written Unix-style, in some kind of plain-text format, and that’s true. Here, for example, are top-level DST rules for the United States since 1918:

# Rule NAME FROM  TO    IN   ON         AT      SAVE    LETTER/S
Rule   US   1918  1919  Mar  lastSun    2:00    1:00    D
Rule   US   1918  1919  Oct  lastSun    2:00    0       S
Rule   US   1942  only  Feb  9          2:00    1:00    W # War
Rule   US   1945  only  Aug  14         23:00u  1:00    P # Peace
Rule   US   1945  only  Sep  30         2:00    0       S
Rule   US   1967  2006  Oct  lastSun    2:00    0       S
Rule   US   1967  1973  Apr  lastSun    2:00    1:00    D
Rule   US   1974  only  Jan  6          2:00    1:00    D
Rule   US   1975  only  Feb  23         2:00    1:00    D
Rule   US   1976  1986  Apr  lastSun    2:00    1:00    D
Rule   US   1987  2006  Apr  Sun>=1     2:00    1:00    D
Rule   US   2007  max   Mar  Sun>=8     2:00    1:00    D
Rule   US   2007  max   Nov  Sun>=1     2:00    0       S

What I didn’t appreciate, until I finally unzipped and untarred a copy of ftp://elsie.nci.nih.gov/pub/tzdata2009o.tar.gz, is the historical scholarship scribbled in the margins of this remarkable database, or document, or hybrid of the two.

You can see a glimpse of that scholarship in the above example. The most recent two rules define the latest (2007) change to US daylight savings. The spring forward rule says: “On the second Sunday in March, at 2AM, save one hour, and use D to change EST to EDT.” Likewise, on the fast-approaching first Sunday in November, spend one hour and go back to EST.

But look at the rules for Feb 9 1942 and Aug 14 1945. The letters are W and P instead of D and S. And the comments tell us that during that period there were timezones like Eastern War Time (EWT) and Eastern Peace Time (EPT). Arthur David Olson elaborates:

From Arthur David Olson (2000-09-25):

Last night I heard part of a rebroadcast of a 1945 Arch Oboler radio drama. In the introduction, Oboler spoke of “Eastern Peace Time.” An AltaVista search turned up :”When the time is announced over the radio now, it is ‘Eastern Peace Time’ instead of the old familiar ‘Eastern War Time.’ Peace is wonderful.”

 

Most of this Talmudic scholarship comes from founding contributor Arthur David Olson and editor Paul Eggert, both of whose Wikipedia pages, although referenced from the Zoneinfo page, strangely do not exist.

But the Olson/Eggert commentary is also interspersed with many contributions, like this one about the Mount Washington Observatory.

From Dave Cantor (2004-11-02)

Early this summer I had the occasion to visit the Mount Washington Observatory weather station atop (of course!) Mount Washington [, NH]…. One of the staff members said that the station was on Eastern Standard Time and didn’t change their clocks for Daylight Saving … so that their reports will always have times which are 5 hours behind UTC.

 

Since Mount Washington has a climate all its own, I guess it makes sense for it to have its own time as well.

Here’s a glimpse of Alaska’s timezone history:

From Paul Eggert (2001-05-30):

Howse writes that Alaska switched from the Julian to the Gregorian calendar, and from east-of-GMT to west-of-GMT days, when the US bought it from Russia. This was on 1867-10-18, a Friday; the previous day was 1867-10-06 Julian, also a Friday. Include only the time zone part of this transition, ignoring the switch from Julian to Gregorian, since we can’t represent the Julian calendar.

As far as we know, none of the exact locations mentioned below were permanently inhabited in 1867 by anyone using either calendar. (Yakutat was colonized by the Russians in 1799, but the settlement was destroyed in 1805 by a Yakutat-kon war party.) However, there were nearby inhabitants in some cases and for our purposes perhaps it’s best to simply use the official transition.

 

You have to have a sense of humor about this stuff, and Paul Eggert does:

From Paul Eggert (1999-03-31):

Shanks writes that Michigan started using standard time on 1885-09-18, but Howse writes (pp 124-125, referring to Popular Astronomy, 1901-01) that Detroit kept

local time until 1900 when the City Council decreed that clocks should be put back twenty-eight minutes to Central Standard Time. Half the city obeyed, half refused. After considerable debate, the decision was rescinded and the city reverted to Sun time. A derisive offer to erect a sundial in front of the city hall was referred to the Committee on Sewers. Then, in 1905, Central time was adopted by city vote.

 

This story is too entertaining to be false, so go with Howse over Shanks.

 

The document is chock full of these sorts of you-can’t-make-this-stuff-up tales:

From Paul Eggert (2001-03-06), following a tip by Markus Kuhn:

Pam Belluck reported in the New York Times (2001-01-31) that the Indiana Legislature is considering a bill to adopt DST statewide. Her article mentioned Vevay, whose post office observes a different
time zone from Danner’s Hardware across the street.

 

I love this one about the cranky Portuguese prime minister:

Martin Bruckmann (1996-02-29) reports via Peter Ilieve

that Portugal is reverting to 0:00 by not moving its clocks this spring.
The new Prime Minister was fed up with getting up in the dark in the winter.

 

Of course Gaza could hardly fail to exhibit weirdness:

From Ephraim Silverberg (1997-03-04, 1998-03-16, 1998-12-28, 2000-01-17 and 2000-07-25):

According to the Office of the Secretary General of the Ministry of Interior, there is NO set rule for Daylight-Savings/Standard time changes. One thing is entrenched in law, however: that there must be at least 150 days of daylight savings time annually.

 

The rule names for this zone are poignant too:

# Zone  NAME            GMTOFF  RULES   FORMAT  [UNTIL]
Zone    Asia/Gaza       2:17:52 -       LMT     1900 Oct
                        2:00    Zion    EET     1948 May 15
                        2:00 EgyptAsia  EE%sT   1967 Jun  5
                        2:00    Zion    I%sT    1996
                        2:00    Jordan  EE%sT   1999
                        2:00 Palestine  EE%sT

There’s also some wonderful commentary in the various software libraries that embody the Olson database. Here’s Stuart Bishop on why pytz, the Python implementation, supports almost all of the Olson timezones:

As Saudi Arabia gave up trying to cope with their timezone definition, I see no reason to complicate my code further to cope with them. (I understand the intention was to set sunset to 0:00 local time, the start of the Islamic day. In the best case caused the DST offset to change daily and worst case caused the DST offset to change each instant depending on how you interpreted the ruling.)

 

It’s all deliciously absurd. And according to Paul Eggert, Ben Franklin is having the last laugh:

From Paul Eggert (2001-03-06):

Daylight Saving Time was first suggested as a joke by Benjamin Franklin in his whimsical essay “An Economical Project for Diminishing the Cost of Light” published in the Journal de Paris (1784-04-26). Not everyone is happy with the results.

 

So is Olson/Zoneinfo/tz a database or a document? Clearly both. And its synthesis of the two modes is, I would argue, a nice example of literate programming.

Recently I’ve posted two examples[1, 2] of Python idioms alongside corresponding C# idioms. It always intrigues me to look at the same concept through different lenses, and it seems to intrigue others as well, so here’s a third installment.

Today’s example comes from a real scenario. I’ve recently added a feature to the elmcity service that enables curators to control their hubs by sending Twitter direct messages to the service. One method, GetDirectMessagesFromTwitter, calls the Twitter API and returns a list of direct messages sent to the elmcity service. Another method, GetDirectMessagesFromAzure, calls the Azure table storage API and returns a list of direct messages stored there. The difference between the two lists — if any — represents new messages to be processed.

Here’s one take on Python and C# idioms for finding the difference between two lists:

fetched_messages =
  GetDirectMessagesFromTwitter();
stored_messages =
  GetDirectMessagesFromAzure();
diff = set(fetched_messages) -
  set(stored_messages)
return list(diff)

var fetched_messages =
  GetDirectMessagesFromTwitter();
var stored_messages =
  GetDirectMessagesFromAzure();
var diff = fetched_messages.Except(
  stored_messages);
return diff.ToList();

I can’t decide which one I prefer. Python’s set arithmetic is mathematically pure. But C#’s noun-verb syntax is appealing too. Which do you prefer? And why?

PS: The Python example above is slightly concocted. It won’t work as shown here because I’m modeling Twitter direct messages as .NET objects. IronPython can use those objects, but the set subtraction fails because the objects returned from the two API calls aren’t directly comparable.

A real working example would add something like this:

fetched_message_sigs = [x.text+x.datetime for x in fetched_messages]
stored_message_sigs = [x.text+x.datetime for x in stored_messages]
diff = list(set(fetched_message_sigs) - set(stored_message_sigs))

But that’s a detail that would only obscure the side-by-side comparison I’m making here.

Because I am lazy, curious, and evangelical, the elmcity service works in an unusual way. Anything that I can delegate to other services I do. So when curators add feeds to hubs, or modify the behavior of hubs, they do it by bookmarking and tagging URLs at delicious.com. It would be foolish to only keep that registry and configuration data in delicious, so I don’t, I persist it to Azure tables. But for now, I’m delegating the data entry interface to delicious.

It’s a lazy approach, in the good sense of lazy. I don’t want to build my own data entry system unless I can add important value, and in this case I can’t.

I’m also curious to see how far this approach can take us. As the project has evolved, so has the tag vocabulary spoken between curators and the service. It’s an easy and natural process, and I don’t see any roadblocks ahead.

Finally, I’m evangelizing this way of doing things because I continue to think that more people should appreciate it.

In this scenario I’ve delegated something else to delicious: authentication. My service doesn’t have its own user accounts. Instead, as the administrator of the service, I tell it to trust a specific set of delicious accounts. When one of those accounts bookmarks an iCalendar URL, and tags it in a particular way, the service regards that as an authenticated request to add the feed to that hub’s registry.

Other requests that curators can make include:

Make the radius for my hub 5 miles.

Make my timezone Arizona.

Make my CSS file to this URL.

But here’s one that curators have wanted to make and couldn’t:

I just added a feed or changed a configuration option. Please reprocess my hub ASAP.

We could represent this message with a tag. Or we could use the rudimentary messaging system in delicious. But these approaches seemed awkward, and I rejected them.

Well, why not Twitter? True, it means that curators who want to send messages to the service will now need accounts in two places. But if they don’t already have accounts on both delicious and Twitter, they can create them. And those accounts will serve them in a variety of ways, unlike a single-purpose account on elmcity.

So, it’s done. As the curator for Keene, I’ve added the tag twitter=judell to the delicious account that controls the Keene hub. As the elmcity service periodically scans its designated set of delicious accounts, it follows any Twitter handle it isn’t already following. Those Twitter accounts can then send direct messages to the Twitter account of the elmcity service.

For now there’s only one thing a curator can say to the service in a direct message — “start” — which means “please reprocess my hub ASAP.” But I’m sure the control vocabulary will evolve. And of course the service can use the channel to send notifications back to curators.

Twitter is famously unreliable, but that should be OK for my purposes. We’re not controlling the space shuttle. If a message doesn’t get through to the service on the first or second try, it’ll get through eventually, and that’ll be good enough.

Someday I may have to build a data entry system and an accounts system. Then again, maybe not. Meanwhile I’m going to keep exploring this lightweight approach. It’s effective and, not coincidentally, it’s fun.

Instead of mourning the lost art of personal customer service, I would rather celebrate examples that show it’s still possible. Yesterday I found two gems.

First, Southwest Airlines. I had booked a round-trip flight and then needed to change to one-way. You can’t do that online. So I clenched my jaw, called customer service, and prepared for the long wait.

Instead, this:

IVR: “Would you like us to call you back in about 20 minutes?”

Me: “Why…yes! Beep, beep, beep, beep, beep, beep, beep, #.”

My jaw relaxed.

Twenty or so minutes later, an agent called back and we made the change. Now the unclenched jaw morphed into a smile.

Second, FindTape.com. I’m making interior storm windows and I need double-stick tape for the project. Which, sure, you can buy online. But the smorgasbord of choices is paralyzing. I wasted a half-hour trying to figure out which product would best suit my unusual application and made no progress whatsoever.

Then, at FindTape.com, I read this:

If you have a specific question related to which tape would work best in your application please fill out and submit the following fields so that we can have an appropriate representative get back in contact with you.

A fellow named Kevin wrote back, we’ve have been discussing my options, and now I’m ready to buy.

Both examples remind me of Michael Nielsen’s luminous phrase: the restructuring of expert attention. He coined it to define a new era of scientific collaboration, but it applies more broadly.

We’ve been told that companies can’t afford to focus expert attention on customers. The truth, of course, is that they can’t afford not to.

For a generation and more we’ve driven a wedge between people who have expertise with products and services and people who need that expertise. How’s that working for you? Me neither.

It’s true that expert attention is a scarce resource. But we’re living through a Cambrian explosion of awareness networks and communication modes. Used adroitly, they can optimize the allocation of that scarce resource. Which is a fancy way of saying: Maybe personal customer service isn’t a lost art after all.