Sensitivity of the data will decide if an application is suitable for processing in the cloud.

The writer does not mention, and probably is unaware of, the principle of translucent data. In a translucent database, the data is encrypted and thus opaque to the operator of the database. Users of the data share keys to unlock the data, and can do anything with cleartext copies that they keep locally. Can real and useful applications be built in this kind of regime? We don’t really know, because hardly anybody has tried. But if it turns out to be possible, it could become a foundation of cloud computing.

I wanted to advance the story. In particular, I wanted to help make a connection between that statement in the Economist and the idea of data translucency. I’ve written about translucency on my blog, and those entries are tagged on delicious. But nowadays the attention stream flows mainly through Twitter. So I composed this tweet:

Economist: “Sensitivity of the data will decide if an application is suitable for processing in the cloud.” Unless the data is #translucent.

There’s a limit to what you can do in 140 characters. That tweet uses all 140, but still falls short of what I wanted to do:

• Quote from the Economist
• Link to the Economist
• Colonize a formerly empty hashtag namespace (#translucency)
• Connect that namespace to its delicious counterpart

Inevitably I failed to do all that in 140 characters. Reflecting on the failure, I made this LazyWeb wish:

I wish I could tweet the command “join http://delicious.com/judell/translucency to #translucent and #translucency”

I’ve had some success joining tag namespaces from different domains. I mentioned the idea in this entry, and a commenter (engtech) provided a nifty solution based on Yahoo Pipes. I have since used it to keep track of items tagged icalvalid on blogs, on delicious, and on Twitter.¹

My LazyWeb wish came from that experience, plus another which I wrote up in an entry entitled To: elmcity, From: @curator, Message: start. That entry describes how elmcity curators can now use Twitter direct messages to send commands to the elmcity service. The mechanism harkens back to Rael Dornfest’s brilliant Sandy, a service that acted as a personal assistant and responded to a repertoire of command messages.

Sandy lost her job when Rael went to work for Twitter. I’ve wondered if she would be rehired there. If so, a command like the one I proposed might be an example of the kind of thing she could do.

On further reflection, I’m not really sure what such a command would mean, or whether it would make sense to use Twitter to send it, or indeed whether it would make sense for Twitter (rather than some other service) to respond to it. But I’m in an exploratory mood, so let’s explore.

It would be straightforward to create a service that would take the Yahoo Pipes trick to the next level. Instead of editing and saving a Yahoo Pipe, you’d just command that service to merge the set of feeds for some tag. That command might best take the form of a URL:

http://tagjoiner.org/join/TAG?delicious=yes&twitter=yes&wordpress=yes

As is true for my combined icalvalid feed, the result formats could be HTML for viewing and RSS for feed splicing. As the creator of the joined feed, I’m aware that it exists, and I can cite it when I want to direct people’s attention to the union of the namespaces.

But suppose I wanted the joined namespace to be more discoverable than that? Here’s where it might make sense for Twitter to be involved. If a hashtag search on Twitter did the join, it could be made evident to the followers of the person making the join request, or even to anyone searching for the hashtag involved in the request.

This is almost surely too indirect and too abstract to ever make sense as a mainstream feature. But it’s fun to imagine. If I’ve made an investment in a tag on delicious, or WordPress, or somewhere else, I’d like to be able to bring those items to the attention of people who encounter the corresponding Twitter hashtag.

The general idea behind all this goes way beyond Twitter, of course. Waiting in the wings is a whole class of services that reconcile different web namespaces.

¹ That feed used to include a mix of items marked [DELICIOUS] and [TWITTER]. But the Twitter items are less durable and seem to have aged out of the combined feed.

Scott writes:

I’m a geek, I like tools and I solve problems in my own niche way.

That statement, along with the essay’s tagline — …Tips for the Geek-Minded Person — has been bothering me ever since I read it. Why is it geeky to marshal the best available data? Why is it geeky to use that data to improve your interaction with people and processes?

My Wikipedia page includes this sentence:

Udell has said, “I’m often described as a leading-edge alpha geek, and that’s fair”. ¹

I did say that, it’s true. But I’ve come to regret that I did. For a while I thought that was because geek was once defined primarily as a carnival freak. That’s changed, of course. Nowadays the primary senses of the word are obsessive technical enthusiasm and social awkwardness. Which is better than being somebody who bites the heads off chickens. But it’s still not how I want to identify myself. Much more importantly, it’s not how I want the world to identify the highest and best principles of geek identity and culture.

Fluency with digital tools and techniques shouldn’t be a badge of membership in a separate tribe. In conversations with Jeannette Wing and Joan Peckham I’ve explored the idea that what they and others call computational thinking is a form of literacy that needs to become a fourth ‘R’ along with Reading, Writing, and Arithmetic.

The term computational thinking is itself, of course, a problem. In comments here, several folks suggested systems thinking which seems better.

Here’s a nice example of that kind of thinking, from Scott’s essay:

#3 Make their job easy

Speak their language and tell them what they can do to get you out of their hair. Refer to flights by number when calling reservations, it saves huge amounts of time. For example, today I called United and I said:

“Hi, I’m on delayed United 686 to LGA from Chicago. Can you get me on standby on United 680?”

Simple and sweet. I noted that UA680 was the FIRST of the 6 flights delayed and the next one to leave. I made a simple, clear request that was easy to grant. I told them where I was, what happened, and what I needed all in one breath. You want to ask questions where the easiest answer is “Sure!”

I see two related kinds of systems thinking at work here. One engages with an information system in order to marshal data. Another engages with a business process — and with the people who implement that process — in a way that leverages the data, reduces process friction, and also reduces interpersonal friction.

These are basic life skills that everyone should want to master. If we taught them broadly, and if everyone learned them, then this sort of mastery wouldn’t attract the geek label. But we don’t teach these skills broadly, most people don’t learn them, and the language we use isn’t our friend. If systems thinking is geeky then only geeks will be systems thinkers. We can’t afford for that to be true. We need everyone to be a systems thinker.

¹ Actually I’d say that Scott Hanselman is a leading-edge alpha geek. I am, at best, a trailing-edge beta or gamma geek. But if someone were to remove the word entirely from my Wikipedia page, I’d be fine with that. I no longer want to be labeled as any kind of geek.

For most of human history, he argues, we were limited by lack of knowledge. We just didn’t know how to do things right. Now that knowledge is abundant the enemy is no longer ignorance but rather ineptitude — the failure to marshal and apply what we know.

The surprising thing Atul Gawande learned, and now passionately conveys, is that simple checklists turn out to be extraordinarily powerful tools for marshalling knowledge and for ensuring its correct use.

The biggest roadblock is pushback from highly-trained experts who are offended by the idea. After 8 years of medical school, and in a regime that already demands vast amounts of paperwork, why should a doctor have to check off basic items on a list? Because we are fallible in the face of complexity, Gawande says, and because checklists work. Although he led research in this area he was skeptical about adopting checklists in his own operating rooms. But when he did, he made two critical discoveries. First, well-made checklists are easy to use. Second, they almost always caught errors.

Most of those errors turned out to be non-critical. Only a few of the catches saved lives. That alone, of course, is enough reason to adopt checklist discipline. But it was shocking for the medical teams to discover that simple and basic procedures, which they thought were being carried out with 100% fidelity, in fact weren’t.

We are willing to tolerate failure when it results from unavoidable ignorance, Gawande says. If we really don’t know how to cure a disease, then OK. You tried your best, you failed, that’s how it is. But if we do know, and screw up, that’s unforgivable. What do you mean she died because somebody forgot to administer the antibiotic, or to wash his hands? Unacceptable.

The struggle with complexity I know best happens in the realm of software. What do our checklists look like? One obvious form is the test suite. If my software keeps passing its tests as I evolve it, there’s still plenty that can and will go wrong, but at least I know it still does what the tests say it does. Once, recently, I deployed a version of the service I’m building that failed in a way my tests would have caught. How did that happen? I was so sure I hadn’t changed anything that the tests would catch that I didn’t bother to rerun them. That’s an unforgivable lapse of discipline I don’t plan to repeat.

But software tests aren’t really the sort of checklist that Gawande writes and speaks about. Here’s something closer to what he means: Best practices in web development with Python and Django. That list comes from Christopher Groskopf, a web developer at the Chicago Tribune, who writes:

In our fast-paced environment there is little justification for being confused when it could have been avoided by simply writing it down.

We need to recognize and honor this kind of work. It is unsexy but heroic, and I use that word deliberately. The power of the checklist discipline, Gawande says, should prompt us to rethink our definition of heroism. Consider Capt. Chesley “Sully” Sullenberger:

It was fascinating to watch people responding to the miracle on the Hudson. All of us, staring in amazement, thinking what a hero he was. But none of us willing to listen to what he really was saying. He kept saying it wasn’t flight ability, but instead adherence to discipline, and teamwork. But it was as if we couldn’t process what he was trying to tell us.

Because there were checklists, and because everybody used them, Sully could rise above the dumb stuff and focus on the one key decision for which human judgement was required. The heroic part of that flight was not the flight ability of Capt. Sullenberger, it was the willingness of the entire team — including the flight attendants, who then acted through their protocols to get the passengers off that plane in three minutes — to acknowledge their fallibility, admit that they could fail by relying only on training and memory, and exercise the discipline to overcome that fallibility.

The talk raises important questions for practitioners in every field. What makes checklists easy to use? What makes them effective? In the realm of software, we have plenty of examples to look at: django, WordPress, C#, ASP.NET, etc. It might be fruitful to explore these, merge similar lists, and codify stylistic patterns that can govern all such lists.

Although we tend to focus on the portability of our health care data, the same principles apply to all kinds of service providers. And in many of those cases, we would be less concerned about the privacy of the data.

Why, then, don’t service providers and their customers co-own this data? Is it because providers want to keep high-quality electronic data, while only dispensing low-quality paper data, in order to make their services stickier? It would make a certain kind of sense for Honda to think that way, but I don’t think that’s the answer. Instead:

1. Nobody asks for the data.

2. There’s no convenient way to provide it.

We’ll get over the first hurdle as our cultural expectations evolve. Today it would be weird to find an OData URL printed on your paid work order. In a few years, I hope, that will be normal.

We’ll get over the second hurdle as service providers begin to colonize the cloud. One of the key points I tried to make in a recent interview about cloud computing is that cloud-based services can flip a crucial default setting. If you want to export access to data stored in today’s point-of-sale and back-end systems, you have to swim upstream. But when those systems are cloud-based, you can go with the flow. The data in those systems can still be held closely. But when you’re asked to share it, the request is much easier to satisfy.

Reading both of those books, I felt an implicit connection between principles that I’ve learned in an IT context (e.g., separation of concerns, networks of loosely-coupled services), and principles that can inform the practice of architecture — at the scale of buildings, but also of whole cities. Duncan Wilson, and others lucky enough to be working at the forefront of 21st-century architecture, are making that connection explicit.

Consider, for example, the movement of goods in and out of a city. You’d like to consolidate that activity at the perimeter and reduce truck traffic in the core. That’s doable, but only if retailers and suppliers are willing to share information about what they’re shipping. That began to happen in the late 1990s, Duncan says, when retailers and suppliers began to share trucks. Doing the same kind of thing for a city, as Arup’s engineers envision, would entail both a physical arrangement of consolidation centers on the perimeter, and a virtual arrangement of shared data.

Information, Duncan says, is becoming another of the raw materials from which the built environment is made.

Here’s a different example of IT principles crossing over into other realms, from a podcast I listened to on yesterday’s hike:

When you offer multiple services using the same devices, through the same interfaces, you open up opportunities for creative thinking in the storage community.

If you’re talking about data storage, and the frame of reference is IT, that’s not a very compelling statement. We haven’t fully internalized this service-oriented and network-based way of thinking, but we’re getting there.

But that quote doesn’t refer to data storage, it refers to energy storage. The podcast was Stephen Lacey’s excellent Inside Renewable Energy. In this episode, innovators at Ice Energy and A123 describe business models that are deeply informed by the idea of networks of shared services.

Inspired by a PHP solution to the problem, Scott set out to replicate it for ASP.NET.

But then I realized that a server-side solution wasn’t really necessary.

Could I do it all on the client side? This way anyone could add this feature to their site, regardless of their server-side choice.

One next step, as Scott points out, is to add geolocation so the list of kids you see will be more relevant to you. But there are lots of ways to contextualize that list based on aspects of your identity. And this is what Kynetx applications do: Contextualize your experience of the web based on aspects of your identity.

My own interest in this idea dates back to the LibraryLookup project, which was an early demonstration of the power of client-driven contextualization. It evolved from a bookmarklet to a browser plug-in, but then stalled there for lack of a ubiquitous client-side technology.

Now there is: jQuery. What Scott’s example shows, as do all Kynetx applications, is that we’re ready to make clients more equal partners in the dance of the web. Among other things, this possibility raises horny issues about the control of content — issues that I explored in a 2005 screencast.

But there’s also a deep connection between Phil’s work and the ongoing saga of digital identity. Phil wrote a book on that subject, and has been a key organizer of the Internet Identity Workshop. When he started Kynetx he wasn’t really thinking about a tie-in to Information Cards and the identity metasystem. But the connection emerged organically.

In a Kynetx-enhanced version of the Missing Kids 404 Page application, your browser would present selected aspects of your identity to the services that provide the data, and a Kynetx application would personalize that data in ways meaningful to you.

The Internet began as a network of peers. That arrangement didn’t last long, and there have been several efforts to restore the original symmetry. In the early 2000s, during Napster’s heyday, there was a flurry of interest in peer-to-peer architectures. Thanks to today’s more capable and more standardized browsers, we’re seeing a new wave of interest. I’m looking foward to hanging out at the Kynetx conference and meeting folks who are riding that wave.

Ebooks are another potential source. But so far, Flat World has found that students overwhelmingly prefer to read printed books. Eric Frank has a wonderfully pragmatic view:

Publishers need to be device-agnostic in the broadest sense. The printed book is one of the devices we target.

As and when students indicate a preference for ebook formats, Flat World will provide them. It does seem that ebook readers are on the cusp of mainstream adoption. But it has seemed that way before. “It would be tragic,” Eric says, “to bet your business on that.”

The bet that Flat World is making is on a neutral format, Docbook XML, from which any other format can be automatically derived. I’ve done a lot of my own publishing to multiple formats from a single source. Back in 2003, when XML support was added to Microsoft Word — which was and is the tool of choice for book-length writing — I thought it would end the painful process of converting Word manuscripts into published formats. That mostly hasn’t happened yet. Jon Williams thinks that’s because, until recently, publishers didn’t need to automate the production of various electronic formats. As that need arises, we should finally begin to see end-to-end automation from original manuscript to published formats.

Flat World is a commercial publisher of open textbooks, and Eric is careful to spell out what he means by open. It doesn’t simply mean free, or collaborative, although there are both free and collaborative ways to use Flat World books. It means precisely what the by-nc-sa license says: You are free to use, share, and remix, with attribution, but not for commercial gain. Whenever a work yields commercial value, in any of the ways it might, that money must flow back to Flat World and its stable of authors.

The dominant revenue stream is print-on-demand. If Flat World is able to scale out its catalog — and that’s the biggest if the company faces — its printed textbooks will be an affordable alternative to conventional offerings. Meanwhile, teachers who adopt Flat World books can adapt them to their needs. In theory, a Flat World book can become a nexus of collaboration, grow more valuable as a result, convert some of that value into revenue, and share that revenue with a community of collaborators as well as with the publisher and author. It’s early days, and that hasn’t happened yet. But I like the way that Flat World has opened a door to that possibility, without betting its business on lots of people walking through it anytime soon.

Malmström’s 1997 book Cycles of the Sun, Mysteries of the Moon, which he has also made freely available here, tells the whole story from his point of view. It’s a remarkable tale of geography, religion, culture, computation, science, and human foibles.

The Maya actually used three different calendars. The Tzolk’in ran on a 260-day cycle, and the Haab’ used a 365-day cycle. Then there was the Long Count, which counted days since a mythical beginning of time and also included the other two.

The Long Count’s start date was written, in its full form, like this:

0.0.0.0.0, 4 Ahau 8, Cumku

The first five digits measure days in units of 144,000, 7,200, 360, 20, and 1. 4 Ahau is a Tzolk’in day, based on a cycle of 13 numbers with a cycle of 20 days names. 8 Cumku is a Haab’ day, based on 18 20-day months.

Today’s date is 12.19.17.2.3, which Wikipedia’s Long Count page helpfully computes for you using this markup:

Today, {{CURRENTDATE}}, in the Long Count is {{Maya date}} (GMT correlation)

(Here GMT doesn’t stand for Greenwhich Mean Time, but rather for Goodman-Martinez-Thompson.)

But today might be 12.19.17.2.2, according to this calculator. There has, evidently, been epic confusion and controversy about whether the mythical start date was 584,283 or 584,284 or 584,285 days ago. Thompson originally thought 584,285, then changed his mind and decided on 584,283.

Prof. Malmström likes 584,285, which fixes the start date as August 13, 3114 B.C. Why? Thompson didn’t think there was any astronomical basis for the 260-day calendar, but Malmström figured there had to have been. And he wondered where, in that part of the world, you might observe a 260-day astronomical cycle.

It turns out that at latitude 14.8 º N, the sun is directly overhead on August 13 passing southward, and again on April 30 passing northward, an interval of 260 days. August 13 is also the day after the peak of the Perseid meteor shower. Malmström writes:

The signs were therefore unmistakable. First the heavens would give their notice. All night long the skygazer would watch as stars burst from behind the towering mountains to the northeast and flashed across the sky. And the following morning, as the sun arched higher and higher across the heavens, he would watch as the shadow it cast grew steadily shorter, until, as the sun reached its zenith, its shadow completely disappeared. This then, he decided, was the day for his count to begin.

Why count days? If you’re planting maize, you need to calibrate carefully to the arrival of the monsoon rains. The two solar passages correspond roughly to the beginning of the rainy season at the end of April, and the harvest in mid-August.

Note that these passages, and the associated latitude 14.8 º N, don’t apply to the Maya in the Yucatán Peninsula, but instead to an earlier Olmec civilization to the southwest, on the Pacific coast near what is now the border between Mexico and Guatemala. The Mayan new year was July 26, not August 13. But the 260-day calendar predated the Mayans by a millenium.

Just a few decades after its inception, the 260-day “sacred” calendar was augmented by a 365-day “secular” calendar. The problem was that the sacred calendar didn’t quite work. There were 13 20-day cycles — or 20 13-day cycles — during the sun’s southward passage, and what seemed like 8 more 13-day cycles during the northward passage. So when the calendar started running, things seemed to work out — albeit in a delightfully curious way.

Each time the zenithal sun passed overhead on its way south, a new 260-day cycle would begin on a day numbered “1″ but with a different name. Thus, the skygazer watched as the beginning of each successive cycle shifted from “1 Alligator” to “1 Snake” to “1 Water” to “1 Reed” and then to “1 Earthquake.”

That didn’t last long, though.

Where the priest had erred, of course, was in concluding that the cycle of the sun could be measured in 28 “bundles” of 13 days. This meant that he had equated its annual migration through the heavens with an interval of 364 days, when in actuality it took about a day and a quarter longer than that. Thus, after only four years had elapsed his count was already off by 5 days. This might go unnoticed by the commoners at first, but certainly, as the error increased with each passing year, it wouldn’t be long before “the cat was out of the bag.”

What a colossal screwup! I like to imagine the priests furiously backpedaling.

OK, wait, I know we said 260, but it’s really 365, but we’ll keep both, don’t worry, it’ll work out, trust us, we know what we’re doing.

Of course the fun never stops. We’re less than two years away from Y 13.0.0.0.0. That’s in 2012, on Dec 23. Or on Dec 22, or Dec 21, depending on which correlation constant you choose. On one of those dates the world will end. Or not. Prof. Malmström suggests you choose 584,285. That’ll give you two extra days to put your affairs in order.

For more on the endlessly weird human reckoning of time, see A literay appreciation of the Olson/Zoneinfo/tz database.

The Lazy Web seems not to have taken up the challenge, so I took a crack at it. The solution I came up with is a single-page application, which is just a web page that uses HTML, CSS, and Ajax to do something that’s (hopefully) interesting and useful.

Here’s the page: http://jonudell.net/NamesOfTwitterListsFor.html

It defaults to my Twitter name but you’ll of course want to try yours, and those of others you’re curious about. The first time through, you’ll be prompted to authenticate to api.twitter.com. This looks like the password anti-pattern, but really isn’t. You’re authenticating yourself to the Twitter API in the same way that you normally do to the Twitter website.

Note that since the API call used to build the tag cloud is rate-limited, queries through this page will be charged against your daily allotment of Twitter API usage, just as when you use client applications like TweetDeck or Seesmic.

What will your tag cloud say about you? I don’t think you’ll be surprised. It’s just another of the unique signatures written for us by others. That those signatures do get written, though, and that they can be discovered and read, never ceases to surprise me.

The dynamics of single-page applications also never cease to surprise me. In this case, a tiny 4K web page is all that’s delivered from my modestly-equipped personal webserver. It would probably survive a Slashdotting. If not, the page could be hosted on any other server, or on a other local drive, and would continue to work the same way.

I’m also using jQuery, in this case served from the Microsoft content delivery network, so that’s unlikely to be a bottleneck. The only real limit is Twitter API usage, and that’s spread across all the Twitter users who authenticate through the page.

When you arrange and deploy a tiny amount of HTML, CSS, and JavaScript in this way, you can create a lot of leverage!

Here’s how A Pattern Language summarizes a pattern called FOUR-STORY LIMIT:

Therefore, in any urban area, no matter how dense, keep the majority of buildings four stories high or less.

And here’s how the Portland Pattern Repository summarizes the Singleton Pattern:

Therefore, let the class create and manage the single instance of itself, the Singleton. Wherever in the system you need access to this single instance, query the class.

The stylistic allusion shows a direct literary influence flowing from architectural pattern language to software pattern language. Alexander’s book, by the way, is a pre-Web hypertext. The pattern called FOUR-STORY LIMIT (21), for example, refers to NUMBER OF STORIES (96), DENSITY RINGS (29), BUILDING COMPLEX (95), HOUSING HILL (39), and HIGH PLACES (62). Each of these numbered patterns links to a set of related patterns, as does each page in the Portland Pattern Repository — which was also, of course, the Ur-wiki from which all things wiki are descended.

I suspect we’ve yet to fully elaborate the connections between software, architecture, and networks. Consider these pattern names from Alexander’s 1977 book:

These evocative names, and the sketches that accompany them, arise from a deeply network-oriented way of thinking. Many of the higher-level patterns express core values about connectivity and decentralization. And those values resonate more powerfully now, in our Net-aware world of 2010, than they might have in 1977.

Some of the prescriptions in A Pattern Language can seem absurd. For example, Alexander argues that an optimal urban core should serve a “catch basin” of about 300,000 people, that these cores should be widely distributed, and that each should specialize in some way that makes it world-class. Why?

The problem is clear. On the one hand people will only expend so much effort to get goods and services and attend cultural events, even the very best ones. On the other hand, real variety and choice can only occur where there is concentrated, centralized activity; and when the concentration and centralization becomes too great, then people are no longer willing to take the time to go to it.

Which is fine in theory, but we’ve already built megalopolises surrounded by suburbs. It’s not like we can do it over.

Except when we can. In America the most striking examples are in Michigan where I lived for many years. Detroit, once a city of two million, is being recreated as a city that may end up at less than a third that size. What will become of the rest? It just might be plowed into farmland. If so, a pattern called CITY COUNTRY FINGERS may turn out to be a useful guide.

Likewise there are plans afoot to gather the remaining population of Flint into a few viable neighborhoods and let the vacated land become parks and forests. If that happens, many of the ideas in A Pattern Language, about how to organize neighborhoods and their transportation networks, could come into play.

In Asia, meanwhile, entire new cities are being built from scratch. I recently met an engineer who works for the global consulting firm Arup. For one of their projects, he told me they’re using a simulation of wind flow as one of the constraints on the layout of streets and buildings. The layout is also informed by RING ROADS and LOCAL TRANSPORT AREAS, patterns that yield a tiered distribution network which optimizes the use of delivery trucks.

Networked software is highly malleable, and we take for granted that we can try out different design patterns. The built environment rarely affords the same opportunity. But in this century of urbanization, as circumstances force us to rethink our energy, transportation, and settlement networks, it may turn out to be softer than we suppose, and more open to the influence of pattern languages.

My proposal advanced to the final round and was then rejected. Given my initial ambivalence I was OK with that. But the stated rationale has been bugging me ever since. The letter said:

Because there are thousands of proposals and only a few of them advance, we are able to choose only the most innovative ideas. These are new kinds of technologies or techniques, usually things we have never heard of before.

The meme woven into that paragraph has a name: Shiny New Thing syndrome. It is a plague. Technology journalism feeds it. Thought leaders, including Dave Slusher, Jeremy Zawodny, and Jeff Atwood, have denounced it.

I’m clearly biased, since all my best work involves creative remixing of ideas and technologies that are as common as dirt. But I do wonder about the harm that’s done when we equate innovation with shiny new things.

Old things are full of latent value that we’ve yet to discover and unlock. Why? It takes a long time for real understanding to sink in. In Net infrastructure, consider how long it’s taken us to grok what HTTP, REST, HTML, and JavaScript really are and can do. In education, look at the high-value uses that Sal Khan and Dan Meyer find for low-tech screencasting and blogging tools. In journalism and civic life, read what Alan Rusbridger says about Will Perrin’s compelling — and yet so last-century — use of Typepad to activate communities.

Well, I try to do my part. On my show, which is called Interviews with Innovators, I feature people who are more likely to be evolutionary repurposers than revolutionary creators. Maybe I should rename the show Shiny Old Things.

Data capture

The web and worker roles in my Azure service take periodic snapshots of a set of Windows performance counters, and store those to an Azure table. Although I could be using the recently-released Azure diagnostics API, I’d already come up with my own approach. I keep a list of the counters I want to measure in another Azure table, shown here in Cerebrata’s viewer/editor:

When you query an Azure table like this one, the records come back packaged as content elements within Atom entries:

<entry m:etag="W/datetime'2010-02-09T00:00:53.7164253Z'">
<id>http://elmcity.table.core.windows.net/monitor(PartitionKey='ProcessMonitor',
  RowKey='634012704503641218')</id>
<content type="application/xml">
<m:properties>
<d:PartitionKey>ProcessMonitor</d:PartitionKey>
<d:RowKey>634012704503641218</d:RowKey>
<d:HostName>RD00155D317B3F</d:HostName>
<d:ProcName>WaWorkerHost</d:ProcName>
<d:mem_available_mbytes m:type="Edm.Double">1320</d:mem_available_mbytes>
...snip...
<d:tcp_connections_established m:type="Edm.Double">24</d:tcp_connections_established>
</m:properties>
</content>
</entry>

This isn’t immediately obvious if you use the storage client libary that comes with the Azure SDK, which wraps an ADO.NET Data Services abstraction around the Azure table service. But if you peek under the covers using a tool like Eric Lawrence’s astonishingly capable Fiddler, you’ll see nothing but Atom entries. In order to get direct access to them, I don’t actually use the storage client library in the SDK, but instead use an alternate interface that exposes the underlying HTTP/REST machinery.

Exposing data services

If the Azure table service did not require special authentication, it would itself be an OData service that you could point any OData-aware client at. To fetch recent entries from my table of snapshots, for example, you could use this URL in any browser:

GET http://elmcity.table.core.windows.net/monitor?$filter=Timestamp+gt+datetime’2010-02-08′

(A table named ‘monitor’ is where the telemetry data are stored.)

The table service does require authentication, though, so in order to export data feeds I’m creating wrappers around selected queries. Until recently, I’ve always packaged the query response as a .NET List of Dictionaries. A record in an Azure table maps nicely to a Dictionary. Both are flexible bags of name/value pairs, and a Dictionary is easily consumed from both C# and IronPython.

To enable OData services I just added an alternate method that returns the raw response from an Azure table query. Then I extended the public namespace of my service, adding a /odata mapping that accepts URL parameters for the name of a table, and for the text of a query. I’m doing this in ASP.NET MVC, but there’s nothing special about the technique. If you were working in, say, Rails or Django, it would be just the same. You’d map out a piece of public namespace, and wire it to a parameterized service that returns Atom feeds.

Discovering data services

An OData-aware client can use an Atom service document to find out what feeds are available from a provider. The one I’m using looks kind of like this:

<?xml version='1.0' encoding='utf-8' standalone='yes'?>
<service xmlns:atom='http://www.w3.org/2005/Atom'
  xmlns:app='http://www.w3.org/2007/app' xmlns='http://www.w3.org/2007/app'>
  <workspace>
    <atom:title>elmcity odata feeds</atom:title>
    <collection href='http://elmcity.cloudapp.net/odata?table=monitor&hours_ago=48'>
      <atom:title>recent monitor data (web and worker roles)</atom:title>
    </collection>
    <collection href="http://elmcity.cloudapp.net/odata?table=monitor&hours_ago=48&
      query=ProcName eq 'WaWebHost'">
      <atom:title>recent monitor data (web roles)</atom:title>
    </collection>
    <collection href="http://elmcity.cloudapp.net/odata?table=monitor&hours_ago=48&
         query=ProcName eq 'WaWorkerHost'">
      <atom:title>recent monitor data (worker roles)</atom:title>
    </collection>
    <collection href="http://elmcity.cloudapp.net/odata?table=counters">
      <atom:title>peformance counters</atom:title>
    </collection>
  </workspace>
</service>

PowerPivot is an Excel add-in that knows about this stuff. Here’s a picture of PowerPivot discovering those feeds:

It’s straightforward for any application or service, written in any language, running in any environment, to enable this kind of discovery.

Using data services

In my case, PowerPivot — which is an add-in that brings some nice business intelligence capability to Excel — makes a good consumer of my data services. Here are some charts that slice my service’s request execution times in a couple of different ways:

Again, it’s straightforward for any application or service, written in any language, running in any environment, to do this kind of thing. It’s all just Atom feeds with data-describing payloads. There’s nothing special about it, which is the whole point. If things pan out as I hope, we’ll have a cornucopia of OData feeds — from our banks, from our Internet service providers, from our governments, and from every other source that currently publishes data on paper, or in less useful electronic formats like PDF and HTML. And we’ll have a variety of OData clients, on mobile devices and on our desktops and in the cloud, that enable us to work with those data feeds.

• The new 25-payline special edition of Wheel of Wealth will have you holding your breath in excitement…

• Play Online Slot Machine Game. Coin in – spin – hold your breath……Watch those symbols…..Will it or won’t it?

• After the first two hits you’re holding your breath for the third reel…

We don’t talk about slot-machine apnea but it’s the same syndrome, produced by the same cause: an intermittent, or variable-interval, schedule of reinforcement. Any activity that exhibits this pattern will be powerfully addictive. A dog begging for scraps of food at the table, rewarded only once in a thousand times, will always beg. Likewise a human begging for scraps of attention.

The link between variable-interval reinforcement and email addiction is well known. Less studied is how this plays out in other modes of electronic discourse. The architecture of those modes introduces another key variable: attention payoff. In a group-structured system, like email or Facebook, the payoff is bounded by group size. It’s true that email messages can escape and go viral, but when that happens the attention payoff is never the kind you want.

But in open pub/sub systems, like blogs or Twitter, the payoff is unlimited. Any item that you post could attract worldwide attention, boost your reputation, land you a job, or make a key personal or professional connection. However there’s no guarantee that you’ll get any reinforcement at all. So some fall by the wayside, others become addicted.

“Technology is here to stay,” Linda says. “Can our relationship to it change?”

It must, it can, and it will. But we’ll need to develop some intuitions about global scale and connectedness for which evolution did not prepare us. And then we’ll need to translate them back down to the human scale. Evolution has taught us how to be social. Technology amplifies our ability to give and receive attention, but it doesn’t change the rules of the game. There’s a time to listen, a time to talk, a time to breathe. We’ll remember, and we’ll figure it out.

When I first got interested in screencasting, I envisioned the medium not only as a way to demonstrate software, but also as a way to share knowledge at Internet scale. Sal’s work fulfills that vision, and points the way toward a profound and much-needed disruption of our educational system.

At its core, Sal’s project isn’t about YouTube screencasts. It’s about intuition.

I always got frustrated by what went on in the classroom. You see otherwise intelligent peers memorizing facts and not really caring about the actual intuition. And because they didn’t care about the intution in their junior year, when that same idea pops up in senior year, it’s like they’ve never seen it before. It boggled my mind. You’re just relabeling the same concept over and over.

Sal cares about the intuition, and he wants others to care about the intution too. The first beneficiary of that desire was his cousin Nadia, whom he tutored remotely. Then followed other cousins and family friends. Then it dawned on him that there were no limits. The project could scale out. He could become a superempowered individual, reaching anyone who finds value in his method.

One of the key ingredients of that method is improvisation. These videos aren’t carefully planned, and they aren’t edited. As a viewer, you find yourself looking over the shoulder of a smart and broadly knowledgeable person who is solving problems by thinking on his feet. You watch a practitioner at work: engaged with his medium, wrestling with his tools, correcting false starts.

It was Chris Gemignani who first showed me the value of this approach, in a screencast that teaches how to do unexpectedly powerful and elegant Excel charting. He did it in one take. I’d have been tempted to edit out the false starts. But Chris knew better. Learning how a practitioner really thinks about solving a problem is even more valuable than learning the solution to the problem.

One thing that Sal’s lessons can’t be, of course, is interactive. Nor does he pretend that these videos will make teachers obsolete. But he does suggest, and I violently agree, that teachers can and should become curators of online assets like the ones Sal is creating, and should know when and how to weave those assets into their classes.

Teachers should also become connectors. Sal won’t be the only game in town. Other superempowered tutors will emerge. Each will have a unique style. For a given student, a given subject, and a given problem, one or another of those styles may be right. The best teachers will know their own strengths and limitations, will know which online tutors complement their strengths in a variety of ways, and will connect their students with those tutors.

Sal Khan is on fire. He burns with a passion to share his intuitions with anyone and everyone. It is a beautiful thing to see. He has abandoned a lucrative career in finance to do this fulltime, and I am quite sure he will find a way to keep doing it.

PS: The title of this piece refers to Richard Ankrom’s Los Angeles freeway project. At a busy intersection, millions of motorists have been directed to North 5 by a sign that Caltrans omitted. Ankrom created and installed that missing sign.

PPS: I wrote to my son’s math teacher about Sal Khan. She replied: “Thanks for that link to the Khan Academy. I was overwhelmed by how many video lessons he has! He does seem like an inspiring man. Unfortunately, You Tube is blocked here at the high school.”

The Open Data Protocol (OData) is a web protocol for querying and updating data. OData applies web technologies such as HTTP, Atom Publishing Protocol (AtomPub) and JSON to provide access to information from a variety of applications, services, and stores.

The other day, Pablo Castro wrote an excellent post explaining how developers can implement aspects of the modular OData spec, and outlining some benefits that accrue from each. One of the aspects is query, and Pablo gives this example:

http://ogdi.cloudapp.net/v1/dc/BankLocations?$filter=zipcode eq 20007

One benefit for exposing query to developers, Pablo says, is:

Developers using the Data Services client for .NET would be able to use LINQ against your service, at least for the operators that map to the query options you implemented.

I’d like to suggest that there’s a huge benefit for users as well. Consider Pablo’s example, based on some Washington, DC datasets published using the Open Government Data Initiative toolkit. Let’s look at one of those datasets, BankLocations, through the lens of Excel 2010’s PowerPivot.

PowerPivot adds heavy-duty business analytics to Excel in ways I’m not really qualified to discuss, but for my purposes here that’s beside the point. I’m just using it to show what it can be like, from a user’s perspective, to point an OData-aware client, which could be any desktop or web application, at an OData source, which could be provided by any backend service.

In this case, I pointed PowerPivot at the following URL:

http://ogdi.cloudapp.net/v1/dc/BankLocations

I previewed the Atom feed, selected a subset of the columns, and imported them into a pivot table. I used slicers to help visualize the zipcodes associated with each bank. And I wound up with a view which reports that there are three branches of WashingtonFirst Bank in DC, at three addresses, in two zipcodes.

If I were to name this worksheet, I’d call it WashingonFirst Bank branches in DC. But it has another kind of name, one that’s independent of the user who makes such a view, and of the application used to make it. Here is that other name:

http://ogdi.cloudapp.net/v1/dc/BankLocations?$filter=name eq ‘WashingtonFirst Bank’

If you and I want to have a conversation about banks in Washington, DC, and if we agree that this dataset is an authoritative list of them, then we — and anyone else who cares about this stuff — can converse using a language in which phrases like ‘WashingtonFirst Bank branches in DC’ or ‘banks in zipcode 20007′ are well defined.

If we incorporate this kind of fully articulated web namespace into public online discourse, then others can engage with it too. Suppose, to take just one small example, I find what I think is an error in the dataset. Maybe I think one of the branch addresses is wrong. Or maybe I want to associate some extra information with the address. Today, the way things usually work, I’d visit the source website and look for some kind of feedback mechanism. If there is one, and if I’m willing to provide my feedback in a form it will accept, and if my feedback is accepted, then my effort to engage with that dataset will be successful. But that’s a lot of ifs.

When public datasets provide fully articulated web namespaces, though, things can happen in a more loosely coupled way. I can post my feedback anywhere — for example, right here on this blog. If I have something to say about the WashingtonFirst branch at 1500 K Street, NW, I can refer to it using an URL: 1500 K Street, NW.

That URL is, in effect, a trackback that points to one record in the dataset.¹ The service that hosts the dataset could scan the web for these inbound links and, if desired, reflect them back to its users. Or any other service could do the same. Discourse about the dataset can grow online in a decentralized way. The publisher need not explicitly support, maintain, or be liable for that discourse. But it can be discovered and aggregated by any interested party.

The open data movement, in government and elsewhere, aims to help people engage with and participate in processes represented by the data. When you publish data in a fully articulated way, you build a framework for engagement, a trellis for participation. This is a huge opportunity, and it’s what most excites me about OData.

¹ PowerPivot doesn’t currently expose that URL, but it could, and so could any other OData-aware application.