As I review and improve the elmcity hubs in selected cities, I am again reminded of William Gibson’s wonderful aphorism: “The future is already here, it’s just not evenly distributed.” Yesterday we saw that the future of community calendars hasn’t yet arrived at the University of Michigan. But today I was delighted to see that it has arrived, in a big way, for the Ann Arbor public schools. Almost all of them, it turns out, are making good use of Google Calendar to publish machine-readable calendar information. This morning I rounded up thirty of those calendars and added them to Ann Arbor’s elmcity hub, bringing the total number of feeds from 194 to 224.

Here’s the breakdown of the 309 events from the grade schools:

And the 322 events from the middle schools:

And the 966 events from the high schools:

Among grade schools, Slauson is notable not only for the number of events but for the exemplary self-categorization applied to them. When you click the Google Calendar subscribe button on the Slauson calendar page here’s what you’ll see:

This is a best practice I wish everyone would adopt. It illustrates the seventh of my seven ways to think like the web: #7: Reuse components and services. The Slauson calendar is both a user of other services (the district-wide calendars) and a provider of services. And as a provider, it understands the idea of componentization. In an era of abundance it costs no more to create and manage a dozen calendars, using free services like Google Calendar and Hotmail Calendar, than to jam everything into a single calendar. The benefits are manifold. They include:

Delegation

In most schools and businesses, maintenance of “the” public calendar is one person’s job. That person becomes a bottleneck. When you recognize that logically there isn’t one public calendar, but instead several or many, each with its own appropriate maintainer, then you can break that bottleneck.

Precision

A parent who subscribes to a single undifferentiated school calendar may be overwhelmed by the flow. Parents of kids who are in music programs, or on sports teams, or who are going on the Chicago trip, should be able to focus on those activities.

Scope

Slauson Middle School is part of the larger Ann Arbor community. When Slauson’s calendars are self-categorized, they can align with community-wide views. Here’s a picture of some sports-related activites in Ann Arbor on November 17th:

Slauson’s publication of a set of self-categorized machine-readable calendar feeds enables it to represent its sports activities on a city-wide timeline that includes, in this particular view, events from the Ann Arbor Triathlon Club, the Wolverines basketball teams, and Ann Arbor’s kickball Meetup group.

Well done Slauson! And kudos to all the Ann Arbor public schools. You have become web thinkers. I hope schools everywhere will learn from your example.

Here’s one of my favorite scenes from the movie Brazil,

Sam Lowry (Jonathan Pryce): Are you from central services?

Harry Tuttle (Robert De Niro): Hah! They’re a little overworked these days. Luckily I intercepted your call.

Sam Lowry: Can you fix it?

Harry Tuttle: No, but I can bypass it with one of these.

Back in 2003, my InfoWorld column APIs, protocols, and rogue plumbers made three points:

1. The web of data shouldn’t require the services of Harry Tuttle.

2. Unfortunately it still does.

3. Fortunately the web’s architecture (a series of tubes) enables Harry to intervene when he must.

Those points are all still valid in 2011. From time to time I still have Harry Tuttle moments. Today’s involved the campus events system at my alma mater, the University of Michigan. In the current phase of the elmcity project I’m rebuilding hubs in various cities in order to dramatically beef up the number of feeds and quality of tagging. Ann Arbor’s hub was conspicuously lacking feeds from the University of Michigan. When I investigated I found that the central service, http://events.umich.edu, does offer iCalendar feeds. Yay! You can’t take that for granted, many if not most schools don’t make their public calendars available in a machine-readable way. Unfortunately there’s a problem with the feeds produced by the UM service. They’re invalid. You can’t load them into an iCalendar-aware program like Google Calendar or Outlook, and the elmcity engine can’t aggregate them.

I reported the problem to central services and have been awaiting a fix for some time. Today, because I wanted to get my hands on that data, I unleashed Harry Tuttle. There are two major problems with the iCalendar export from events.umich.edu. First, the lines of text aren’t properly folded. Second, the timezone properties don’t refer to a timezone definition. So I made a filter that fixes these major problems (plus some other minor ones). Here’s what the filter does:

Original: http://jonudell.net/data/failed-ics-umich.ics.txt. Validation result: Unparseable.

Line-folding fixed: http://jonudell.net/data/fixed-ics-umich.ics.txt. Validation result: Better.

VTIMEZONE added: http://jonudell.net/data/clean-ics-umich.ics.txt. Validation result: Valid.

Then I used the filter to add a bunch of feeds to the hub. Here’s one for the Taubman Health Services Libraries, and another for the Gerald R. Ford Presidential Library. These merge with other library feeds, notably from the Ann Arbor District Library, in the hub’s top-level library view.

Now, where do I fill out that twenty-seven-B-stroke-six form?

From Wikipedia’s Brazil_(film) page:

The reference to form 27B-6, without which no work can be done by repairmen of the Department of Public Works, is a reference to George Orwell, who lived at 27B Canonbury Square Apartment 6, while writing Nineteen Eighty-Four.

If you think that the semantic web is just some kind of geek rapture, like the singularity, I can understand why. As with Zeno’s Paradox we’re always advancing but never arriving. Unlike the singularity, though, I do expect a semantic web to emerge in my lifetime. The latest initiative is schema.org, sponsored by Google, by Yahoo!, and by my employer, Microsoft. Schema.org recapitulates prior efforts to define how webmasters can mark up web pages to include structured data. I hope that this time, thanks to collaboration among the major search engines, we’ll finally cross the activation threshold.

Meanwhile I’ve been toiling in another part of the semantic web. I’ve been trying to get webmasters people to understand why and how to publish calendars as machine-readable structured data in addition to human-readable text. If you’ve followed the elmcity saga you know I’m on a crusade to make more and better use of the Internet’s venerable standard for exchanging structured calendar events: iCalendar.

It’s been a struggle. Almost every website run by a school, club, business, or town has an Events page. Those pages are, almost always, data siloes: HTML or PDF files that can be read by people but cannot be processed by machines. Only rarely do such pages offer links to iCalendar feeds served up by Google Calendar or Drupal or Hotmail Calendar or some other service capable of producing such feeds. So when elmcity curators discover one of these rare feeds, it’s cause for rejoicing.

Sadly that joy is sometimes short-lived. A surprising number of iCalendar feeds just plain don’t work. That’s why I invited Doug Day to create the iCalendar Validator, a service that helps producers of iCalendar feeds conform to the specification. It’s always painful when I have to explain to a curator that the shiny new feed they’ve discovered doesn’t conform and won’t deliver events to the hub.

Here are three sources of iCalendar feeds that, I’ve recently discovered, don’t work.

1. The University of Michigan’s UM Events. It’s a major hub that serves many campus websites. As far as I can tell, all of those sites are providing feeds with malformed descriptions. Here’s an example of the problem and the fix. (iCalendar producer ID: UM//UM*Events)

2. CiviCRM is a “free, libre and open source software constituent relationship management solution.” Its feeds also have malformed descriptions. Here’s an example of the problem and the fix. (iCalendar producer ID: CiviCRM//NONSGML CiviEvent iCal)

3. Drupal is a popular open source content management system. I’ve seen Drupal feeds used successfully, but today I found one that fails for two reasons: a malformed recurrence rule, and a missing timezone definition. Here’s an example of the problem and the fix. (iCalendar producer ID: Drupal iCal API)

These problems are minor and would be easy to resolve. I’ll try to contact the authors of these iCalendar producers; if you can help put me in touch I’d appreciate that. I’m also going to look through the logs written by the iCalendar Validator, compile a list of producers of invalid feeds, and try to contact them as well.

Where’s the connection to the semantic web? At the end of the day, as RSS/Atom validator co-creator Sam Ruby likes to say, “It’s just data.” But structured data, whether it conforms to the dozen-year-old iCalendar standard or some newfangled microdata standard, is easily screwed up. And the consequences of screwups are often silent. Services that were looking for that structured data find nothing, mutter to themselves, and move along.

As we collectively create the semantic web we’ll need to make sure that the structured data we intend to publish really says what we mean.

If you follow this sort of thing, you already know about IFTTT. It’s a new web service that enables non-programmers to compose other web services. The acronym expands to If This Then That, and here are some ways you can use the metaphor:

If I am tagged in a Facebook photo, then save the photo to Evernote

If the library sends email saying my book on hold is ready, send me a text message

My colleague Scott Hanselman says this is bloody brilliant and I agree. It’s the next step in a journey that began for me back in 1999 when I mashed up Alta Vista’s search engine with Yahoo! directories to measure the mindshare of sites by category. More recently Yahoo! Pipes made service mashing easier for non-programmers. Now IFTTT enables everyone to play. Wonderful!

So why am I less enthusiastic about IFTT than I thought I would be? There are two related reasons. First, here’s what IFTTT says when you ask it to activate its Twitter channel on your behalf:

This application will be able to:

• Read Tweets from your timeline.
• See who you follow, and follow new people.
• Update your profile.
• Post Tweets for you.
• Access your direct messages.

Excellent! This is an example of OAuth, a protocol that enables you to delegate powers to IFTTT without giving up credentials. Most of the services you can use with IFTTT support OAuth, and that represents another huge step forward for the web.

What if I only want to give IFTTT the power to tweet on my behalf, though, and not give up access to my private direct messages? More generally, how can I think about the tradeoffs involved in delegating all versus some versus no powers to IFTTT, across a range of services I might authorize it to use on my behalf?

This leads to the second and broader concern. If I’m not paying for the product, I am the product. As is true for many free services on the web today, I have no contractual relationship with IFTTT. I pay for the service it provides by surrendering access to my data. OAuth helps me negotiate how much access, but if I give up none then IFTTT is powerless.

Why, though, can’t I pay for the product instead of being the product? I want IFTTT to work for me, I want to pay for the service, and in return I want it to promise never to keep or use any of the data it exchanges on my behalf. I realize this may not be a popular option anytime soon. But it’s time to start the conversation. Services don’t only want to be free, they also want to be valuable. That’s rarely a choice nowadays. It needs to become one.

Yesterday’s stream of notifications brought two links paired with invitations for me to comment. The first link points to a NY Times story about how AVOS, the new owner of Delicious, plans to remake that service. Chad Hurley:

The home page would feature browseable “stacks,” or collections of related images, videos and links shared around topical events. The site would also make personalized recommendations for users, based on their sharing habits. “We want to simplify things visually, mainstream the product and make it easier for people to understand what they’re doing,” Mr. Hurley said.

The second link points to a blog post from Mark Surman, executive director of the Mozilla Foundation, who wants to teach the world to code:

This has been the premise behind much of what we have done with Mozilla Drumbeat: people who make stuff on the internet are better creators and better online citizens if they know at least a little bit about the web’s basic building blocks. Even if they only learn a little HTML, the web gets better.

I wish I could broker a conversation between Chad Hurley and Mark Surman. If mainstream folk used Delicious and understood what they’re doing when using it, they’d understand themselves to be makers of things on the Internet. The things they make are custom information systems. They make them by writing code, but not in the languages of HTML or JavaScript. Instead they use tag vocabularies that produce and consume web services. And services, I argue, are the most fundamental of the web’s basic building blocks.

Here is an example: http://www.delicious.com/judell/del.icio.us. Ostensibly it’s a list of dozens of articles I’ve written over the years about what I mean when I say that Delicious enables non-programmers to code and use web services. But it’s not just a list. I think of it as a web service. One aspect of the service provides the list in HTML format for people to read in browsers. Another provides the list in RSS format that enables cooperating services to watch the list and react when new items are added. Another enables the list to combine with other lists. Here, for example, is a subset of my Delicious-related articles that are also related to the elmcity project: http://www.delicious.com/judell/del.icio.us+elmcity.

More than anything before or since, Delicious empowers me to manage web resources — both personally and socially. Once those resources were mainly things we found on the web. Now they’re also things we make on the web. I hope the forthcoming Delicious makeover will help people understand it to be a tool for creating, mixing, and sharing web resources. And I hope it remains the sort of open web tool that Mozilla Drumbeat wants to popularize.

Last September we bought a new dishwasher to replace the old one that had failed. It was a reluctant purchase. We’d actually gone a couple of years doing dishes by hand, partly because we’ve been so disappointed by the modern generation of appliances. When the salesman at Sears mentioned their free 5-year preventive maintenance program, though, we decided to opt in. The plan entitles you to an annual appointment with a dishwasher tech who will come by to inspect your machine and do what’s needed to keep it in good working order.

Stoves, dishwashers, and vacuum cleaners shouldn’t require this kind of life support. I’m not going to win that argument, but I can at least make effective use of the life support service. Doing so requires a bit of a hack, though. Why? Remembering something annually, on a certain date, for five years running, isn’t the sort of thing that humans do well. It’s a task that begs for automation.

Of course Sears could remind me, in September of 2011, 2012, 2013, 2014, and 2015, about my dishwasher checkup. It would be easy for them to do. They’re a mega-corporation with a mighty IT department that can easily perform this feat of magic. But Sears doesn’t want to remind me. They’re betting that I’ll forget, as I’m sure most people do.

So they sounded quite surprised when I called yesterday, on the anniversary of my dishwasher purchase, to schedule my appointment. How did I do it? By casting this magic spell:

BEGIN:VEVENT
DTSTART;VALUE=DATE:20110902
DTEND;VALUE=DATE:20110903
RRULE:FREQ=YEARLY;COUNT=5
SUMMARY:sears dishwasher
END:VEVENT

That’s an event recorded by my calendar program. It happened first on September 2, 2011, and will happen again yearly for five years. I cast my spell using Outlook but you can do the same thing using any calendar program: Google Calendar, Apple iCal, Hotmail Calendar, Lotus Notes, many others.

When I talk about iCalendar feeds in the context of the elmcity project, I tend to focus on the idea that these are data feeds, which they are. But iCalendar has a special property. Some of that data is actually code. In this example, here is the one line of code:

RRULE:FREQ=YEARLY;COUNT=5

In iCalendar lingo, RRULE means recurrence rule. From Outlook’s or Google Calendar’s point of view the rule is a tiny program and they are the operating systems that run it. Here are some other RRULES that calendar applications enable you to express:

every Tuesday at 10AM

every first Sunday until October 2011

every September 2 for five years

You don’t have to do it this way. You can, instead, record each event separately. Even though calendar software makes it quite easy to define repeating events, I’ve got a hunch that in the dishwasher scenario a lot of people would use data (five discrete events) rather than code (one event that expands to five). Why? For the same reason that many people will manually adjust the font size used for every paragraph in a document, rather than creating a rule that governs all the paragraphs.

My list of core principles for web thinkers doesn’t yet include the idea of code that generates data. I’ve found it plenty challenging just to get people to think about the nature of their data, about how they do or don’t control it, and about how it does or doesn’t flow through networks.

But if we want everyone to be able to automate tasks that otherwise get done poorly or not at all, we probably do need to find a way to teach patterns like code-generated data and Don’t Repeat Yourself more broadly than just to students of computer and information sciences.

If you want to read a lovely short essay on calendar software and our relationship to the future, I highly recommend Paul Ford’s Tickler File Forever.

As the environments in which we live and work become augmented by networked information systems, it behooves us to learn something about how those systems work — and about how to work those systems. “But I don’t need to know how my car works,” you say, “why do I need to know how my computer works?” You don’t. That’s the wrong analogy.

Our civilization’s enabling infrastructure exists only because we’ve created many different kinds of specialized knowledge, and because we — the knowers and practitioners of these specialties — participate in networks of exchange. It’s a good thing that most people don’t need to know how to repair faulty disc brakes or laptop hard drives.

It would be really bad, though, if most people didn’t know anything about the physical laws that keep the car’s rubber on the road, or the social laws that govern traffic intersections. That’s the level at which people need to know about networked information systems.

A lot of what the creators of those systems know isn’t at that level. A few things are, though, and I’ve been working up a list of things creators know that users could and arguably should know. From time to time I think of adding another item to the list. So far, though, everything that comes to mind is already, in some essential form, on the list.

The latest example is a rule known as Don’t Repeat Yourself. Programmers invoke DRY when they squeeze redundancy out of code or data. Where does the duplication come from? Information systems, like human cultures, evolve by copying parts of themselves — and of one another — and then by modifying the copies. That’s the natural and healthy trend toward diversity. But systems and cultures also evolve by converging on a core of shared commonality. That’s a natural and healthy countervailing trend. The most successful copies need to merge, at the right level of generality, with the core.

These two styles are in dynamic tension. On the C2 wiki, where Ward Cunningham has for many years hosted a conversation about the core principles that govern information systems, the topics DontRepeatYourself and PrematureGeneralization express that tension. It’s good to DRY things out, but bad to generalize prematurely.

Does that rule belong on the list? Maybe. I wish programmers weren’t the only ones feeling and responding to the tension between DRY and WET¹. Anyone who aims to automate work, for example, needs to learn how to walk the WET/DRY tightrope.

But maybe wringing out redundancy isn’t the essence of DRY. Here’s what the noted software pragmatist Andrew Hunt says on the DontRepeatYourself wiki page:

Every piece of knowledge must have a single, unambiguous, authoritative representation within a system.

That’s an assertion about the value of authority, not the problem of duplication. And the first rule on my list already says: “Be the authoritative source for your own stuff.” In the context of the elmcity project, which is the incubator for the list, that means if you’re promoting a calendar event online, you ought to publish a single authoritative version of that calendar entry to a system that’s accountable to you. You shouldn’t have to repeat yourself by making copies of the calendar entry in systems that aren’t accountable to you.

When you’re the authoritative source for some set of facts, and when you inject those facts into the networked information system called the web, Don’t Repeat Yourself is a great principle to keep in mind. I’ll be delighted if you do. But it’s the underlying core principle of authority that I most want you to embrace. When you’re the authoritative source for your own stuff, you’ll naturally tend to stay DRY.

¹ There’s no agreed-up expansion of DRY’s opposite, WET. Here’s mine: Watch, Emulate, Transcend.

I’ve been mulling Michael Schrage’s recent essay, Why You Should Automate Parts of Your Job to Save It, since I read it last week. Here’s the conclusion:

What is the most important thing you do on your job? What portion of that could be turned into an app that anyone in your organization could effectively use? What portion of that could be automated and fed directly into the larger system with only minimal review by you? What’s the least valuable but essential part of your job? Why aren’t you figuring out ways to automate it on your iPad or Android?

People with the best answers will likely discover they also have the best job security.

I agree with the premise. But something kept bugging me about the argument and today I realized what: the gadget focus. We’ve seen this before. Remember when computers in the schools were the answer? Now it’s smartphones and tablets in the workplace. But these are all just access devices. We focus on them because they seem more real than the networks they connect us to. It’s easy to see that devices made of metal and plastic are tools. It’s much harder to see that networks made of data formats and application protocols and communication topologies are tools. But information networks matter more than the devices we use to access them, or the applications that run on those devices. The key to the automation of knowledge work that Schrage righly prescribes isn’t learning how to use smartphones or tablets. Rather, it’s learning and then applying core principles that govern information networks.

Sadly we don’t teach these principles. Not even, in any systematic way, to information technologists. And certainly not to the bank loan officers and nurses and “iPad-wielding waitresses” in Schrage’s essay. Can it be done? I don’t know but I think I’d enjoy trying.

A couple of years ago, when I needed to call someone, my phone wasn’t where I thought I’d left it. This was a problem. Back then I often called this person but I had no idea what his phone number was. Why would I? Remembering numbers is a job I delegate to my phone.

And finding my phone when I forget where I left it is a job I delegate to other phones. But when I reached for my cordless office phone it wasn’t where I thought I’d left it either. No problem. Finding my cordless when I forget where I left it is a job I delegate to its base station which is tethered and can’t wander.

So I walked over to the base station, paged the cordless, found it under a pile of papers, used it to call my cellphone, found it under another pile of papers, and used it to call my friend. Then I paused to reflect on technological augmentation.

Now don’t get me wrong, augmentation is a wonderful thing. Even fish, we have recently learned, use tools. I love delegating mental chores to devices and to the cloud. I am not a Luddite. But I am starting to think more about when not to delegate, and why not.

My mom is almost 90. She’s always known the numbers of the friends she often calls, and she still does. That’s a good thing because macular degeneration makes it impossible to read the numbers in her phone’s directory and difficult even to read the numbers she’s printed extra-large in her address book.

I’m not planning to memorize phone numbers. But I am exploring ways to usefully exercise the memory muscle. Just because I can look everything up doesn’t mean that I always must. Some kinds of things are worth keeping in the cache. But which?

In Behind the Dream Clarence Jones, Martin Luther King’s lawyer and adviser, writes:

What amazed me was that there was absolutely no reference material for Martin to draw upon. There he was [in the Birmingham jail] pulling quote after quote from thin air. The Bible, yes, as might be expected from a Baptist minister, but also British prime minister William Gladstone, Mahatma Gandhi, William Shakespeare, and St. Augustine.

If there’d been a web in 1963, and if MLK could access it from that jail, would the Letter from a Birmingham Jail have turned out differently? Would the differences have mattered? I’m not sure. It’s interesting to note that the quotes Clarence Jones seems to recall being in the letter aren’t all there. I don’t find Gladstone, Gandhi, or Shakespeare. I do find, along with St. Augustine, Socrates, Thomas Aquinas, Paul Tillich, Abraham Lincoln, Thomas Jefferson, T.S. Eliot and others. Does the discrepancy matter? Not really. What Jones remembers, and what I will now remember, is that MLK could remember, not precisely what he did remember.

Clarence Jones continues:

I have often said the sheer processing power of Martin’s mind left me awestruck. His dexterity with memory and words ran along the lines of the cut-and-paste function in today’s computer programs. The “Letter from a Birmingham Jail” showed his recall for the written material of others; his gruelling schedule of speeches illuminated his ability to do the same for his own words. Martin could remember exact phrases from several of his unrelated speeches and discover a new way of linking them together as if they were all parts of a singular ever-evolving speech. And he could do it on the fly.

Jones tells us that it was he who wrote the draft of I Have a Dream speech. And that Martin Luther King began to deliver it more or less as written. Then Mahalia Jackson called out, “Tell ’em about the Dream, Martin.” So he did. He set aside the text that Jones had written, and that he had edited the night before, and gave an impromptu speech for the ages. Among many factors that conspired to make that possible were the processing and memory powers that Jones describes.

As our relationship to devices and the cloud reorganizes our brains, those powers are changing. But we’re not just passively experiencing these changes. We are actively co-creating the powers of our individual brains and of the collective brain we’re becoming.

When I was a kid I used to memorize poetry. A couple of years ago I was feeling no need to do anything like that ever again. But now I find myself wanting to exercise the memory muscle. I’ve decided that certain kinds of things belong in the unaugmented part of my brain. Among them, for me, right now, are fingerstyle guitar arrangements. My repertoire has grown slowly over the years, and it’s been a challenge to learn new arrangements. Lately I’ve been working to improve my ability to memorize them, though, and the effort feels intrinsically rewarding.

I’ve also noticed that one of the best talks I’ve ever given was the product of careful memorization. I think I see now why Ignite talks and Moth stories are done live without notes. I love having an outboard brain. But until we get the implants, we’ll want to keep the onboard one humming.

When I left the pageview business I walked away from an engine that had, for many years, manufactured an audience for my writing. Four years on I’m still adjusting to the change. I always used to cringe when publishers talked about using content to drive traffic. Of course when the traffic was being herded my way I loved the attention. And when it wasn’t I felt — still feel — its absence.

There are plenty of things I don’t miss, though. Among them is the obligation to be an aggressive early adopter of (and opinionator on) every new tech fad. Now I can hang back, wait for the fads to spread beyond the geek echo chamber, and watch how my civilian friends, family and acquaintances react to them. Since none of the civilians I know have moved to Google+, I can’t gauge their reactions yet. While waiting for some of them to jump into the pool I’ve dipped a toe in the water, considered my own reaction to the New Thing, and compared it to the collective reactions of the geek tribe.

Mine seems atypical: I’ve reached into a corner of my closet, pulled out the RSS reader I left there, and used it to find nourishment that online social networking seems no longer to provide. Last night’s 17-course meal was a selection of recent essays by Gardner Campbell, Brian Dear, Lorianne DiSabato, John Faughnan, Paul Ford, Cliff Gerrish, Ned Gulley, Eugene Eric Kim, Adina Levin, Hugh McGuire, Cameron Neylon, John Quimby, Antonio Rodriguez, Scott Rosenberg, Doc Searls, Ed Vielmetti, and Ethan Zuckerman.

These writers are among many who write because they want to and because they can. They write in their own online spaces which I follow in my RSS reader. When I seek nourishment from them I can go directly to their spaces. No business models drive me there. Often, to be sure, I have been led there by way of a comment on one or another of the social networks. That had become so common that I came to accept a lot of distracting chatter as the price of discovering things to read. But Google+ seems to be the camel’s-back-breaking straw. The price has gone too high. So I’m rediscovering what made the blog network so thrilling to me a decade ago: unmediated access to people writing for the love of it in their own online spaces. Distracting chatter has its uses. But it’s optional.

Guy Sorman’s The End of Green Ideology begins:

The meltdown at the Fukushima nuclear plant has sent political aftershocks racing around the globe. More often than not, however, the shocks have been ideological, with no basis in science.

I’ve been reexamining the case for nuclear power for a while now. I still think we’ll probably need to keep developing it, improving it, and including it in the mix. But I’m willing to be convinced otherwise and I’ll watch Germany’s forthcoming nuclear exit with great interest. In any case, I agree with Guy Sorman: let’s make sure our energy future is based on science not ideology.

So I was puzzled to find this assertion in the middle of his critique of green ideology:

If California were to rely on solar power for its electricity consumption, the entire state would have to be covered with photovoltaic cells.

Really? That seems wrong. According to an infographic called Surface Area Required to Power the World (with zero carbon emissions and with solar alone), assumptions and methodology here, it looks like a California of solar panels could more than power the world.

For reasons Sorman alludes to — cost, environmental impact — it probably wouldn’t make sense to build a California of solar capacity, regardless of how it’s distributed around the world. But let’s just focus on the assertion that California would need all of itself to keep the lights on using solar photovoltaics. Is that really wrong? If so, how wrong? What’s the right answer? And how can we find out?

Before the advent of WolframAlpha I would sometimes idly speculate about these kinds of questions, but I wasn’t able to effectively analyze them. Now I can. From various sources, I think I know that a PV panel can produce on the order of 10 watts per square foot. So we can ask WA directly:

area of california in square feet * 10 watts per square foot = 45.64 TW

How can we think about 45 terawatts? I recall from Saul Griffith’s Climate Change Recalculated that the whole world runs on about 16TW. This confirms what I gleaned by eyeballing the infographic: Solar California could more than power the world.

But where does Saul’s 16 TW number come from? Can we cross-check it? The infographic assumes that we were running at 500 quadrillion BTU in 2008, projected to rise to around 680 quadrillion BTU in 2030. The US Energy Administration’s International Energy Outlook 2010 cites similar numbers. I’m not good at unit conversions, but WA is:

500 quadrillion btu in watt years / 1 year = 16.73 TW

This matches so closely that I wonder about the provenance of the number. Do all these analyses use the same source, or are there independent estimates that converge? But anyway, let’s assume that 16 TW is in the ballpark. How much electricity does California use? (Recall that this was the quantity of energy Guy Sorman thinks would require a California-sized solar panel to produce.) A couple of sources say that California uses about a quarter million gigawatt hours of electricity in a year. How much is that? WA provides useful comparisons:

a quarter million gigawatt hours = 1/15 total production of electrical energy in US in 2001 = 2 * electricity consumption of Norway in 1998 = 28 * daily electric energy production of all nuclear power plants

a quarter million gigawatt hours / 1 year 2000 hours = 28.5 125 GW

45 TW is about 1600 360 times more than that, so I think that’s how wrong Sorman’s claim is.

If a California-sized solar panel could produce 45 TW, then how much of California would be needed to produce 28.5 125 GW? Let’s ask:

area of California / 1600 360 = 102 455 square miles = 1/15 1/3 area of Rhode Island = 2.2 * area of Walt Disney World 1.1 * land area of Hong Kong

As with other essays in my energy literacy series I am mainly interested here in how tools like WolframAlpha can help us reason our way through a thicket of quantities and unit conversions. Hundreds of people liked Guy Sorman’s essay on Facebook. Dozens tweeted it and the one comment I can retrieve calls it a “balanced and scientific assessment of world’s current and future energy outlook.”

I don’t know enough to agree nor disagree with Guy Sorman’s conclusion:

Future energy supplies will most likely rely more and more on miniaturized nuclear plants and shale gas — a mix capable of responding to a rapidly urbanizing world population’s growing demand for electricity.

But I do know that we need to be able to verify and reproduce analyses that purport to be scientific. WolframAlpha has become the View Source button that helps me do that.

This National Geographic video about 3D printing exemplifies the worst kind of gee-whiz reporting. Just scan a crescent wrench, print it, and bingo, you’ve copied a real tool with moving parts!

Not.

A commenter notes differences between the copy and the original and concludes:

If the real wrench was simply scanned, this would not have happened. A human has built the design data.

3d printing is cool, why do they feel they have to lie about the input method?

The input method is, of course, 3D CAD. From the product brochure for the printer:

Z Corp.’s 3D printing technology leverages 3D source data, which often takes the form of computer-aided design (CAD) models.

Gee-whiz reporting insults our intelligence and trivializes its subject matter. It’s fun to imagine a magic replicator, but it’s more interesting to know about the human/computer interaction that makes real replication possible.

Once I wrote a review of a dozen 3D CAD programs for BYTE Magazine. The benchmark was a model that I commissioned an architect to design. We called it the BYTE Pantheon and it looked like this:

My job was to construct that model in each of the dozen CAD programs. It was hard! That was partly because I had no prior experience with CAD software. But it was also because each program had its own way of using 2D gestures to manipulate 3D objects. That was my main takeaway from the project. There wasn’t (and I think still isn’t) a standard suite of gestures. Even if there were, and even (I suspect) when you can use 3D gestures, it would still be hard because you are making a precise description of a complex object. Wikipedia calls CAD an “industrial art” for good reason. Models with the same functional qualities can differ in terms of style and sophistication. Those differences come into play when the model is shared and modified. Or so I imagine, anyway. I’m not a 3D modeler but I understand 3D modeling to be a process akin to programming.

A friend of mine, Gary Spykman, describes himself as a designer, furniture maker, and artisan. He has also become a 3D modeler, and uses SketchUp to explore his designs and render them for clients. A couple of years ago, Gary designed and built what he calls his cabana. Here it is as designed in SketchUp, and as built in Gary’s back yard.

When I saw what Gary was doing in SketchUp I was inspired to try using the program for some simple needs of my own — to visualize my geodesic tomato suspension dome, and more ambitiously to visualize a remodeling of our kitchen. And you know what? It was just as hard as I remembered! If you do this stuff for a living, as Gary does, then it becomes second nature. But if you only do it occasionally, like me, you’ll be impressed every time with the level of skill required to precisely describe an object or a scene.

Like the commenter on YouTube I have to ask: why lie about this? National Geographic’s gee-whiz reporting doesn’t just fail to inform. It also fails to celebrate the synergy between computational power and human skill that makes 3D modeling so fascinating.

Last week I made my own modest contribution to the growing literature on how to prepare for an Ignite talk. Now the talk is up on YouTube. It’s about barefoot running and whether we really need an app for that.

Recently I did my first Ignite-style talk[1]. It’s an interesting format: a 5-minute 20-frame slideshow set to auto-advance every 15 seconds. The format has its roots in the 5-minute lightning talks that I remember from early Perl conferences. In lightning talks the slides were optional, you just had to finish on time or get gonged. (I can still see Larry Wall cheerfully ringing the gong on others and just as cheerfully having it rung on him.) Ignite makes slides mandatory. The 15-second cadence invites you to think in stanzas; it’s a nice constraint to embrace.

The lore on how to prepare for these talks also has roots that connect Mark Fowler’s 2004 Giving Lightning talks to Jason Grigsby’s 2008 How to Give a Successful Ignite Presentation to many others more recently. Everyone agrees: practice. But how?

For mine I wrote a script in 20 stanzas and then set about tuning them to the required intervals. My first thought was to refer to a clock while reading the script aloud and editing it, but it was hard for me to look back and forth between the clock and the script. So I made a couple of audio tracks for timing. countdown-slide.mp3 says “3, 2, 1, slide” and then every fifteen seconds, “slide” again, finishing with “end.” countdown-5-10-slide.mp3 adds “five” and “10” at the appropriate intervals. Both turned out to be helpful in different ways.

For tuning the written script to the intervals, I used the countdown-5-10-slide track which gave me plenty of cues to help gauge the edits. For practicing the script I used the countdown-slide track which emits exactly the stream of cues you get in the talk: start, a cue every 15 seconds, then stop.

As is often appropriate for an Ignite talk my slides were mainly pictures not words. I tried to search for and use only images licensed for sharing, but the discoverable pool of such images isn’t nearly broad or deep enough. So I fell back to grabbing images from Google and Bing searches, feeling guilty about that, and wondering what it will take to make the pool a lot broader and deeper.

I wasn’t sure at first I’d need the countdown-slide track — the one that just says “slide” every 15 seconds. If you practice and memorize the script while watching the slides then you’ve got your cues, no need for audio timing. But after a few run-throughs I got antsy and wanted to go for a hike. That’s where the countdown-slide track really worked, in a couple of ways. It not only provided the cues, it prompted me to visualize the slides. From then on I could practice anywhere I wouldn’t mind being seen talking to myself: running, hiking, driving to the airport. I hardly used the slides again.

When I did use the slides, in a few practice runs and then in the talk, I saw how helpful it had been to have visualized them, but not seen them, while practicing. I’d learned to do the talk from memory without the slides. Doing it with them was, by comparision, easier.

This makes perfect sense, of course. I think it’s related to how musicians memorize music: hear the tune, see the notes on the page, feel your fingers on the instrument. Then selectively omit the audio track, the sheet music, and even the instrument, and do the hearing, seeing, and feeling in your mind.

A couple of weeks later I was on a panel where I had up to 10 minutes to speak. I wound up using only 4 minutes, and while there weren’t any slides, I still wrote it out as a story told in memorizable stanzas. I think it turned out better than if I’d used the whole time for something less tightly constructed.

The 4-minute panel talk turned out to be harder to learn than the 5-minute Ignite talk, and now I see why. Even though slides weren’t required, I could have used them as practice cues! I guess that’s what Joshua Foer and other memory experts keep telling us: divide things into chunks, tag the chunks with pictures.

[1] The subject was barefoot running. I think it’ll be posted soon.

Yesterday I joined a panel at the New England Conference of Public Utilities Commissioners. I was the odd man out in the group but that was the point. The organizers wanted somebody other than the Usual Suspects to bring an outside perspective to the panel. Here’s roughly what I said.

My phone bill is itemized down to the last minute and second — every call, every number. For the longest time I’ve wanted my electric bill to be itemized in the same way, right down to the watts actually used by every appliance.

How can you decide whether to replace your old fridge if you don’t know what it really costs to run it? Or how guilty should you feel, or not feel, about turning on the air conditioner in June, if you don’t have the data?

Well I’ve finally got the data, at least sort of, thanks to the smartmeter I installed recently. It’s a first-generation device; it’s a little flaky; it doesn’t track individual appliances. But it does give me realtime feedback about how many watts my house is burning.

That’s a really interesting number. When you turn off some lights, you see the difference right away, in watts and in pennies per hour. It’s a powerful behavior changer.

Around the time I was installing this thing I heard a commentary on NPR about smartmeters, The title of the piece was “Smart Meter, Big Brother,” and you can guess what it was about. Smartmeters are an unpredictable new technology, they bring unforeseen privacy risks that none of our statutes and regulations have ever thought about.

What was the guy worried about? Well, suppose you come home every night around the time the bars close. The electric company can tell because it sees your lights and appliances come on. Then somehow it leaks that data to your insurance company, which raises your premiums.

Really? I don’t know, to me there’s nothing new here. Most people I talk to have given up on privacy. They just expect that’s what would happen with your data.

Here’s what would be a new twist. Why do we just assume that a smartmeter will automatically phone its data home to the electric company? Mine doesn’t. It only feeds data to my own home computer, and from there to wherever I route it. PSNH doesn’t know anything about this. [ed: Well, I guess they do now :-)]

This reminds me of what we’ve been seeing in IT for quite a while. Consumer technologies at the edge of the network have disrupted enterprise technologies at the core, Telecom feels this disruption intensely right now. I’m wondering if other utilities, like power, will start to feel it too.

I know a guy who runs a company that does demand management for big box retailers like Michaels and Petco. He drops a package of instrumentation and controls into your store, he connects it to the web, and then when there’s a rolling brownout in California he can dial down the lights and ventilation and AC in all the buildings across his network.

He picked the retail sector because every one of those stores has the energy footprint of 20 or 30 homes. And he avoided the residential sector because it’d be a lot harder to have the same kind of impact there.

But now I’m wondering if we’re going to start to see the crowdsourcing of demand management. I’ve already got a do-it-yourself Internet-connected smartmeter. How much longer until I can add DIY controls to dim the lights or schedule the dishwasher to run off peak? And then how much longer before I can connect up with a bunch of other people who want to do the same things?

I’m not saying this will happen. But it could. And if it did, we wouldn’t need to wait for legislators and regulators to figure out how to deal with some new privacy threat, because there wouldn’t be a threat. It would just be people choosing to share their data with other people in order to conserve energy. And, by the way, they’d be having a lot of fun doing it too! Think massively multiplayer online game for the smart grid.

Now maybe I’ve told the wrong crowd about my DIY smartmeter. Maybe I’ve already broken some rule I don’t know about. But if not, or in any case, I’d like to put this crazy idea onto the table for discussion.

For me it was a chance to hang out with a bunch of public utility regulators and watch them wrestle with thorny issues. Does regulation often stifle innovation? Yes. Is regulation a means to socially just ends, like rural broadband? Yes. I found it fascinating.

Also, I got to spend an evening and a morning at the fabulous Mount Washington hotel during a perfect New Hampshire solstice.

I guess because I lost my own dad a few years ago, I was really moved by Sunday’s outpouring of Father’s Day sentiment. On Twitter, on Facebook, on blogs, everywhere you looked online you found people sharing an “I miss my dad” moment. Then today, while searching YouTube for renditions of a beautiful Hawaiian slack key guitar tune I’m learning, called Moe ‘Uhane, I found this wonderful version. And even better, this comment:

I was there when he taped this on my little tape recorder back in the 80’s. I was sleeping and woke up to find him playing still in his pj’s. Miss my dad. Thanks for representing. Sounds awesome!

– Mahina Chillingworth

Sweet! The tune came to Sonny Chillingworth in a dream, and then his dreaming daughter woke up to hear him recording it.

Ernest Hebert is an author and Dartmouth professor who was born in Keene and writes fiction about this area. During an interview with the Keene Sentinel he unloaded a gorgeous rant about William Faulkner. It’ll soon vanish behind the Sentinel’s paywall. So as a public service, and to ensure I can refer to it later, I’m placing it here for safekeeping.

Q: How do you feel about being compared to William Faulkner?

A: I hate being compared to Faulkner — this kind of uppity, snooty southerner with his turgid prose based more or less on the Bible. I can’t bear to read Faulkner. It makes me want to puke. I just loathe Faulkner. And you can quote me on all of that.

Done!

The WSJ reported recently that the FBI, looking for fresh leads in the 1982 case of Tylenol poisonings, suspects Ted “Unabomber” Kaczynski and is trying to get hold of a sample of his DNA. Coincidentally I was just thinking about that case thanks to Bruce Schneier. In his recent TED talk he mentions that the Tylenol incident led to tamper-proof caps — a perfect example of what Schneier likes to call “security theater”:

As a homework assignment, think of 10 ways to get around it. I’ll give you one, a syringe.

So far this is typical Schneier. It’s a great point, but one I’ve heard him make many times before. In the next sentence, though, he breaks new ground:

But it made people feel better. It made their feeling of security more match the reality.

Bruce Schneier used to mock the theatrical dimension of security. Now it seems his thinking has evolved — and in a really interesting way. He’s alway viewed security in a relativistic way, and as a game of economic tradeoffs. Here he twists the lens to bring something else into focus: the relationship between how secure we feel and how secure we are.

We know that modern hominids evaluate risk poorly:

One way to think of it is that we’re highly optimized for risk decisions that are endemic to living in small family groups in the East African highlands in 100,000 B.C. — 2010 New York, not so much.

There are two ways our feelings about our security can disconnect from reality. We can have a false sense of security, as when we underestimate the risk of automobile travel. Or we can have a false sense of insecurity, as when we overestimate the risk of air travel.

What matters most, Schneier suggests, is that we align feelings and reality. And if security theater helps us do that, then it’s a perfectly good thing. The Tylenol episode represents the kind of risk that the media thrive on and magnify. People felt unsafe even though they were astronomically more likely to die in other ways. Tamper-proof caps may be a purely theatrical response, but that may still be useful.

Technologists too rarely display this kind of nuanced thinking; it’s refreshing to see it.

The elmcity project thinks like the web, and invites users to do the same. Direct users of the elmcity service (hereafter just elmcity) are curators who run calendar syndication hubs. Indirect users are people who contribute calendar feeds that flow through the hubs. Everyone involved in this dance practices key principles of web thinking. Here are some ways:

In these scenarios the elmcity service partners with a number of other services, most notably Delicious. During last December’s Delicious-is-going-away scare I compiled a list of essays I’ve written since 2004 about how and why Delicious is far more useful than the tagline “social bookmarking” might suggest. Here’s the gist: Delicious, more than any other online service I know, encourages and rewards the practice of web thinking. That makes it a natural partner of elmcity, a service that invites communities to think like the web.

There are, of course, other ways to manage hub settings and feed registries. Various social bookmarking services can do what Delicious does. More conventionally, the elmcity service could manage this data directly. I’ve always been prepared to swap out Delicious for something else, and when it looked like Delicious might go away I reviewed my options.

Then came news that Delicious had been acquired by YouTube’s founders and would continue as part of a new company called AVOS. The transition promises to be seamless. In order to continue using their existing accounts and data after July 2011, Delicious users need only opt in to the AVOS terms of service.

For typical users it’s all straightforward, though you’ll want to note that AVOS declines responsibility for backing up your contributed data. I’m glad to see that in writing. Too many people assume, wrongly, that the data they send to free cloud services, in order to socialize with other people and other people’s data, will be backed up. It’s the cloud, right? The cloud can back up your stuff, right? Yes it can, but don’t expect that to happen unless you’re paying for it.

The elmcity service isn’t a typical user of Delicious, though. It accesses Delicious programmatically, scanning curators’ bookmarks several times a day in order to check for updates to their hub settings and feed registries. That may not square with this item from the list of Things Not To Do in the AVOS terms of service:

Attempt to access or search the Service or Service Content or download Service Content from the Service through the use of any engine, software, tool, agent, device or mechanism (including spiders, robots, crawlers, data mining tools or the like) other than the software and/or search agents provided by AVOS or other generally available third party web browsers.

That’s quite a change from the comparable paragraph in the existing terms of service:

Delicious provides access to portions of Delicious via RSS feeds and an API; for the purposes of these Delicious Terms, such access constitutes use of Delicious. Delicious asks that you use these features respectfully, as outlined in the documentation.

Elmcity actually makes very little use of the Delicious API. But it relies heavily on the fact that for every HTML page that Delicious presents, there is a corresponding RSS feed. For example, curators add iCalendar feeds to their hub registries by bookmarking the URLs of those feeds, and tagging them in a certain way. As curator of the hub for Keene, NH, I review my hub’s registry using this Delicious query:

http://delicious.com/elmcity/trusted+ics+feed

When the elmcity service checks for updates to my registry, it uses the corresponding RSS feed:

http://feeds.delicious.com/v2/elmcity/trusted+ics+feed

That same feed serves another purpose. The elmcity support forum and project collaboration space is on FriendFeed. One of FriendFeed’s endearing features is ad-hoc RSS aggregation for groups. The elmcity group on FriendFeed compiles project status from a variety of RSS feeds, including Delicious feeds of the form shown above. What that means is that when any curator discovers a new iCalendar feed and adds it to his or her hub, everyone in the FriendFeed space is notified. It’s a wonderful example of how pub/sub networks can enable ambient awareness.

I hope that AVOS wants to celebrate, not forbid, this kind of emergent capability. But it does require accessing the service with tools and services other than “generally available third party web browsers.” I have always done this kind of access respectfully, and would like to continue on that basis. But if that won’t be acceptable I’ll make other arrangements. Either way I want to thank Delicious for many years of stalwart service and creative stimulation, and to thank AVOS for carrying on. I’m sending the Delicious/AVOS support team a link to this post, and I’ll use comments here to report what I find out.

I love small airports. When we moved to Keene, NH, over 20 years ago, I could roll out of bed, drive three minutes to our town’s airfield, hop onto a flight to LaGuardia, arrive in midtown Manhattan for a day of meetings, and be home for a late dinner. It was a good thing that didn’t last. The federal subsidy that kept those flights going ran dry, and Keene’s commercial air service ended long ago.

Years later I read James Fallows’ Free Flight and fell in love with its vision of an alternative air travel system that thought like the web: a decentralized and fluid network of interoperable resources.

Still later an old acquaintance, Ed Iacobucci, made a valiant effort to bring that vision to life. Our 2007 conversation about DayJet, the on-demand regional jet travel service Ed had then just launched, inspired me to hope that I might again fly to and from the Keene airport. Our hopes were dashed by the 2008 economic collapse, and now that vision is on hold. But last night I had a taste of what air travel once was and might someday be again.

I’m in Toronto for a couple of days. On my last two visits, not knowing another way, I used Toronto’s international airport, Pearson. It’s a typically unfriendly major hub, made even less friendly by its atypical lack of rail service into the city center. But on my last trip, during a morning lakeside run, I saw small planes landing at the city airport on Toronto Island within a stone’s throw of downtown. I wondered what it would be like to arrive on one of those flights.

Well, now I know. It’s delightful! I flew Porter Airlines from Boston Logan direct to Toronto Island Airport. It was a trip from a bygone era. The plane was a turboprop. The complimentary wine was served in a real glass. The customs queue was quick. The ferry crossing to downtown, over a hundred yards of water, was seamless. Then I simply walked, less than a mile, to my downtown hotel. True, it cost a hundred bucks more than flying into Pearson. But I saved almost that much money, plus a chunk of time, by not cabbing from Pearson. And I arrived refreshed and happy. When does that ever happen?

Porter Airlines isn’t the kind of service that James Fallows’ book imagined and Ed Iacobucci’s company created. But the retro 1950s-era experience that Porter offers may help to remind us that small airports are everywhere, waiting to offer pleasure and convenenience once again — if we can find ways to reactivate them.

I recently asked a web server for something and it responded like a teenager:

403 Forbidden: The server understood the request, but is refusing to fulfill it.

I thought the description of this status code was a joke, but no, it’s right there in the HTTP specification. Here are some others that you won’t find there:

200 OK: Are you happy now?

202 Accepted: I’ll do it tomorrow.

206 Partial Content: I’ll do the rest of it tomorrow.

301 Moved Permanently: It’s not my job.

305 Use Proxy: I can’t remember, ask mom.

401 Unauthorized: You’re not the boss of me.

404 Not Found: Why does everybody always ask me for that?

410 Gone: Zzzzzzzzzzzzzz.

417 Expectation Failed: I thought you were going to put gas in the car.

500 Internal Server Error: I’m not allowed to make a mistake?

The most interesting app on my phone at the moment is called PowerNow. Here’s the display:

At 12:14AM today, when I took this snapshot, my house was using 1090 watts. The data comes from the TED5000 smartmeter I recently installed. It reports all kinds of data but that single number matters most. When you can monitor your power use in realtime, you can regulate it far more intelligently than you can if your only instrument is a monthly bill.

The other day, for example, I finally got around to swapping out a bunch of incandescent bulbs for compact fluorescents. I took a baseline measurement with PowerNow, and then swapped out a total of 17 bulbs in six batches. After each batch I took another measurement. You can see all the data here. Step by step the wattage drops, by a total of 900. And step by step the hourly cost drops, by a total of $0.15/hr. Of course I won’t really save $0.15/hr because I took the baseline with everything turned on, which isn’t normally the case. But it’s wonderfully motivating to be able to measure the effect directly.

The TED5000 comes with a wireless display that I could have used to take those snapshots, but its ZigBee controller isn’t working well and I’m awaiting a replacement. So I turned to Google PowerMeter, which can receive data posted from the TED. But PowerMeter only reports cumulative use, not realtime use. When I put up my own experimental service to receive TED data, I found out why: cumulative data is all that the TED sends.

There are actually two TED APIs, though. The one used by PowerMeter, and by other services like the one I wrote, requires an activation protocol and then accepts data packets sent from the TED gateway. In other words: TED pushes to the service.

There’s also another API that enables a service to pull data from TED. It’s just this:

http://ted5000/api/LiveData.xml

This URL refers to a TED gateway that’s hooked into a home network which resolves the name ted5000 to some private address like 192.168.2.100. There are variant URLS for fetching historical data but this URL just brings back the realtime data which is all I want for now.

One way to view that data from anywhere would be to open up an inbound port on the router. But I’d rather not, so instead I’m running a scheduled process on one of my networked PCs. It polls the TED, extracts the one number I want, and posts it to a web page along with the date and time.

If I were in marketing, here’s how I’d describe this “cool app”:

Monitors realtime power use in your home!

Works anywhere, anytime!

Compatible with all smartphones, including iPhone, Android, BlackBerry, and Windows Phone!

And that’s all true. But of course it isn’t really an app, it’s just a web page backed by a suite of cooperating services: the TED’s software, my data extractor, and the website that hosts the page.

Or is it, in fact, an app? For me the word connotes not just software that runs on your phone, but rather software that runs in various places including your phone, and ties together various services.

Apps defined that way can be ridiculously easy to write. I tweeted that PowerNow is just 6 lines of Python. I went back and checked and, well, it’s a bit more, but not much more:

import urllib2, re, ftplib, datetime
from ftplib import FTP
from datetime import datetime

url = 'http://ted5000/api/LiveData.xml'
s = urllib2.urlopen(url).read()
power = re.findall('<PowerNow>(\d+)</PowerNow>', s)[0]

html = """<html>
<head><title>PowerNow</title></head>
<body>
<p style="font-size:x-large;text-align:center">%s</p>
<p style="text-align:center">%s</p>
</body>
</html>""" % (power, datetime.now().strftime('%A %B %d %H:%M'))

fname = '/users/jon/dev/PowerNow.html'

f = open(fname, 'w')
f.write(html)
f.close()

ftp = FTP('_______.net')
ftp.login('_______','________')
f = open(fname)
ftp.storbinary('STOR PowerNow.html', f)
f.close()
ftp.quit()

That’s the script that polls the TED, extracts the current power use in watts, packages it in HTML, and posts it to a webserver. To run it, I scheduled a once-per-minute process using the Windows 7 Task Scheduler. I hadn’t used Task Scheduler in a while, and I noticed something non-obvious about it. When you specify repetition, it looks like you only have these choices:

What’s non-obvious is that you can edit the values in the list, like so:

I also chose the One time setting:

That’s how you set up a task to repeat with arbitrary frequency. Then you tell it what to do. In my case:

I avoid home automation projects that require soldering, it just isn’t my thing. But stitching web componentry together? I can do that in my sleep. I’m really looking forward to the coming wave of appliances that will enable me to do that.

When I wrote about my experience with the Kill-A-Watt, commenters alerted me to the next level in the real-time power monitoring game: gadgets that watch the whole house instead of one appliance at a time. Back then the only solutions I could find were for UK power systems, not US power systems. But recently I found The Energy Detective and ordered a TED 5000 straightaway.

The bottom line is that it works. I’ve got a real-time display showing how many watts my house is burning at any given time, along with the equivalent hourly cost. When I turn off lights or appliances that don’t need to be on, I can see the difference. There’s much more data available, but continuous instant feedback is the essential thing. It’s a powerful behavior changer.

Unfortunately my TED system isn’t yet working as well as it should. One of the components is ailing. It’s the gateway that receives data from the transmitter I put in my distribution panel, and sends it along to the wireless display and to my LAN. The gateway’s local webserver won’t respond to requests, and the gateway’s connection to the wireless display is flaky. So I’ve ordered a replacement. I hope it’ll work properly, and if it does I’ll have more to say about the software aspects of the TED. But meanwhile, this is a good time to talk about the basic hardware setup. It’s pretty straightforward, but I think the vendor, Energy Inc., explains it misleadingly. That’s a shame. I hope lots of people will be able to use this remarkable tool, and I want them to have good experiences with it. So here’s what I think Energy Inc. does wrong with its installation guide, and how I think it could be improved.

The guide says:

A technically-savvy homeowner, neighbor, friend or electrician can install TED in 10-15 minutes.

And that’s true. But there are different ways to be technically savvy. The TED is a mashup of two different technologies: electrical distribution and data networking. Each has its own set of concepts and associated terminology. An electrician may know little about data networking. A homeowner like me may know a lot about data networking but very little about electrical distribution. A map of concepts and a glossary of terms would be really helpful.

On the electrical side, the terminology includes 120/240V single-phase, A phase, B phase, two-pole breaker, and handle tie. If you’re an electrician it won’t occur to you that the terms single-phase, A phase, and B phase will confuse somebody who isn’t an electrician. Like me. If it’s a single-phase system, I wondered, how is there an A phase and a B phase? I had to consult Wikipedia’s article on split-phase electric power:

A transformer supplying a 3-wire distribution system has a single-phase input (primary) winding. The output (secondary) winding is center-tapped and the center tap connected to a grounded neutral. This 3-wire system is common in countries with a standard phase-neutral voltage of 120 V. In this case, the transformer voltage is 120 V on either side of the center tap, giving 240 V between the two live conductors, shown as V1 and V2 in Fig. 1. The two outputs are properly called “legs”, not “phases”.

Ah. Now I could see why the TED’s MTU (measuring transmitting unit) needs (I wrongly thought) to be connected to the A “phase” and the B “phase” — all the wiring in my house is divided into two “legs” and the MTU (I wrongly thought) needs to reach both of them. Here’s the recommended procedure:

Step 3.

A.) Connect the black and red wires from the MTU power cord to a spare 15, 20, or 30 amp two-pole circuit breaker.

B.) If there is no spare circuit breaker in the panel, it can be attached to any 15, 20, or 30 amp two-pole breaker in the panel. If a two-pole breaker is not available, then use an approved handle-tie to create one.

At this point, as a data-network-savvy but electrical-distribution-naive homeowner, I’m ready to call in an electrician. It’s lucky I didn’t, though, because a data-network-naive electrician would likely have missed the significance of this cryptic remark stuck in between steps A and B:

(Option: For increased signal strength, connect only the black wire.)

Huh? Oh. Now my data networking knowledge kicks in. The MTU uses power line networking. That means it sends data over electrical wiring. Where does it send data? To the gateway, a little box that plugs into an outlet somewhere and talks to the wireless display and also to a computer or, if you run a local area network, to your Ethernet router. The recommended place to plug in the gateway is an outlet near your router, so you can control it and read data from it using any computer on your LAN.

The instructions tell you to connect the MTU to both legs of your electrical distribution. In theory that enables you to plug the gateway into any outlet in the house. In practice, as you’ll find if you try, fail, and then consult the troubleshooting guide, probably not.

But now I began to see that you don’t need to do that. You could connect just one of its wires to one breaker that controls an outlet near your router. And in fact that’d be better, because power line communication can be flaky and you want to use the most direct signal path. (The outlet should preferably be near your router but not actually on a circuit that powers noisy devices like the router, or computers, or the TV. And I found one that met that requirement.)

I found confirmation for this direct approach in the troubleshooting guide:

3.7. Considering the factors affecting PLC communication, if the Gateway is still not receiving a signal, do the following:

3.7.1. Connect the black wire of the MTU to the circuit on which the Gateway is connected.

3.7.2. Remove the red wire from the circuit in which it is connected, and put a wire nut onto the red wire.

Further confirmation came from the tech support guy I spoke with when I reported the sick gateway. “Yeah, I don’t know why the manual says to use a spare two-pole breaker,” he told me. “A lot of times that doesn’t work, then people call me up and I tell them to do it the way you did.”

Documentation that spelled out the concepts and terminology in each domain, and the relationships between the domains, would help people grounded in one or the other of the domains see the whole picture.

On the data networking side, the terminology includes PLC (power line communication), Ethernet, and ZigBee. There are actually three kinds of data networking happening. The MTU talks PLC to the gateway. The gateway talks Ethernet to your computer or, if you’re running a local area network as many now do, to any computer on the LAN. And the gateway talks ZigBee to the TED handheld wireless display. All this was obvious to me, but might not be to an electrician or electrically-savvy homeowner. Spelling out concepts and terminology would help that person sort out what gets distributed where:

– Electricity to the whole house, via the two legs of the power system

– Data to the gateway, via the electrical wiring

– Data to/from a computer or router via Ethernet cable

– In a LAN situation, data to/from one or more computers via Ethernet cable and/or WiFi

– Data to the wireless display via ZigBee

Maybe a lot of this context was provided by other TED users in the support forum. Except, oops:

Several weeks ago our Forum Server crashed, and we were unable to recover the valuable library of comments that our users had posted over the prior year.

Those forum members have now learned, the hard way, to appreciate the first of the seven ways to think like the web:

1. Be the authoritative source for your own data

The Energy Inc. site goes on to say:

Our new Forum Server has built-in safeguards to prevent the loss of data should there be a hardware malfunction. We do encourage those of you who have been avid online supporters to please continue to assist those new TED users in their quest to become experts!

Sure, happy to do it, but not by posting this article to the forum. Instead I’ll invoke the second Thinking Like The Web principle:

2. Pass by reference not by value

I want this article to help new TED users. Maybe somebody in the new forum will link here. Maybe I’ll do it myself if I can be bothered to create Yet Another Account. But even if that doesn’t happen, there’s an implicit connection via search. Anybody searching for key terms in this article, along with TED 5000, will land here.

I’m glad the forum server has backup now. But I still don’t want to commit a lot of my own keystrokes to somebody else’s cloud database. I want to keep those keystrokes in a cloud database that’s accountable to me, and then link them to wherever else they need to show up.