For everything that’s gained by our ability to store and maintain more information than ever before, something is lost that has to do with texture, context and association. The science journalist Joshua Foer, author of “Moonwalking With Einstein: The Art and Science of Remembering Everything,” said in a lecture to the Royal Society of Arts that people once “invested in their memories, they cultivated them. They studiously furnished their minds. They remembered. Today, of course, we’ve got books and computers and smartphones to hold our memories for us. We’ve outsourced our memories to external devices. The result is that we no longer trust our memories. We see every small, forgotten thing as evidence that they’re failing us altogether.” As we store more and more of what makes us us outside of ourselves, he said, “we’ve forgotten how to remember.”

The mnemonic techniques rediscovered in Moonwalking with Einstein were first popularized by Cicero. You bind memories to images, and then you bind the images to a path through the rooms and hallways of a “memory palace.”

Here’s another technique that isn’t so well known. I attribute it to Carlton Fisk by way of a story I heard from the baseball writer Roger Angell. Somewhere in the 2000s, Angell asked Fisk to reflect on what had most altered the game of baseball since his playing days. The salaries? The drugs?

No. The game-changer, Fisk said, was instant replay. His game-winning 1975 home run is one of most-remembered moments in all of sports. The video of that event is one of the most-watched clips. You might think that Carlton Fisk has seen that clip a million times. But in fact, he told Roger Angell, he never watches it. That’s because he doesn’t want to overwrite the original memory, which is his alone, recorded from a point of view that was his alone, with a memory we all share that was recorded by a camera up in the stands.

We can’t do away with instant replay, nor do we want to. But it’s worth remembering how to experience life, even when we know it’s being recorded externally, as if the only cameras are the ones in our heads.

Here’s the chart posted on Google+ by Bruno Oliveira:

A couple of things bothered me about this. First, there’s the adversarial tone. The subtext is a favorite geek quotation:

First they ignore you, then they laugh at you, then they fight you, then you win.

That gem is often attributed to Gandhi. Wikiquote disputes that and finds a close variant in Nicholas Klein’s 1918 speech to the Amalgamated Clothing Workers of America. Either way it’s the story of a persistent underdog overcoming an oppressor.

In geek ideology the oppressors are pointy-haired bosses and clueless users. Geeks believe (correctly) that clueless users can’t imagine, never mind implement, automated improvements to repetitive manual chores. The chart divides the world into geeks and non-geeks, and it portrays software-assisted process improvement as a contest that geeks eventually win. This Manichean worldview is unhelpful.

But the chart also fails to capture the reality of repetition and automation in the realm of information systems. Here’s an alternative world view that I choose to imagine and strive to create:

In this view of the world, tasks that involve data manipulation (as so many modern chores do) are undertaken by teams. There is an infinite supply of manual chores. Everybody tackles them. Ideally there is one member of the team I call the toolsmith. Working shoulder to shoulder with the team, the toolsmith spots an opportunity to automate some piece of the work, writes some code, deploys it, observes how it gets used (or doesn’t get used), assesses its impact (or lack of impact), and then iterates on the code. Meanwhile the toolsmith keeps working alongside the team, chipping away at the never-ending and always-evolving list of manual chores, looking for more opportunities to automate, and exploiting them in an incremental and collaborative way.

Software-assisted automation of repetitive work isn’t an event, it’s a process. And if you see it as a contest with winners and losers you are, in my view, doing it wrong.

Phil, I suspect, is actually one of five identical quintuplets. One teaches at BYU. Another runs a startup called Kynetx. A couple of others take care of family, church, and political matters. Finally there’s the one who does a podcast and also serves as executive producer of ITConversations. I don’t think I could handle being a quintuplet but it’s lucky for all of us that Phil can!

It was a real pleasure to meet Toby and learn about his comprehensive vision for a world in which our buildings, our cars, and our energy grid work with us, appearing as the intelligent agents that science fiction always encouraged us to imagine. One of his favorite examples involves a meeting room that’s scheduled for a 9AM meeting. A really intelligent system wouldn’t start heating or cooling the room at 9AM, it would start sooner. How much sooner? That would depend on the number of people attending.

Orchestrating that kind of dance requires the sort of loosely-coupled event-driven programming at the heart of the Kynetx technology that Phil is creating. Toby is layering that orchestration on top of existing Internet standards: iCalendar (for events), vCard (for resources), LDAP (for directory services). I love that approach for two reasons. First, I’m forever being reminded that we have barely scratched the surface of what might be accomplished with these existing standards if we really put them to use. Second, I violently agree with Toby that automated systems need to build on the standards that people actually use.

Toby chairs the OASIS Web Services Calendar Technical Committee and is an editor of the WS-Calendar specification. If standards whose names begin with WS- give you the heebie-jeebies, take a deep breath. I guess at some point there will be a SOAP profile for WS-Calendar but for now it’s a straightforward set of iCalendar extensions that define intervals, sequences, and relationships.

The web succeeds, in part, because its atomic particle — the hyperlink — can be manipulated both by automated systems and by people. When systems use hyperlinks it’s called RESTful web services. When people do it’s called emailing links. We tend to forget how profoundly the dual nature of the hyperlink binds the web together.

The hyperlink is atomic with respect to location. It says where. The fundamental particle of calendaring — iCalendar’s VEVENT — is atomic with respect to time. It says when. To me it makes perfect sense that smart buildings and energy microgrids will schedule their interactions with us in the same way that we schedule dentist appointments and soccer games.

It’s not technology. Slauson Middle School is using the same web services (in this case, Google Calendar) available to everyone.

It’s not budget. The web services required for this solution are free.

It’s a way of thinking. I wrote to Slauson’s principal, Chris Curtis, to congratulate him on the excellent example his school is setting, and to identify the thinker responsible. That thinker turns out to be Chris Curtis himself. And it’s no accident that the implementation pattern on display at Slauson is also evident at Pioneer High. Chris did the same thing there before coming to Slauson.

Now I am not an educator, I only watch from the sidelines. But to me the K-12 “computer skills” curriculum seems uniformed by the kinds of core principles that will make students effective in a web-augmented world. So I asked Chris:

What you’ve done at Pioneer and now Slauson builds on an important conceptual foundation. Do you think that K-12 education could build that foundation?

Here’s what he said:

I agree with the notion that the basic principles of computer science should be generalized more broadly across the curriculum. In many ways, teaching computer and technology skills courses absent a meaningful application of them is ineffective and silly. We wouldn’t teach driver’s education and not let students drive. We don’t teach a “pencil skills class” in which we learn the skills for using this technology tool without an immediate opportunity to apply the skills and then begin to consider and explore the many ways that the pencil and writing change how we organize, perceive, and interact with our world.

This issue gets at the heart of the challenge of technology and education. Often the world seems to divide into separate interest areas: those interested in technology and those interested in education. The result is often to send the technology nerds to a room and make them teach technology and send the other teachers to their rooms and let them teach. In order to be effective at integrating technology into the instructional environment there has to be a merger between a technology interest and and educational interest, within the same person. The awareness of what is possible via technology resources and the desire to perform educational functions can lead to the educator realizing that a task could be done differently, more efficiently, more effectively, with more precision, or in some other manner improved.

Of course the schism that separates technologists from educators also affects practitioners of all kinds. In his most recent essay, Bret Victor meditates on this point:

My piano teacher played the piano. Like, all the time. He had to; it’s not easy to make a living as a musician. Between tours, his band played restaurants, bars, weddings, anywhere they could get a gig. He chose this life because he loved music, and when he taught music, he was teaching what he did. In that way, his teaching was honest.

…

Back in high school, I was taught differential equations by a working engineer. He spent his days at Lawrence Livermore Laboratory, and for whatever reason, chose to spend his evenings at the local community college. Differential equations wasn’t some abstract arcana to him. It was his bread-and-butter, and he apparently found it important enough to share.

…

My information theory professor would teach me information theory in the morning, and then spend the afternoon furthering the field. Sure, what she taught was somewhat elementary by her standards, but she was well aware that this elementary theory was the foundation on which her life’s research was built. It showed, and it stayed with me.

…

Real teaching is not about transferring “the material”, as if knowledge were some sort of mass-produced commodity that ships from Amazon. Real teaching is about conveying a way of thinking. How can a teacher convey a way of thinking when he doesn’t genuinely think that way?

I’m preoccupied with a related question. The way of thinking that I most want to convey is web thinking. Which is, by definition, openly available to anyone who wants to learn. Schools everywhere can observe and emulate what Chris Curtis is doing at Slauson. In so doing they can become practitioners in the way that Chris is. Their students might then see them as practitioners and learn from their examples.

I would be delighted if the elmcity project could help bootstrap that virtuous cycle.

Dave Winer’s Why apps are not the future reminded me of that. The ancient construction method we are forgetting, in this case, is linking. We had already started to forget back in 2004. Then, writing for InfoWorld, I argued that rich Internet apps needed to learn what web apps always knew: RESTful design that exposes internal state as links. Kevin Lynch, who was then chief software architect of Macromedia, accepted the challenge and showed how a Flash app could behave that way.

But while you can bolt RESTful design and linking onto a rich Internet app, or a phone app, these principles aren’t built in. So apps are inherently less open to the sort of lightweight service composition that I’ve championed for a decade, ever since the LibraryLookup project showed me just how easily and powerfully a humble hyperlink can connect two otherwise separate information silos.

The point about service composition and linking is so obvious that we often fail to notice it. Yesterday I booked a hotel in Chicago for an upcoming conference. The organizer sent me an email containing this link for hotel booking:

http://www.marriott.com/hotels/travel/chidm-chicago-marriott/?toDate=1/22/12&groupCode=GROUPCODE&fromDate=1/20/12&app=resvlink

When I followed the link I landed in Marriot’s service with context appropriate to my status as an attendee. Nobody at Marriott specifically designed that capability. It’s just something that’s possible because the site works RESTfully, exposing its state as links that people can hand around.

Now to be honest, you have to be a geek-minded person — like the guy who invited me to this conference — to recognize and exploit this possibility. Most people don’t expect things to work this way, and most web designers don’t try to encourage that expectation.

My fear with rich Internet apps, and now phone apps, is that they tend to obscure an important lesson we never learned well enough in the first place.

1. Select the File / Import and Export… Menu option.
2. Select “Import from another program or file” and click Next.
3. Select “Comma Separated Values (Windows)” and click Next.
4. Find the desired CSV file and select “Do not import duplicate items” and click Next.
5. Select Calendar and click Next.
6. The next window will have a title that says ‘The following actions will be performed:’ above an option to ‘Import “(filename).csv” into folder: Calendar.’
7. Check that box and and click on ‘Import “(filename).csv” into folder: Calendar’ From there, click and drag needed fields from the left window into the right window. Its fairly self-explanatory, but the basic corresponding fields are below: SUBJECT to Subject START_DATE to Start Date START_TIME to Start Time END_TIME to End Time DESCRIPTION to Description LOCATION to Location

It’s hard to believe any Outlook user has ever followed those instructions. And if you’re using a calendar other than Outlook, you’re on your own.

So I asked Harry to make a filter that turns those CSV files into iCalendar feeds. It’s specific to that particular format for now, but as we find other examples in the wild we’ll generalize accordingly.

My next challenge involved feed categorization. I’m finding pockets of excellence in various places around the country. As I mentioned last month, the Slauson Middle School in Ann Arbor is one of them. Its calendar looks like a single feed, but is actually a merged set of 11 distinct Google Calendars, including one for Slauson clubs, several for Slauson sports, and one for Slauson’s upcoming Chicago trip.

As curator of Ann Arbor’s community hub I was thrilled to see this kind of self-categorization. It enables me to merge Slauson’s sports and music events with other sports and music events in Ann Arbor. But if it’s rare to find a public calendar in machine-readable iCalendar format, which it unfortunately is, it’s even rarer to find a set of machine-readable public calendars that can be syndicated by category.

I asked Harry to weigh in on this problem too. He’s a do-it-yourself guy who doesn’t like to wait for Central Services to take care of things. “If the sources aren’t providing you with the categories you need,” he said, “maybe we can create them ourselves.” Next thing you know, he’d whipped up a filter that selects subsets of iCalendar feeds by keyword and/or time.

Here’s one example of how I’m using it. Vinology, a restaurant and wine bar in Ann Arbor, publishes upcoming events using a Google Calendar. The feed from that calendar mixes two different kinds of events: restaurant business (Happy Hour) and a jazz schedule (Meri Slaven trio). I’d like to select just the jazz events for Ann Arbor’s music and jazz views.

Harry’s filter solves this problem. Actually, it solves it in two different ways. The first way is to filter Vinology’s feed for events where the SUMMARY or DESCRIPTION fields contain the keyword jazz. The second way is to filter the feed for events that happen after 8PM. Either of these methods produces the jazz-categorized iCalendar feed that Vinology might someday create for itself.

Harry’s filter also solves a related problem. The Berkeley Art Museum / Pacific Film Archive is, as the name suggests, a venue for both art exhibits and film showings. The calendar is, happily, available as an iCalendar feed. Unhappily it mixes art exhibits with film showings and I couldn’t find any times or keywords that would reliably separate the two streams. But then I found a keyword that would. The BAM/PFA website’s URLs look like http://www.bampfa.berkeley.edu/exhibition/faulders_bamscape) and http://www.bampfa.berkeley.edu/filmseries/clouzot). And those same URLs appear in the iCalendar feed. So Harry enhanced the filter to look for those boldedkeywords in the URL field of iCalendar events. Now the BAM/PFA’s film series shows up where it belongs, in the film category of the Berkeley hub.

Senator Ted Stevens got it right. The Internet really is a series of tubes. When they don’t fit together quite the way you need them to, Harry Tuttle can sometimes help.

1. An Eventful venue. (iCal feed.)

2. An Upcoming venue. (iCal feed.)

3. A Facebook Page. (iCal feed.)

4. The Aquarium’s own calendar page. (iCal feed.)

If you visit #4, the Aquarium’s calendar page, you’ll see that it does offer an Export as iCalendar link. So why isn’t that link hot here? Because it’s only wired to JavaScript code that imports a snapshot of the data into a calendar program. No URL points directly to the data. That means people can’t subscribe to the feed — and neither can the Seattle hub.

If the Aquarium were truly thinking like the web it would offer its calendar as a first-class web resource addressed by an iCalendar URL. Even better, it would offer one such URL for each view of the calendar page: Educators, Family / Kids, Members Only, Out and About, Speaker Series, Special Events, Summer Camps, Volunteer Orientation.

Since those views aren’t available as feeds I had to fall back to the other three sources. At first I merged all three to create the aquarium view of the Seattle hub. When I compared the results of the merge, though, something didn’t cross-check. An event called Marine Mammal Talk, which showed up in the Upcoming feed, wasn’t in the Eventful or Facebook feeds or on the Aquarium’s own page. Yet Upcoming’s Marine Mammal Talk page says this event was discovered by the Upcoming Robot, has occurred 498 times already, and will occur 77 more times. What’s up with that?

I searched the Aquarium’s site and found the event. Here are the details:

Tue, 15 Nov, 2011 2:43 PM – 2:43 PM

Three weekends of special activities all about the Aquarium’s Marine Mammals – Otters, Harbor Seals & Fur Seals. Join us for special talks, demonstrations and hands-on activities.

Fees: Included with regular Aquarium Admission

Calendar: Programs & Events Calendar

Category: Special Events

Repeats: Weekly on Sunday, Friday, Saturday until 4/24/2011

You can see why the Upcoming Robot was confused. The event’s date is given as Tue 15 Nov 2011. But its recurrence rule says that the last recurrence was back in April of this year.

Of course this fragment of text doesn’t really specify a date or a recurrence rule. The Upcoming Robot just inferred those, erroneously, from unstructured text. And then, like the Sorcerer’s Apprentice, it began pumping those errors onto the web. It has evidently done so for a couple of years and will, if unchecked, continue into 2013.

I don’t blame the Upcoming Robot. Denied access to the structured data it craved, the poor thing scavenged what it could. The lesson here is for the Seattle Aquarium and for all who mean to publish data online. If you don’t establish yourself as the authoritative source for that data then others will step in to do it for you. And they are liable to get it wrong.

Here’s the breakdown of the 309 events from the grade schools:

And the 322 events from the middle schools:

And the 966 events from the high schools:

Among grade schools, Slauson is notable not only for the number of events but for the exemplary self-categorization applied to them. When you click the Google Calendar subscribe button on the Slauson calendar page here’s what you’ll see:

This is a best practice I wish everyone would adopt. It illustrates the seventh of my seven ways to think like the web: #7: Reuse components and services. The Slauson calendar is both a user of other services (the district-wide calendars) and a provider of services. And as a provider, it understands the idea of componentization. In an era of abundance it costs no more to create and manage a dozen calendars, using free services like Google Calendar and Hotmail Calendar, than to jam everything into a single calendar. The benefits are manifold. They include:

Delegation

In most schools and businesses, maintenance of “the” public calendar is one person’s job. That person becomes a bottleneck. When you recognize that logically there isn’t one public calendar, but instead several or many, each with its own appropriate maintainer, then you can break that bottleneck.

Precision

A parent who subscribes to a single undifferentiated school calendar may be overwhelmed by the flow. Parents of kids who are in music programs, or on sports teams, or who are going on the Chicago trip, should be able to focus on those activities.

Scope

Slauson Middle School is part of the larger Ann Arbor community. When Slauson’s calendars are self-categorized, they can align with community-wide views. Here’s a picture of some sports-related activites in Ann Arbor on November 17th:

Slauson’s publication of a set of self-categorized machine-readable calendar feeds enables it to represent its sports activities on a city-wide timeline that includes, in this particular view, events from the Ann Arbor Triathlon Club, the Wolverines basketball teams, and Ann Arbor’s kickball Meetup group.

Well done Slauson! And kudos to all the Ann Arbor public schools. You have become web thinkers. I hope schools everywhere will learn from your example.

Here’s one of my favorite scenes from the movie Brazil,

Sam Lowry (Jonathan Pryce): Are you from central services?

Harry Tuttle (Robert De Niro): Hah! They’re a little overworked these days. Luckily I intercepted your call.

Sam Lowry: Can you fix it?

Harry Tuttle: No, but I can bypass it with one of these.

Back in 2003, my InfoWorld column APIs, protocols, and rogue plumbers made three points:

1. The web of data shouldn’t require the services of Harry Tuttle.

2. Unfortunately it still does.

3. Fortunately the web’s architecture (a series of tubes) enables Harry to intervene when he must.

Those points are all still valid in 2011. From time to time I still have Harry Tuttle moments. Today’s involved the campus events system at my alma mater, the University of Michigan. In the current phase of the elmcity project I’m rebuilding hubs in various cities in order to dramatically beef up the number of feeds and quality of tagging. Ann Arbor’s hub was conspicuously lacking feeds from the University of Michigan. When I investigated I found that the central service, http://events.umich.edu, does offer iCalendar feeds. Yay! You can’t take that for granted, many if not most schools don’t make their public calendars available in a machine-readable way. Unfortunately there’s a problem with the feeds produced by the UM service. They’re invalid. You can’t load them into an iCalendar-aware program like Google Calendar or Outlook, and the elmcity engine can’t aggregate them.

I reported the problem to central services and have been awaiting a fix for some time. Today, because I wanted to get my hands on that data, I unleashed Harry Tuttle. There are two major problems with the iCalendar export from events.umich.edu. First, the lines of text aren’t properly folded. Second, the timezone properties don’t refer to a timezone definition. So I made a filter that fixes these major problems (plus some other minor ones). Here’s what the filter does:

Original: http://jonudell.net/data/failed-ics-umich.ics.txt. Validation result: Unparseable.

Line-folding fixed: http://jonudell.net/data/fixed-ics-umich.ics.txt. Validation result: Better.

VTIMEZONE added: http://jonudell.net/data/clean-ics-umich.ics.txt. Validation result: Valid.

Then I used the filter to add a bunch of feeds to the hub. Here’s one for the Taubman Health Services Libraries, and another for the Gerald R. Ford Presidential Library. These merge with other library feeds, notably from the Ann Arbor District Library, in the hub’s top-level library view.

Now, where do I fill out that twenty-seven-B-stroke-six form?

From Wikipedia’s Brazil_(film) page:

The reference to form 27B-6, without which no work can be done by repairmen of the Department of Public Works, is a reference to George Orwell, who lived at 27B Canonbury Square Apartment 6, while writing Nineteen Eighty-Four.

google calendar	hotmail calendar

On the left you see Google Calendar displaying two calendars, each representing a single event — the Brower Youth Awards on October 18 at the Herbst Theater in San Francisco — in a different way. On the right you see Hotmail Calendar displaying the same two calendars. The event will happen at 5:30 Pacific time on the 18th. I found it on the Berkeleyside site whose events page offers a companion iCalendar feed.

If you load that feed into both Google Calendar and Hotmail Calendar, and if your calendars are set to Eastern time, you’ll see what’s shown above. If your calendars are set to another timezone the times will be shifted but the pink ones still won’t match.

The green ones should always match and should always be what you’d expect. For me, looking at a 5:30 Pacific event through the lense of calendars set to Eastern, I’d expect both calendars to display 8:30.

What’s the difference between the pink calendar and the green calendar? Here’s the pink one. It’s just the original calendar reduced to a single event. Like the original it declares its timezone using the nonstandard X-WR-TIMEZONE property:

BEGIN:VCALENDAR
VERSION:2.0
METHOD:PUBLISH
CALSCALE:GREGORIAN
PRODID:-//Refresh Web Development//Helios Calendar//EN
X-FROM-URL:http://www.berkeleyside.com/BerkeleysideCalendar/events/
X-WR-RELCALID:Berkeleyside
X-WR-CALNAME:Berkeleyside
X-WR-TIMEZONE:America/Los_Angeles
BEGIN:VEVENT
URL;VALUE=URI:http://www.berkeleyside.com/BerkeleysideCalendar/events/index.php?com=detail&eID=6
DTSTART:20111018T173000
DTEND:20111018T210000
SUMMARY:Brower Youth Awards 2011
LOCATION:Herbst Theater - 401 Van Ness \, San Francisco\, CA US 94102
END:VEVENT
END:VCALENDAR

And here’s the green one. Again it’s a derivation of the original calendar that reduces to a single event. But it also declares its timezone in the standard way, using a VTIMEZONE component and then referring to it using the TZID parameter of the DTSTART and DTEND properties:

BEGIN:VCALENDAR
VERSION:2.0
METHOD:PUBLISH
CALSCALE:GREGORIAN
PRODID:-//Refresh Web Development//Helios Calendar//EN
X-FROM-URL:http://www.berkeleyside.com/BerkeleysideCalendar/events/
X-WR-RELCALID:Berkeleyside
X-WR-CALNAME:Berkeleyside
BEGIN:VTIMEZONE
TZID:America/Los_Angeles
X-LIC-LOCATION:America/Los_Angeles
BEGIN:DAYLIGHT
TZOFFSETFROM:-0800
TZOFFSETTO:-0700
TZNAME:PDT
DTSTART:19700308T020000
RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=2SU
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0700
TZOFFSETTO:-0800
TZNAME:PST
DTSTART:19701101T020000
RRULE:FREQ=YEARLY;BYMONTH=11;BYDAY=1SU
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
URL;VALUE=URI:http://www.berkeleyside.com/BerkeleysideCalendar/events/index.php?com=detail&eID=6
DTSTART;TZID=America/Los_Angeles:20111018T173000
DTEND;TZID=America/Los_Angeles:20111018T210000
SUMMARY:Brower Youth Awards 2011
LOCATION:Herbst Theater - 401 Van Ness \, San Francisco\, CA US 94102
END:VEVENT
END:VCALENDAR

As we see in the picture above, the event time on the green calendar shows up the same way in both Google Calendar and Hotmail Calendar. It should also be the same in any calendar program that supports the iCalendar standard.

The event time on the pink calendar, though, is a up for grabs. Calendar programs that strictly follow the iCalendar standard should ignore X-WR-TIMEZONE and always display the local time, 5:30PM, which will be right for people in the Pacific timezone and wrong for everybody else. Hotmail Calendar does this. Programs that use X-WR-TIMEZONE, on the other hand, can render this calendar just as they would render a standard calendar. Google Calendar does this.

Why do I care? I have to decide whether the elmcity service will or won’t consider X-WR-TIMEZONE to be meaningful. The service is based on DDay.iCal, the same standards-based parser that powers the iCalendar Validator. So when the the service reads the pink calendar, and renders it for users in Berkeley, it will do the wrong thing from their point of view.

To do the right thing for Berkeley it would need to do the wrong thing by iCalendar: transform the nonstandard X-WR-TIMEZONE property into a standard VTIMEZONE component, and then transform all the dates so that they refer to the VTIMEZONE’s TZID. In order to create that VTIMEZONE component, it would interpret X-WR-TIMEZONE value as a TZID (timezone ID) from the Olson database. A Unix-based service would look up the TZID in Olson, find the rule for the timezone — i.e., offsets from GMT for standard time and daylight savings time, and when to appy them — and express the rule using VTIMEZONE syntax. A service running on Windows Azure, like mine, would instead need to map the Olson name to a Windows timezone name, look up the rule using a Windows API, and then express the rule in VTIMEZONE syntax.

Of course this is a slippery slope because, in the end, I’m only guessing what X-WR-TIMEZONE is supposed to mean. Here’s Rick DeNatale engaging in the same kind of guesswork:

Someone pointed me to this icalendar file of Australian holidays for a test case:

http://icalx.com/public/rohanl/Australian32Holidays.ics

This contains NO VTIMEZONE components, but does have the calendar property: X-WR-TIMEZONE:Australia/Sydney

Googling indicates that this is a non-standardized property, but it seems to be used by several calendar apps including Apple’s ical.app
and Google calendar.

I know that it’s non-standard, but it seems to be somewhat important for interoperability. I’m looking for some kind of information about
what it means in general.

It seems to indicate a default tzid for the whole calendar. In the absence of timezone components I’m not sure how to interpret the tzid, though.

Australia/Sydney IS a time zone identifier in the Olsen database, is it standard practice to use olsen tzids in X-WR-TIMEZONE calendar attributes?

Fortunately I can bring some data to bear on this question. Thanks to the iCalendar Validator I can analyze public calendars produced by a variety of iCalendar producers. In The long tail of the iCalendar ecosystem I listed the names of about 600 producers seen recently by the Validator. Of those, about 100 use X-WR-TIMEZONE instead of VTIMEZONE, and 70 of those 100 use local rather that UTC date syntax which implies they are depending on X-WR-TIMEZONE for correct interpretation of those dates.

Note that Google Calendar, the 800-pound gorilla in this space, is not one of those 70 producers. When it writes iCalendar format it uses both X-WR-TIMEZONE and VTIMEZONE; the latter ensures that Google Calendars can be understood properly by standard parsers that don’t support X-WR-TIMEZONE. The 100 producers I’m talking about, though, are using only X-WR-CALENDAR in a way that suggests they expect a nonstandard transformation. The fact that Google Calendar performs that transformation is, of course, a major reason why producers would expect it to happen everywhere.

Should X-WR-TIMEZONE be standard? That’s debatable. It would certainly make life easier for iCalendar producers. They could just mention a timezone rather than having to extract its rule from their operating systems and express the rule in VTIMEZONE syntax. One of the reasons for the success of RSS and Atom, after all, is that it’s always been easy to whip up an RSS or Atom feed which you can then check with the Feed Validator. An analogous simplicity for iCalendar producers would help grow the iCalendar ecosystem.

On the other hand if an iCalendar feed were to only mention a timezone without fully defining it, then the consumer would have to do the work that the producer didn’t. That’s problematic as Doug Day, author of the iCalendar Validator, notes in a recent email exchange:

Unfortunately, without including the actual time zone information in the calendar (i.e. via VTIMEZONE), you can’t be sure that the date/times you’re representing are accurate, even when using X-WR-TIMEZONE. For example, if you’re on a Windows XP machine that hasn’t been updated in 5 or 6 years, your system time zone information will be inaccurate. However, if the VTIMEZONE were included in the calendar, it would remain accurate, even on an older machine with out-of-date time zone definitions. Also, in order to interpret X-WR-TIMEZONE, you’d need to be in an environment where interpreting Olson time zone is realistic (easy on Linux, harder on Windows). I know a global, online time zone registry is in the works, but I don’t think it’s to a point where it’s useful, yet.

You might wonder why all this timezone stuff is even necessary. After all, an iCalendar feed can simply omit VTIMEZONE (and/or X-WR-TIMEZONE), express dates and times in UTC (Coordinated Universal Time), and use UTC syntax for all dates and times. Why not just do that? I asked Doug Day about this a while ago, and here was his reply:

The biggest problem is with recurring events and daylight/standard time transitions. For example, consider the following (all hypothetical):

1. I live in Salt Lake City, Utah

2. I want to schedule a meeting, starting on September 7, 2009 at 9:00 A.M, which recurs every month on the first Monday.

3. Some of the people attending this meeting live outside my current time zone.

So, here are the occurrences you’re ultimately after:

- September 7, 2009 – 9:00 A.M. (3:00 PM UTC)

- October 5, 2009 – 9:00 A.M. (3:00 PM UTC)

- November 2, 2009 – 9:00 A.M. (4:00 PM UTC)

As you can see, once the time changes from daylight back to standard time, so does the UTC representation of that time. So, if you had scheduled your event in UTC time, when the time zone changes, your event time will actually have changed (to 10:00 A.M., rather than 9:00 A.M.)

For this reason, among others, it’s always best to include time zone information whenever available. Traditionally, it’s been pretty difficult to include that information, and it’s more often left out than included.

So what shall I do with X-WR-TIMEZONE? I’ve decided to support it experimentally. If you start a new hub on the elmcity service, the default is to ignore X-WR-TIMEZONE. But if your hub has important sources that depend on it, as Berkeley does, then you can override the default so the times will be as you expect.

Meanwhile we’re going to update the iCalendar Validator to warn producers about this issue. There’s nothing technically invalid about a calendar that uses X-WR-TIMEZONE without VTIMEZONE. To a parser that strictly interprets the RFC5545 standard, that property is just a name that’s “reserved for experimental use.” But as has always been true of the RSS/ATOM Validator, the iCalendar Validator aims to deliver useful real-world guidance. Producers that use X-WR-TIMEZONE alone to declare a timezone should know that while it may often yield expected results, it’s not guaranteed to do so. It would be better to use a standard VTIMEZONE.

The next step will be to analyze how well these these producers conform to the validator’s interpretation of the iCalendar spec. But the list itself forms an interesting data set. We know intuitively that, after 12 years of evolution, the iCalendar ecosystem must have become broad and diverse. Here’s a nice illustration of that breadth and diversity.

(Note that these counts don’t necessarily reflect the real distribution of iCalendar producers. The iCalendar Validator is closely associated with the elmcity project, so certain producers used heavily there — DDay.iCal, EVDB, Meetup — are overrepresented. On the whole, though, I’d guess this is a reasonable proxy for the distribution of producers.)

Meanwhile I’ve been toiling in another part of the semantic web. I’ve been trying to get webmasters people to understand why and how to publish calendars as machine-readable structured data in addition to human-readable text. If you’ve followed the elmcity saga you know I’m on a crusade to make more and better use of the Internet’s venerable standard for exchanging structured calendar events: iCalendar.

It’s been a struggle. Almost every website run by a school, club, business, or town has an Events page. Those pages are, almost always, data siloes: HTML or PDF files that can be read by people but cannot be processed by machines. Only rarely do such pages offer links to iCalendar feeds served up by Google Calendar or Drupal or Hotmail Calendar or some other service capable of producing such feeds. So when elmcity curators discover one of these rare feeds, it’s cause for rejoicing.

Sadly that joy is sometimes short-lived. A surprising number of iCalendar feeds just plain don’t work. That’s why I invited Doug Day to create the iCalendar Validator, a service that helps producers of iCalendar feeds conform to the specification. It’s always painful when I have to explain to a curator that the shiny new feed they’ve discovered doesn’t conform and won’t deliver events to the hub.

Here are three sources of iCalendar feeds that, I’ve recently discovered, don’t work.

1. The University of Michigan’s UM Events. It’s a major hub that serves many campus websites. As far as I can tell, all of those sites are providing feeds with malformed descriptions. Here’s an example of the problem and the fix. (iCalendar producer ID: UM//UM*Events)

2. CiviCRM is a “free, libre and open source software constituent relationship management solution.” Its feeds also have malformed descriptions. Here’s an example of the problem and the fix. (iCalendar producer ID: CiviCRM//NONSGML CiviEvent iCal)

3. Drupal is a popular open source content management system. I’ve seen Drupal feeds used successfully, but today I found one that fails for two reasons: a malformed recurrence rule, and a missing timezone definition. Here’s an example of the problem and the fix. (iCalendar producer ID: Drupal iCal API)

These problems are minor and would be easy to resolve. I’ll try to contact the authors of these iCalendar producers; if you can help put me in touch I’d appreciate that. I’m also going to look through the logs written by the iCalendar Validator, compile a list of producers of invalid feeds, and try to contact them as well.

Where’s the connection to the semantic web? At the end of the day, as RSS/Atom validator co-creator Sam Ruby likes to say, “It’s just data.” But structured data, whether it conforms to the dozen-year-old iCalendar standard or some newfangled microdata standard, is easily screwed up. And the consequences of screwups are often silent. Services that were looking for that structured data find nothing, mutter to themselves, and move along.

As we collectively create the semantic web we’ll need to make sure that the structured data we intend to publish really says what we mean.

If I am tagged in a Facebook photo, then save the photo to Evernote

If the library sends email saying my book on hold is ready, send me a text message

My colleague Scott Hanselman says this is bloody brilliant and I agree. It’s the next step in a journey that began for me back in 1999 when I mashed up Alta Vista’s search engine with Yahoo! directories to measure the mindshare of sites by category. More recently Yahoo! Pipes made service mashing easier for non-programmers. Now IFTTT enables everyone to play. Wonderful!

So why am I less enthusiastic about IFTT than I thought I would be? There are two related reasons. First, here’s what IFTTT says when you ask it to activate its Twitter channel on your behalf:

This application will be able to:

• Read Tweets from your timeline.
• See who you follow, and follow new people.
• Update your profile.
• Post Tweets for you.
• Access your direct messages.

Excellent! This is an example of OAuth, a protocol that enables you to delegate powers to IFTTT without giving up credentials. Most of the services you can use with IFTTT support OAuth, and that represents another huge step forward for the web.

What if I only want to give IFTTT the power to tweet on my behalf, though, and not give up access to my private direct messages? More generally, how can I think about the tradeoffs involved in delegating all versus some versus no powers to IFTTT, across a range of services I might authorize it to use on my behalf?

This leads to the second and broader concern. If I’m not paying for the product, I am the product. As is true for many free services on the web today, I have no contractual relationship with IFTTT. I pay for the service it provides by surrendering access to my data. OAuth helps me negotiate how much access, but if I give up none then IFTTT is powerless.

Why, though, can’t I pay for the product instead of being the product? I want IFTTT to work for me, I want to pay for the service, and in return I want it to promise never to keep or use any of the data it exchanges on my behalf. I realize this may not be a popular option anytime soon. But it’s time to start the conversation. Services don’t only want to be free, they also want to be valuable. That’s rarely a choice nowadays. It needs to become one.

The home page would feature browseable “stacks,” or collections of related images, videos and links shared around topical events. The site would also make personalized recommendations for users, based on their sharing habits. “We want to simplify things visually, mainstream the product and make it easier for people to understand what they’re doing,” Mr. Hurley said.

The second link points to a blog post from Mark Surman, executive director of the Mozilla Foundation, who wants to teach the world to code:

This has been the premise behind much of what we have done with Mozilla Drumbeat: people who make stuff on the internet are better creators and better online citizens if they know at least a little bit about the web’s basic building blocks. Even if they only learn a little HTML, the web gets better.

I wish I could broker a conversation between Chad Hurley and Mark Surman. If mainstream folk used Delicious and understood what they’re doing when using it, they’d understand themselves to be makers of things on the Internet. The things they make are custom information systems. They make them by writing code, but not in the languages of HTML or JavaScript. Instead they use tag vocabularies that produce and consume web services. And services, I argue, are the most fundamental of the web’s basic building blocks.

Here is an example: http://www.delicious.com/judell/del.icio.us. Ostensibly it’s a list of dozens of articles I’ve written over the years about what I mean when I say that Delicious enables non-programmers to code and use web services. But it’s not just a list. I think of it as a web service. One aspect of the service provides the list in HTML format for people to read in browsers. Another provides the list in RSS format that enables cooperating services to watch the list and react when new items are added. Another enables the list to combine with other lists. Here, for example, is a subset of my Delicious-related articles that are also related to the elmcity project: http://www.delicious.com/judell/del.icio.us+elmcity.

More than anything before or since, Delicious empowers me to manage web resources — both personally and socially. Once those resources were mainly things we found on the web. Now they’re also things we make on the web. I hope the forthcoming Delicious makeover will help people understand it to be a tool for creating, mixing, and sharing web resources. And I hope it remains the sort of open web tool that Mozilla Drumbeat wants to popularize.

Stoves, dishwashers, and vacuum cleaners shouldn’t require this kind of life support. I’m not going to win that argument, but I can at least make effective use of the life support service. Doing so requires a bit of a hack, though. Why? Remembering something annually, on a certain date, for five years running, isn’t the sort of thing that humans do well. It’s a task that begs for automation.

Of course Sears could remind me, in September of 2011, 2012, 2013, 2014, and 2015, about my dishwasher checkup. It would be easy for them to do. They’re a mega-corporation with a mighty IT department that can easily perform this feat of magic. But Sears doesn’t want to remind me. They’re betting that I’ll forget, as I’m sure most people do.

So they sounded quite surprised when I called yesterday, on the anniversary of my dishwasher purchase, to schedule my appointment. How did I do it? By casting this magic spell:

BEGIN:VEVENT
DTSTART;VALUE=DATE:20110902
DTEND;VALUE=DATE:20110903
RRULE:FREQ=YEARLY;COUNT=5
SUMMARY:sears dishwasher
END:VEVENT

That’s an event recorded by my calendar program. It happened first on September 2, 2011, and will happen again yearly for five years. I cast my spell using Outlook but you can do the same thing using any calendar program: Google Calendar, Apple iCal, Hotmail Calendar, Lotus Notes, many others.

When I talk about iCalendar feeds in the context of the elmcity project, I tend to focus on the idea that these are data feeds, which they are. But iCalendar has a special property. Some of that data is actually code. In this example, here is the one line of code:

RRULE:FREQ=YEARLY;COUNT=5

In iCalendar lingo, RRULE means recurrence rule. From Outlook’s or Google Calendar’s point of view the rule is a tiny program and they are the operating systems that run it. Here are some other RRULES that calendar applications enable you to express:

every Tuesday at 10AM

every first Sunday until October 2011

every September 2 for five years

You don’t have to do it this way. You can, instead, record each event separately. Even though calendar software makes it quite easy to define repeating events, I’ve got a hunch that in the dishwasher scenario a lot of people would use data (five discrete events) rather than code (one event that expands to five). Why? For the same reason that many people will manually adjust the font size used for every paragraph in a document, rather than creating a rule that governs all the paragraphs.

My list of core principles for web thinkers doesn’t yet include the idea of code that generates data. I’ve found it plenty challenging just to get people to think about the nature of their data, about how they do or don’t control it, and about how it does or doesn’t flow through networks.

But if we want everyone to be able to automate tasks that otherwise get done poorly or not at all, we probably do need to find a way to teach patterns like code-generated data and Don’t Repeat Yourself more broadly than just to students of computer and information sciences.

If you want to read a lovely short essay on calendar software and our relationship to the future, I highly recommend Paul Ford’s Tickler File Forever.