It’s almost 10 years since I began producing and consuming data feeds, initially in RSS format. Although I regard the syndication of data feeds, in general, as a transformative technology, the concept still makes no sense to civilians and has little or no effect on their lives.
In order to understand why not, and as a way of figuring out how to motivate a practical understanding of syndication, I’m tackling a problem whose solution doesn’t involve RSS, or Atom, or microformats, or XML. The problem is calendar syndication, and part of the solution is iCalendar, a non-XML format that all widely-used calendar programs support well enough for my purposes.
It’s only part of the solution because the real problem is that most people, most of the time, for most of their calendar-related activities, don’t use calendar programs. They use spreadsheets and wordprocessors, and they produce unstructured web pages and PDF files.
There was a time when, behind their backs, I would mock them for doing so. No longer. As I meet with intelligent and well-educated professionals in my community, and talk with them about how to synchronize calendar information from a variety of sources, I realize that they simply have no intuition about the difference between a PDF file and an ICS file that contain the same calendar information. Both are computer files, right? Both can be posted to the web, right? Both can be searched, right? Problem solved.
There are really two aspects to this missing intuition. First, the concept that some kinds of computer files are more structured than other kinds. Second, the concept that the structured kind can flow easily around the Net without loss of fidelity, and can deliver use value in a variety of contexts, whereas the unstructured kind is inert.
These are ways of computational thinking unknown to most people. As a school administrator, librarian, city planner, social worker, or retail store owner, nobody expects you to understand and apply these principles.
And yet almost everybody needs to harmonize personal and organizational calendars. And many individuals and organizations need to flow their calendar data into other contexts to promote and coordinate their activities.
So here’s my approach. I’m scooping up all the calendar information I can find for my community, in whatever form I can find it, and flowing it into a coommon view. Then I’m syndicating that view elsewhere to show that there’s nothing special about my aggregation.
The idea is to establish a critical mass by brute force, and allow people to see how, over time, sources that are structured and can syndicate will remain in the game, and sources that aren’t will have to sit out on the sidelines.
It’s turning into a nice case study of how organizations and individuals can negotiate shared responsibility for calendar information that’s of common interest. But that’s a story for another day. First things first. I need to give people a reason to care about using a calendar program — any calendar program, could be Outlook or Apple iCal or Google Calendar, so long as it exports iCalendar — in preference to a spreadsheet or word processor. Although the geek tribe can scarcely imagine why, that first step is a doozy.
Fascinating. I think we should not be surprised by your experience — everything I learn about user behavior points in this direction. It is all mystery meat below the GUI/browser display.
I think the geek insight is to realize that there is no earthly reason why it should be otherwise.
I wonder about this when we talk about other kinds of interoperability, too, along with identity and data portability, social grids and such.
“they simply have no intuition about the difference between a PDF file and an ICS file that contain the same calendar information.”
A friend, a fairly light computer user for almost 20 years, surprised me the other day when he asked me how to edit a PDF file. My response: “you can’t; PDF files are for reading or printing, but you can’t really edit them” (the simple answer anyway). He didn’t believe me. He claimed he’d been editing PDF files with “some editor” for years. An editor like Word. He just couldn’t find the editor.
I thought it was funny at first, but then I realized, why WOULDN’T you think you could edit PDF files? For much the same reason as you suggest. Moreover, a lot of people use Word, Powerpoint, etc just to ‘view’ files people send to them as attachments. Occasionally, they’ll edit a file.
Jon,
For what it’s worth, I just rewrote the UNH Cooperative Extension events calendar program and created ics files for each event and iCalendar feeds for each calendar (there’s a Cheshire County calendar if you want to grab it.) I’m not sure most folks will ever use the feeds, but I didn’t feel right about not creating them, and it’s not hard to do.
Steve
> I’m not sure most folks will ever use the feeds
Maybe not directly, at least not anytime soon, but look:
http://elmcity.info/events#2008-04-29
There you are at 5:30 PM, nestled in between the high school, library, medical center, little league, and softball stuff.
This is /exactly/ the network effect I’m seeking to illustrate.
Thanks for providing a new iCal source! I’m on the hunt for all I can find relevant to Cheshire County NH, so feel free to point me to any I’ve missed.
> then I realized, why WOULDN’T you think you could edit PDF files?
Exactly.
> I wonder about this when we talk about other kinds of
> interoperability, too, along with identity and data portability,
> social grids and such.
Right. There are no intuitions about this stuff: what’s hard or not, why or why not, how to make easy things easy and hard things possible.
Of course there was the intuition that stuff should Just Work. But that so rarely happened that people gave up expecting it to.
Then, tragically, easy stuff that could Just Work goes unnoticed. Like having the UNH Cooperative Extension and the public library and the high school and the YMCA all show up in the same calendar view.
Hello Jon,
I’m trying to follow a new FLOSS calendar aggregator project – http://calagator.org/ but I’m hampered by my lack of technical literacy. I thought I would bring it to your attention in case it was relevant, and just say that I would love to hear more about it (interview?).
Also, I hope you’re thinking about events on facebook as potential data for your calendar.
The problem is deeper than that (of course) in that the limits of conventional machine parsing (let alone “understanding”) are unknown to most; consider how much software is commissioned, built and written to scrape or convert data into another format — it would be great, for example, if all research or Govt. data was published in machine-readable format (my personal pet peeve is this ftp://ftp.cpc.ncep.noaa.gov/htdocs/products/analysis_monitoring/cdus/degree_days/archives/Heating%20degree%20Days/Monthly%20City/)
Jon,
I’ve been on a similar mission for a couple of years now. Read more about it here: http://cannell.org/blog/calendar-syndication-and-critical-mass
Jon,
Have you checked out FuseCal? They sort of scrape Web pages to extract calendar entries. They don’t do PDFs yet. That would be a great addition.
Scott Mace
We’ve been working on similar problems for a while at Spongecell. Promoters place great value in getting an event into a prominent place in someone’s life – it’s the most effective way to drive attendance. Similar to you, we started out thinking that the “prominent place” was always a calendar, but even people who have adopted iCal/Outlook/gCal don’t interact with them very often – most will check email, social networks, cell phones, and desktop widgets far more frequently.
It’s a hard road to plow if you require people to change platforms to absorb your information. As a publisher of event information you should use a system (plug – like http://spongecell.com) that allows your audience to move event information into whatever format or platform they are most comfortable with.
We are actually developing technology to aggregate calendar data into one place, though our calendar product is primarily designed for schools and school districts. The key is a centralized, very scalable database and web accessibility.
Our biggest competition? Pen and paper. Busy parents don’t have the time or want to spend the time glued to their computers updating events. So they’ll resort to the good ole’ calendar on the fridge.
So then, data feeds to a central calendar are key. Most calendar data potentially comes from schools (for parents with kids) so we provide RSS feeds that can be filtered based on their children’s involvements. That way, other than some simple initial setup, parents are automatically apprised on events, updates and cancellations in various formats.
Getting the whole thing to be as un-computer-like as possible while leveraging great tech has been a huge challenge but a worthy one.
Hi Jon,
I was drawn in and had to identify myself and my efforts to you. I’m manually scraping and mashing up the data to create the basic value of my site Deaf Access Films dot com.
I’ve been fumbling through WYSIWYG tools and repeatedly trying to simplify things for myself and for the users I hope to serve, for about 10 years so far.
I’d love to offer my site or self up for vivisection by Raymond Yee for a chance to participate and learn his mash up discipline, with the real-world user ever in mind – as you note in the interview.
FireFox and some of its tools have helped ad bit of valued speed to scrape paste, sort, store, etc.
I’m gearing the site as for-profit, trying to make it self-sustain on film marketing budgets, as my service helps Hollywood sell more tickets to Deaf. All future tense, so far.
If you ever get a chance to check us out or give some feedback, that would be wonderful.
thanks,
Greg
It is great to see that someone is doing this important work. what’s the url of the site you’re working on?
I’m looking at http://elmcity.info/events — Is that the site you’ve constructed with all the aggregate Calendar info?
Unfortunately, it doesn’t give me a means of integrating (importing) it with my own electronic calendar. No .ics files, no microformats… It’s little more useful than posting a .PDF to the Website. Yes, I can go to each individual website referenced and grab the calendar info from there, but the value added by the aggregation of all the different sites is itself inaccessible.
–Bob.
> even people who have adopted iCal/Outlook/gCal don’t
> interact with them very often – most will check email, social
> networks, cell phones, and desktop widgets far more frequently.
On the consumption side, agreed. But on the production side, some kind of calendar program — or equivalent UI that’s aware of dates and times and venues and recurrence — is required.
Seems important to show people that the iCal/Outlook/gCal apps are a good way to plug in events that then flow to all the various sites and modes of consumption.
> the value added by the aggregation of all the different sites is
> itself inaccessible
There is not one person in a 1000 in this community — and probably in most — who expects to consume .ics files or microformats directly.
Eventually I want to show how and why to do that. But where I have erred in the past is in presenting that concept up front, before motivating it properly.
So this time around I’m taking a first-things-first approach. Get the production flowing, and use the aggregator to show the effects that are possible. Then I’ll start talking about alternate modes of consumption.
Actually there is one alternate mode happening already. See: http://citizenkeene.ning.com. Any other site can do the same by sourcing a JS file.
> FuseCal
Handy! Thanks!
> our calendar product is primarily designed for schools and
> school districts
I have encountered an instance of it!
Namely: http://mrhsnhus.intand.com/
Nice job!
That one is for my town’s neighboring school district. I’m trying to show folks here how and why this approach is so much more useful than the weekly.pdf that our own school district currently publishes.
But how will people realize that iCal structured data is so much better than a weekly.pdf? without having structured calendar data yourself it’s going to be difficult to convince others to structure theirs.
Sorry, I don’t mean to slag your efforts — I run into the same issues with clients, and ususally just gnash my teeth and quietly re-key the data. I love what you’re doing, and am impatient for the rest of the world to catch up…
–Bob.
> But how will people realize that iCal structured data is so much
> better than a weekly.pdf?
My theory is that this can only happen in the teachable moment when organization B asks the question: “Why isn’t our stuff included in the combined calendar, like organization A’s stuff?”
Answer: “Great question! Let’s compare the methods that organizations A and B are using to publish their calendar info.”
Great article, Jon!
This profoundly resonated with all of us on the FuseCal team. We’re all about adding structure to the unstructured (or understructured) calendar info out on the web… Great to see that we’re not alone in that vision of the future.
I’d love to know if there’s any way we can help with your project, or if you have any feedback on how we can improve FuseCal, to make it even more useful towards your goal.
cheers,
Matt Gillooly
Product Manager, FuseCal.com
> I’d love to know if there’s any way we can
> help with your project
You already have! FuseCal is doing a better job than I was able to do with this:
http://www.libraryinsight.com/calendar.asp?jx=ea
I was too lazy to parse out the times, I only got dates. But you got it all. Excellent!
Will the FuseCal feed that I made persist even though I made it as an anon user?
One interesting detail here: I’ve run into a few places where it’s helpful to look ahead. So in this case, the current month is almost over, and you’d like to look ahead to:
http://www.libraryinsight.com/calendar.asp?sm=&jx=eap&nMonth=5&nYear=2008&x=3
I was doing that in my otherwise inferior scraper. It’d be nice to be able to give FuseCal a rule about looking ahead.
> If you have any feedback on how we can
> improve FuseCal
That’s hard to answer because ultimately I don’t think FuseCal should have to exist. In general, I would like to see the effort that’s being invested in reverse-engineering of web pages — at FuseCal, at EveryBlock, and elsewhere — spent on a) building systems that invite people to manage information better from the get-go, and b) enabling all the great things that would happen if they did.
Here in the real world, of course, what you are doing matters enormously, and will for a long time to come. How to build bridges from here to a very unevenly distributed future is a very interesting question, I’d be interested to know your thoughts.
Hmm, interesting post. Food for thought.
Thanks,
John
Hey Jon,
To answer your first question, yes the feed will persist, as long as your calendar app continues to request updates. We do periodically clear out anonymous feeds that haven’t had any hits in a while. However, I’d still recommend registering an account, so that you can manage your feeds in the future and from other computers. If your browser session hasn’t yet expired, your existing feeds will automatically get added to the account you create. This will let you add and change event filters, tag your calendars and subscribe to tag feeds, embed your calendars into web pages, opt in to emails alerts about changing events, and more.
Automatically handling the look-ahead is on our (very long) todo list. In the meanwhile, one hack would be to choose a unique tag to represent that calendar. As you add the URL for each month of the calendar to FuseCal, you would use that tag and the events would be added to the corresponding tag feed. In that way, you can at least create a single feed with all the events you want.
I agree that, ideally, FuseCal wouldn’t have to exist- in the same way that, ideally, hospitals and prisons wouldn’t have to exist. :-)
Seriously, though, I think you need to provide a lot of incentive in order to get people to change the way they behave. It’s much easier to sell a Tylenol than a vitamin. I think you’re on the right track with waiting for the “teachable moments.” I’m not totally sure if it’s even possible to build a startup around providing those incentives, but I think it makes a lot of sense as a non-profit community project, which is part of the reason I recently founded the MashableCity.org project.
That said, I’d love to be proven wrong and shown how to make money while fighting the good fight!
cheers,
Matt
1. Check out http://www.calconnect.org, the Calendaring & Scheduling Symposium, they have been working on calendar interoperability for quite some time. Microsoft only recently (in the last year?) became a member, but now all the major players are.
2. I posted to their site two XSLT files which transform iCalendar to XML and vice versa. The iCal->XML transform currently requires you to embed the iCalendar data in a small bit of XML to make it an actual ‘document’ processable by XSLT. These two files are freely usable by anyone, just provide attribution (see the comments). I personally believe in the power of XSL transforms in combination with XML to make all sorts of data usable in all sorts of ways.
3. The IETF workgroup on iCalendar simplification has been making headway as well (but XML is not in their charter at the moment).
The missing piece, in addition to the education part that you mention, is publishing. It is currently not that easy to take a list of events in some program (even a calendaring program) and publish them all to the web where each event has an .ics file link that will open in the user’s calendaring tool and perhaps the entire group of events has a link to an .ics file which includes them all. Therefore people publish their calendar events with things they know how to use, like Word or Excel.
Tim-
have you checked out FuseCal’s publishing tools? We have an interface that anyone can embed in their web page to allow their visitors to choose events and subscribe to them, similar to what you’re suggesting. If you get a chance to check it out, I’d love to hear your thoughts.
Jon-
This relates to something I wanted to clarify about my earlier comment. Joking aside, we do fully expect the info extraction aspects of FuseCal to become less important as people do embrace calendar data standards. That’s actually a very good thing for us, because the other parts of FuseCal – publishing and syndication tools, analytics, and the potential to monetize events through contextual analysis of the events – all become much more interesting when there is more structured event data available.
However, we’re stuck first with the problem of accumulating that mass of structured event data. For now, our answer is to create that structure ourselves, and I think it will be a long time before the need for that service completely disappears.
> I think it will be a long time before the
> need for that service completely disappears.
I think that’s a very good bet!
Matt – the demo looks interesting, but not exactly what I’m talking about.
What I’m thinking of:
1. Select the group of events in your calendaring program of choice.
2. Choose Publish (or whatever it’s called)
3. Supply destination information (directory, URL)
4. The application creates one .ics file for each event, an .ics file containing all the events, and an XML file with links to all of those; an XSL file is used to transform it into whatever HTML you like for viewing (yes, you could just go to XHTML I guess)
The point here is to make it painless to publish. What FuseCal does, it seems, is make it painless to subscribe to a calendar once published.
hel;o! If somebody knows, tell me! I can’t find an informttion about spiders. What spiders the most poisonous and what type of spiders exist? if you have some information about this, I will be glad to read it :) I will be grateful
_________________
| cialias erection |
Cheap cialis works in much the same way as Sildenafil Citrate, but is much faster, lasting up to 36 hours vaigra generic soft tabs ! For viagra citrate very fast delivery worldwide. Save your time – buy tadalafil pill on line! Easy to get tadalafil pills online. Find out more about ed treatment at the famouse calis low cost site…
I do not understand exactly how I discovered your blog because I had been researching information on Real Estate in Oviedo, FL, but anyway, I have enjoyed reading it, keep it up!