I started with the two web applications that make publishing calendar feeds as simple as it can be: Google Calendar and Hotmail Calendar. Here’s the new and improved explanation of How to publish a calendar feed from Google Calendar or Hotmail Calendar. I’ll soon learn how well this new explanation works for the intended audience. Meanwhile, here are some notes on the techniques I discovered.

Using sparkcasts

My earlier explanations didn’t use screencasts, they were just textual narrations with embedded static screenshots. That’s an appropriate way to explain step-by-step procedures. Screencasts can be overkill.

But it is nice to be able to show motion and sequential flow. I wondered: if sparklines are “intense word-sized graphics” that appear inline with text, is there a similarly lightweight way to use screencasts? Well, that’s how this explanation works. The steps are all written as text. Each step comes with a Watch! link that opens a quick little inline screencast.

Using Firebug to prepare screencasts

As I mentioned the other day, Firebug is an amazing tool for making mockups based on live web pages. And sometimes you can do more than just take screenshots of those mockups. Sometimes you can interact with them.

I used that method to temporarily alter both Google Calendar and Hotmail Calendar for these sparkcasts. In my case, both include several private calendars that were irrelevant to my explanation. Using Firebug I was able to remove them. I also made the address shown in Gmail look like the generic YourAccountName@gmail.com instead of judell@gmail.com. If subsequent interaction with the page doesn’t trigger a refresh of these altered elements, the changes will persist. Handy!

Using Camtasia to make animated GIFs

A movie-style screencast is a fairly heavy object, and it would be cumbersome to embed lots of these on a page. Animated GIFs are lighter. And it turns out that Camtasia can produce that format. Nice!

Using jQuery to reload animated GIFs

On the first try, I made each sparkcast loop forever, and used jQuery to show only the current one. But that wasn’t ideal. What I really wanted was for each sparkcast to play once when shown, then stop and offer a Replay link. If you view the source of this page, you’ll see the solution I came up with. It turns out that when you dynamically reload an animated GIF, you need to decorate its URL in a unique way; otherwise the browser won’t replay it.

Using Image Magick to control the looping of animated GIFs

Once I figured out how to replay a play-once GIF, I had to make all the GIFs stop looping. I’m sure that PhotoShop and others of its ilk can do that, but I like Image Magick’s ability to automate repetitive tasks. The Image Magick incantation to make an animated GIF play once goes like this:

mogrify -loop 1 howto-01.gif

Now that I’ve written this down, I can forget it until I need it again!

Software teleprompters

After writing the script, I realized I’d need a teleprompter in order to read it effectively into the camera. You’d expect to find lots of software prompters floating around on the web, including some free ones, and you’d be right about that. But I had to work through a bunch of them before I found one that worked well for me. I tried CuePrompter, TeleKast, and many others. All failed in some dimension of control: margins, speed, transport. Finally I settled on PromptDog, which is free to try but is the one I’ll buy when I go this route again. It does everything well, but what really put it over the top for me was the way it wires scroll speed to the mouse wheel.

If you’ve read from a software-based teleprompter before, you’ll already know this, but it was new to me. No matter what scroll speed you choose, you need to vary it as you go along. That’s because words and sentences take varying amounts of time to speak, but you need to keep your eyes focused near the top of the screen where the camera sits. With most of the programs I tried, you manage this focal zone by stopping and starting the scroll. But for me, at least, the stops and starts were distracting. PromptDog’s mouse-wheel-driven variable speed control made it much easier to stay in the focal zone. Reading from a software teleprompter is hard, at least for me. I was happy for all the help I could get.

Picture-in-picture video

For this screencast, I upgraded from Camtasia 5 to Camtasia 7. It can record directly from a camcorder, but my second-hand Panasonic PV-GS400 doesn’t seem to work well in that mode. So I recorded to tape, imported the results to a file, and imported that file into Camtasia as a PIP (picture-in-picture) video. On import you tell Camtasia how big your PIP window will be, and where it will show up in the larger video window. I made the PIP window a quarter the size of, vertically centered in, and flush right with the larger 1024×768 video window.

I’d sssumed you could move the PIP window around, and grow it or shrink it, to accomodate different kinds of underlying screencast action. But that assumption was wrong. For a given segment of PIP video, the window stays where you put it. This leads to my first feature request for Camtasia 8: a PIP preview rectangle when recording the screen.

Often it’s OK to let the PIP video just overlay the screen action. But sometimes you don’t want it to hide an essential part of the screen. To avoid that, you have to compose the screen around the PIP window. Lacking a visual cue for the PIP window’s borders, I had to guess. Often I guessed wrong, and had to recompose and reshoot a piece of screen action.

Note that you can vary the size and location of the PIP window by splitting the PIP video into segments and assigning different sizes and locations to each segment. That’s a lot of work, though. And you don’t really want to split the PIP video into segments because then you can’t manipulate the whole track.

Editing audio, motion video, and screen video all together

I made things hard on myself because I’d forgotten that Camtasia invites you to do more integrated editing than you should. In principle you can, for example, run a noise-reduction pass on your audio in Camtasia. In practice, I would prefer to do that in Adobe Audition, which does the job faster and better. What I should have done is grab the sound track out of the captured motion video, run Audition’s noise reducer, recombine the audio and video, and then import into Camtasia for editing.

Instead I edited everything down in Camtasia, then tried to do an export/process/import pass on the audio. When you export, Camtasia renders the audio based on your edits. Unfortunately it came out a few seconds longer than expected. I think that’s because the differing frame rates for screen video on the one hand, and motion video plus audio on the other, make it hard to keep things in synch. Next time around I’m going to try matching the frame rates to see if that helps.

(In the end I decided it was worth redoing the edit anyway, so I split the AVI file I’d recorded from the camcorder, fixed the audio, imported it back into Camtasia, and redid the relatively few edits I’d made to the PIP video.)

In the past, I’ve done some carefully edited screencasts where things that I say are tightly synched to things happening on screen. (“…when I click on this link, we see that …) It’s easy to pull that off when you can’t see the speaker, because you can mess with the screen video, or the audio, or both. When you can see the speaker, it’s much harder. Motion video isn’t nearly so forgiving as audio, so you have to do almost all the synch adjustment in the screen video. Or else re-record some or all of the motion video.

To PIP or not to PIP?

Is all this effort worth the trouble? When Scott Hanselman surveyed his readers about screencasts, he asked, among other things, “PIP or no PIP?” More than half agreed with the statement: “Too much PIP (Picture in Picture) video of the presenter is distracting.” And I think that’s true for screencasts that show how to do stuff with software.

When a screencast shows why to do stuff with software, though, I think the talking head may make more sense. Now, my instinct is to be a voice only, as I am on my podcast. But if the screencast is going to represent me at an event, it seems like I should try to project myself there.

More broadly, the topic is something I care about and have struggled to communicate effectively. If this method of presentation works better than others I’ve tried, even if only for some people, then it’s worth doing. My communication kit needs as many tools as I can pack into it.

Now that I’ve knocked the rust off my screencasting skills, I’m looking forward to redoing this video based on feedback. And since it was made for a ten-minute conference slot, I should probably also do some shorter versions that will work in different contexts.

One thing that’s becoming terribly clear: If I want these to make sense to broad audiences, I need to speak plainly and illustrate with simple everyday examples. I’ve been embarrassingly slow to figure that out, but I am learning. In the screencast I just wrapped, which is all about syndication, I never use that word. It’s a start!

Using deepzoom.py to create Deep Zoom images and collections

I’m using this Python component to create Deep Zoom images and collections. I made the following changes to it:

1. tile_size=256 (not 254) at line 59, line 160, and line 224

2. source_path.name instead of source_path at line 291

3. destination + '.xml' instead of destination at line 341

Let’s assume that Python is installed, along with the Python Imaging Library, and that your current directory contains the files 001.jpg, 002.jpg, and 003.jpg:

001.jpg
002.jpg
003.jpg

For each image file, you could run deepzoom.py thrice from the command line, like so:

python deepzoom.py -d 001.xml 001.jpg
python deepzoom.py -d 002.xml 002.jpg
python deepzoom.py -d 003.xml 003.jpg

My script doesn’t actually do it that way, it enumerates JPEGs and instantiates deepzoom.py’s ImageCreator object once for each. But either way, for each JPEG you end up with a DZI (Deep Zoom Image) package that consists of (for 001.jpg):

• A settings file: 001.xml
• A subdirectory: 001_files
• More subdirectories (named 0, 1, etc.) inside 001_files
• JPG files inside those subdirectories

Now, in this case, the current directory looks like this (using -> to mark additions):

001.jpg
-> 001.xml
-> 001_files
002.jpg
-> 002.xml
-> 002_files
003.jpg
-> 003.xml
-> 003_files

To build a collection, do something like this in Python:

from deepzoom import *
images = ['001.xml','002.xml', '003.xml']
creator = CollectionCreator()
creator.create(images, 'dzc_output')

Now the current directory looks like:

001.jpg
001.xml
001_files
002.jpg
002.xml
002_files
003.jpg
003.xml
003_files
-> dzc_output.xml
-> dzc_output_files

The Pivot collection’s CXML file will refer to dzc_output.xml, like so:

<Items ImgBase="dzc_output.xml">

Using IECapt to grab screenshots

This tool uses Internet Explorer, so only works on Windows. There is also CutyCapt for WebKit, which I haven’t tried but would be curious to hear about.

Here’s an example of the IECapt command line I’m using:

iecapt –url=http://blog.jonudell.net/… –delay=1000 –out=tmp.jpg

The result in most cases is a tall skinny JPEG, because it renders the whole page — which can be very long — before imaging it. When I ran it over a 600-item collection, it hung a couple of times because of JavaScript errors. So I went to Internet Options->Browsing in IE, checked Disable script debugging, and unchecked Display a notification about every script error.

Using ImageMagic to crop screenshots

Here’s a picture of an image produced by IECapt, overlaid with a rectangle marking where I want to crop:

The rectangle’s origin is at x=30 and y=180. Its width is 530 pixels, and height 500. Here’s the ImageMagick command to crop a captured image in tmp.jpg into a cropped image in 001.jpg:

convert -quality 100 -crop 530×500+30+180 -border 1×1 -bordercolor Black tmp.jpg 001.jpg

I’m writing this down here mainly for myself. ImageMagic can do everything under the sun, but it always takes me a while to dig up the recipe for a given operation.

Parsing the WordPress export file

I found to my surprise that WordPress currently exports invalid XML. So the script starts with a search-and-replace that looks for this:

xmlns:wp="http://wordpress.org/export/1.0/"

And replaces it with this:

xmlns:wp="http://wordpress.org/export/1.0/"
xmlns:atom="http://www.w3.org/2005/Atom"

Then it walks through the items in the Atom feed, extracting the various things that will become Pivot facets. For the description, it tries to parse the content:encoded element as XML, and find the first paragraph element within it. If that fails, it just treats the element as text and grabs the beginning of it.

Weaving the collection

There are two control files that need to be synchronized. First, there’s dzc_output.xml, for the Deep Zoom collection. It has elements like this:

<I Id=”596″ N=”596″ Source=”2245.xml”>

Then there’s pivot.cxml which drives the visualization. It has elements like this:

<Item Id="596" Img="#596"
  Name="Freebase Gridworks: A power tool for data scrubbers"
  Href="http://blog.jonudell.net/2010/03/26/...
<Description><![CDATA[
I've had many conversations with Stefano Mazzocchi and David Huynh [1, 2, 3]
about the data magic they performed at MIT's Project Simile and now perform
at Metaweb. If you're somebody who values clean data and has wrestled with
the dirty stuff, these screencasts about a forthcoming product called
Freebase Gridworks will make you weep with joy.
]]></Description>
<Facets>
  <Facet Name="date">
    <DateTime Value="2010-03-26T00:00:00-00:00" />
  </Facet>
<Facet Name="tag">
<String Value="freebase" />
<String Value="gridworks" />
<String Value="metaweb" />
</Facet>
  <Facet Name="comments">
    <Number Value="24" />
  </Facet>
</Facets>
</Item>

In this example, Source="2245.xml" in dzc_output.xml refers to a Deep Zoom image whose name comes from the WordPress post_id for that entry, which is:

<wp:post_id>2245</wp:post_id>

But Id="596", which is the connection between dzc_output.xml and pivot.cxml, comes from a counter in the script that increments for each item processed. I don’t know why the numbering of items in the WordPress export file is sparse, but it is, hence the difference.

Things to do

Here are some ideas for next steps.

1. Check the comment logic. I just noticed the counts seem odd. Maybe because I’m counting all comments instead of approved comments?

2. Use HTML Tidy to ensure that item content will parse as XML, and then count various kinds of elements within it: tables, images, etc.

2. Use APIs of various services — Twitter, bit.ly, etc. — to count reactions to each item.

There’s one by David, and another by Stefano. Using common public datasets about food, international disasters, and US government contracts, they fly through a series of transformations that:

• Merge similar names using a host of methods:
  • Automatic title-casing
  • A rich expression language
  • Analysis of “edit distance” between similar phrases, using several clustering algorithms
• Split multi-valued facets
• Create new facets (e.g., a year column from a data column)
• Morph linear scales to log scales where appropriate

It’s all live, undoable, and fully instrumented, by which I mean that every transformation updates the counts of the values in each facet, and displays histograms of the new distribution of values — along with sliders for selecting and focusing on subsets.

As the open data juggernaut picks up steam, a lot of folks are going to discover what some of us have known all along. Much of the data that’s lying around is a mess. That’s partly because nobody has ever really looked at it. As a new wave of visualization tools arrives, there will be more eyeballs on more data, and that’s a great thing. But we’ll also need to be able to lay hands on the data and clean up the messes we can begin to see. As we do, we’ll want to be using tools that do the kinds of things shown in the Gridworks screencasts.

When I first got interested in screencasting, I envisioned the medium not only as a way to demonstrate software, but also as a way to share knowledge at Internet scale. Sal’s work fulfills that vision, and points the way toward a profound and much-needed disruption of our educational system.

At its core, Sal’s project isn’t about YouTube screencasts. It’s about intuition.

I always got frustrated by what went on in the classroom. You see otherwise intelligent peers memorizing facts and not really caring about the actual intuition. And because they didn’t care about the intution in their junior year, when that same idea pops up in senior year, it’s like they’ve never seen it before. It boggled my mind. You’re just relabeling the same concept over and over.

Sal cares about the intuition, and he wants others to care about the intution too. The first beneficiary of that desire was his cousin Nadia, whom he tutored remotely. Then followed other cousins and family friends. Then it dawned on him that there were no limits. The project could scale out. He could become a superempowered individual, reaching anyone who finds value in his method.

One of the key ingredients of that method is improvisation. These videos aren’t carefully planned, and they aren’t edited. As a viewer, you find yourself looking over the shoulder of a smart and broadly knowledgeable person who is solving problems by thinking on his feet. You watch a practitioner at work: engaged with his medium, wrestling with his tools, correcting false starts.

It was Chris Gemignani who first showed me the value of this approach, in a screencast that teaches how to do unexpectedly powerful and elegant Excel charting. He did it in one take. I’d have been tempted to edit out the false starts. But Chris knew better. Learning how a practitioner really thinks about solving a problem is even more valuable than learning the solution to the problem.

One thing that Sal’s lessons can’t be, of course, is interactive. Nor does he pretend that these videos will make teachers obsolete. But he does suggest, and I violently agree, that teachers can and should become curators of online assets like the ones Sal is creating, and should know when and how to weave those assets into their classes.

Teachers should also become connectors. Sal won’t be the only game in town. Other superempowered tutors will emerge. Each will have a unique style. For a given student, a given subject, and a given problem, one or another of those styles may be right. The best teachers will know their own strengths and limitations, will know which online tutors complement their strengths in a variety of ways, and will connect their students with those tutors.

Sal Khan is on fire. He burns with a passion to share his intuitions with anyone and everyone. It is a beautiful thing to see. He has abandoned a lucrative career in finance to do this fulltime, and I am quite sure he will find a way to keep doing it.

PS: The title of this piece refers to Richard Ankrom’s Los Angeles freeway project. At a busy intersection, millions of motorists have been directed to North 5 by a sign that Caltrans omitted. Ankrom created and installed that missing sign.

PPS: I wrote to my son’s math teacher about Sal Khan. She replied: “Thanks for that link to the Khan Academy. I was overwhelmed by how many video lessons he has! He does seem like an inspiring man. Unfortunately, You Tube is blocked here at the high school.”

I’m sure there are thousands of Web developers out there right now asking themselves “would my app be better if users could see each others’ edits in real time?”,”should we add a playback feature to our service as well” [ed note - wikipedia could really use this] and “why don’t we support seamless drag and drop in our application?”. All inspired by their exposure to Google Wave.

Indeed, every application that preserves a change history needs playback. Wikipedia, as Dare notes, is a prime candidate. Back in 2006, I made this LazyWeb request:

Animation is the best way to visualize the flow of change, as I discovered when I made my Wikipedia screencast. For Wikipedia, and indeed for all kinds of living documents supported by revision history and diff tools, I can imagine being able to isolate a paragraph or section and autogenerate the screencast of its evolution. I can even imagine the content of such visualizations being considered not just cutting-room floor debris but, rather, part of the “real” document, like footnotes.

Andy Baio responded by sponsoring a contest for a tool that would do just that. And I made a screencast demonstrating Dan Phiffer’s winning entry.

That script is unavailable at the moment because, ironically, Dan’s server reports:

Oh noes! I got HACK*D. I’m sifting through my files and should restore things back to normal soon.

In any case, it probably wasn’t practical for routine use. Fetching every revision on the fly really hammers Wikipedia. What’s really needed — again, not just for Wikipedia but everywhere — is a general way to query change history, and return a stream of versions and differences.

One way of doing the latter would be to use FeedSync, an open extension to RSS/Atom that supports synchronization in Live Mesh. Another would be to use Google’s Wave protocol. Because FeedSync deals with lists of items, which can be arbitrary chunks of content, whereas Wave deals with lists of document-mutation operations, like delete-element and start-annotation, it seems to me that FeedSync is more general, albeit less immediately useful for collaborative editing.

To explain why generality matters, consider change animation in a very different domain: software configuration. My wife, for example, sometimes changes her settings — in Word or Firefox — in ways that cause problems. If these apps persisted their settings to Live Mesh, as they could and arguably should, I’d be able to debug a mishap locally or remotely. But ideally, the change visualization would be sufficiently user-friendly so that she’d have a shot at figuring it out for herself.

PS: Speaking of history and restoration, I’ve been feeling like an amnesiac ever since my InfoWorld archive went dark. So in spare moments I’ve been reconstructing and republishing it. I’ll have the text of all the old blog entries up soon. And I’ve been restoring the screencasts as well. I’m keeping track of my progress at delicious.com/judell/screencast+restored.

There were a couple of surprises for me. First, I was intrigued to learn about Jean-Claude’s vision for mechanized research. I’ve always thought of open notebook science as a way to speed up the iterative cycle of research and publication, and to engage more human minds in collaboration. Of course Jean-Claude thinks so too. But he also thinks that when data are published in accessible formats, and exposed to computational processes running in the cloud, we’ll be able to automate certain aspects of research.

It reminds me of George Hripcsak’s effort to mechanize the interpretation of electronic health records. In general, we’re collecting way more data than the collectors can analyze. Crowdsourcing is one solution to this problem. Mechanization is another. We’ll need both.

The other surprise was hearing about Drexel’s fairly aggressive use of Second Life. I’ve been an amused skeptic on that front, but Jean-Claude’s passionate advocacy requires me to rethink that stance.

What didn’t surprise me, but might well surprise tuition-paying parents of Drexel students, was Jean-Claude’s attitude toward the classroom. He mostly doesn’t see a need for it. The content delivery aspect of education, he feels, is best handled in other ways, including screencasts and podcasts as well as traditional texts. There can, and should, be a range of sources, to accommodate the differing inclinations of learners. And teachers need to be competent producers and orchestrators of those sources. But for Jean-Claude, the best way to engage directly with students is to meet with individuals, not with whole classes.

Now admittedly, a chemistry class doesn’t invite and thrive on group discussion in the same way that, for example, a literature class does. And yet Jean-Claude says that a literature class was one of the models for his use of Second Life. When group interaction is central to the educational experience, he thinks that virtual environments — though he doesn’t require their use — may outperform real ones.

I remain skeptical on that point, but I’m always open-minded, so I hope Jean-Claude will take me up on my offer to visit one of his virtual environments and document the interactions that happen there.

But from Phil’s perspective and mine, this is a seriously useful tool as well. He’s planning to annotate screencasts with it. And I found a great use for it here.

That short video features Bob Coffey, the senior climber at our YMCA. When I made and posted the video, I wasn’t quite sure how senior Bob was so I didn’t say. Yesterday I remembered to ask. Turns out he is 79.

It’s painful to add new information to a video. Opening up the raw file (if you even kept it around), adding a caption, recompressing, reuploading — it’s too much overhead, and unless there’s a compelling need you’re just not going to bother.

Of course you can update the textual wrapper, and alter the title or description. But in this case, I didn’t want to that. The information is much more effective when inserted midstream. After he’s scampered halfway up the wall, the popup annotation saying “Bob Coffey is 79 years old” makes the point more subtly and powerfully.

The point, by the way, is that we can do more, physically, at all ages, than we think. I’ve known a few people over the years who have redefined what’s possible, and it’s always an inspiring thing to see.

The format is an audio podcast and a blog, where the blog provides a partial (but substantial) text transcription in order to make these conversations accessible to folks who don’t listen to podcasts, and also to expose them to the Net’s ecosystem of search, linking, and aggregation. Where appropriate, I’ll also use screencasts to show software in action.

Perspectives runs on the same publishing platform that supports Channel 10 (for enthusiasts), Channel 8 (for students), TechNet Edge (for IT pros), and VisitMIX (for Web designers and developers). (Channel 9, the original site, will migrate to this platform too.) Perspectives intersects with the interests of all these sites, but it doesn’t really belong in any of them, so we’ve created an independent home for it. Thanks to the EvNet team, especially Duncan Mackenzie, David Shadle, and Jeff Sandquist, for making that happen.

The first episode, with Henrik Nielsen and Tandy Trower, explores the Microsoft Robotics initiative. We discuss why robotics is — as futurist Paul Saffo believes — a Next Big Thing. And Henrik and Tandy explain how the concurrency and decentralized-services infrastructure that supports the robotics platform is broadly relevant in an era of loosely-coupled services.

In The social scripting continuum I suggested that a system like CoScripter could automate desktop and web applications in a common way. Here’s one way to think of the benefit of doing that. Today, I can share software-related task knowledge in a social manner by creating and posting screencasts. But you can only watch a screencast. If I could instead share that task knowledge in the form of standardized high-level scripts, you wouldn’t need to watch the screencast. Of course, you might want to, for other reasons, but not simply to get the procedural knowledge transferred from my brain and fingers to yours.

Given how popular screencasts have become in three years, I’ve got a hunch that taking things to that next level would be huge. And lord knows I’d love to be able to convey packages of procedural knowledge to my mom that way.

Preparation

As the interviewee, you need not prepare anything. Your life is the preparation. You might, however, want to help me prepare, by referring me to background materials that I may not already know about. I don’t show up with a script in mind, but I do like to be as informed as I can be.

Recording

My preference is that you use a landline, not a cellphone or a speakerphone. If you have a strong preference for Skype, I can accommodate.

Either way, it’s ideal if you can make a decent recording of your half of the call — for example, by using a USB microphone plugged into your computer, or a standalone digital audio recorder — and convey that recording to me as uncompressed audio. It’s easy to splice the two halves of the conversation together in post-production, and if you got a decent result on your end, the combined result will be way better than any current scheme for squirting audio through a long-haul network. If the local recording doesn’t pan out, we’ll just fall back to the phone recording that will occur in parallel.

Editing

As discussed in this essay on the audio digital darkroom, I’m fairly aggressive about editing audio interviews. As a result, you and I will come out sounding somewhat better than we really are. I do this out of respect both for the listeners’ attention, and for the importance of the ideas we’re discussing.

The amount of editing varies from show to show. Some hour-long interviews have produced twenty-minute shows, other hour-long interviews have produced fifty-five-minute shows. I would say the compression is normally in the ten-to-fifteen-percent range. In all cases, I apply one rule: Focus on the most interesting and important stuff. Interviewees have so far always been pleased with the results.

One of the useful consequences of this approach is that, since you know there’s a safety net, you can relax, there’s no pressure to perform flawlessly, and we can work together to capture the interesting and important stuff.

Will there be a role for IronPython (or IronRuby) here, someday, such that you could use these languages inside Excel? That’d be very cool.

Several folks suggested that I should take a look at Resolver, an IronPython-based spreadsheet that deeply unifies Pythonic object-oriented programming with the sort of direct manipulation that makes the spreadsheet so useful. Resolver was and still is in private beta, but today’s screencast (Flash, Silverlight) will give you a good sense of what it’s all about.

The presenters are Giles Thomas, managing director and CTO of Resolver Systems (and creator of his own Resolver screencast), and Michael Foord, who blogs about Python, contributes to the IronPython cookbook, and is also working on the forthcoming book IronPython in Action.

If you are (or would like to be) using Python to wrangle business data, Resolver will make sense immediately. You’ll love the idea of wielding Python’s powerful data manipulation features in that context. You’ll appreciate what it would mean to harness not only the Python standard libraries but, because Resolver is IronPython-based, also the .NET Framework and the universe of third-party .NET assemblies. And you’ll be intrigued by the way in which the IronPython code that represents and animates a Resolver spreadsheet can be reused elsewhere — for example, in web applications.

But there’s more to the story. Because a cell in a Resolver spreadsheet can contain a reference to any .NET object, Resolver creates, as Giles Thomas says, “a somewhat pathological but entirely new way of programming using a spreadsheet.” You can, for example, define an anonymous function — say, a function that returns the square of its argument — and store it in cell B4. Then you can place a value — say, 5 — in cell A2. Then you can store this formula in cell B6:

=B4(A2)

That says: “Apply the squaring function in B4 to the value in A2.” The result in B6 will be 25.

I’ve long argued that the interactive and exploratory style of dynamic object-oriented languages is an important but underappreciated benefit. As I may have mentioned before, IronPython’s creator Jim Hugunin told me that when he first showed IronPython to folks at Microsoft, he was surprised by their reaction. He thought the big wow would be IronPython’s ability to streamline and accelerate use of the .NET Framework. But while people did appreciate that, they were truly wowed by something that’s second nature to every Python programmer — the read/eval/print loop which traces all the way back to the earliest Lisp systems.

It is a magical and powerful thing to be able to explore and modify a running program’s code and data. From those early Lisp systems to today’s Python and Ruby implementations, we have been doing that exploration and modification using a command line.¹ We can trick it out with recall, name completion, and search, but it’s still a command line with all the limitations that entails. If I’ve defined an object A and stored some code or data there, my definition and invocations of A will scroll out of view as I continue to work. They won’t be visually persistent.

In a Resolver spreadsheet, these objects are visually persistent. I haven’t yet got my hands on Resolver, but here’s an example of what I think that will mean. Suppose that I have a data set I want to transform, against which I’m testing five different versions of a transformation function. I’d put the data in cell A1, the functions in cells B1..B5, and the results in C1..C5. Now I’ll see everything at a glance. The spreadsheet that would conventionally have been the results viewer at the end of a series of tests becomes the environment in which the tests are written, performed, and evaluated.

The spreadsheet is also an important bridge between programmers and their business sponsors. It’s no accident that Ward Cunningham’s FIT (Framework for Integrated Test) was originally inspired by Ward’s experience of inviting business analysts to write test cases in spreadsheets. In its current form, FIT uses HTML tables in a wiki as the bridge between analysts who write tests and developers who write the code that must pass those tests. I think Resolver and FIT may prove to be a marriage made in heaven.

While Resolver will initially appeal to business programmers who appreciate Python as a language, and IronPython as a way of leveraging the .NET Framework and .NET-based business logic, the ideas it embodies transcend Python and .NET. I’ll be fascinated to see how this “pathological but entirely new way of programming using a spreadsheet” will evolve.

Some important variables, from a screencaster’s perspective, are: legibility, file size, and convenience of production, deployment, and viewing.

That legibility matters seems obvious, but I see an awful lot of screencasts delivered at squinty resolutions. This puzzles me. The purpose of a screencast is to show and describe on-screen action. If you can’t read the screen, what’s the point?

All four of these examples are legible. The Quicktime version achieves the best clarity, but there’s a tradeoff: it’s also the largest file.

That size matters is perhaps less obvious to those of us living in the developed world. But as I’ve been recently reminded by both Beth Kanter and Barbara Aronson, much of the world remains bandwidth-challenged. Videos that don’t squeeze themselves down will not be seen in many places where they should be.

Among these four examples, Windows Media weighs in lightest at under half a megabyte. That works out to about a megabyte per minute, which is the target I like to shoot for. If it’s possible to deliver a legible screencast at a data rate significantly less then that, I’d like to know how.

The sizes of the other versions in this example, in ascending order: Flash 1.2MB, Silverlight 1.5MB, Quicktime 2MB.

Of course these sizes depend on which encoder is used, and on which settings are applied. For these tests, I produced all of the screencasts in Camtasia. For Quicktime and Windows Media, Camtasia uses the encoders that come with those platforms. For Flash, it supplies an encoder. For Silverlight, it doesn’t yet supply an encoder so I produced an uncompressed AVI and then used Expression Encoder to create a Silverlight-compatible WMV file.

I should add here that, despite all the work I’ve done in this area, I’m still a bit vague on the concept of a screen encoder — that is, a video encoder that’s tuned for the kinds of low-motion but text-rich content that’s typical of screencasts. In beta versions of Silverlight and Expression Encoder, for example, there wasn’t a screen video option, so the only way to produce a legible screencast was to crank up a motion-video encoder to the maximum data rate, which produced a massive file. Now Expression Encoder provides a screen encoding option, which I used for this test and which Silverlight 1.0 can obviously play back.

It seems to me that Camtasia should be able to use that encoder directly, but until I figure out how, it will be less convenient to produce Silverlight screencasts from Camtasia than to produce the other formats. Rendering to AVI as an intermediate step is doable, but time-consuming.

In terms of deployment convenience, one measure is the number of supporting HTML, JavaScript, configuration, and other files required in order to play a screencast. I’m a minimalist, so when I deploy Camtasia screencasts I throw away the wrappers that Camtasia generates and go with the Simplest Thing That Could Possibly Work. From my perspective, that winds up being an OBJECT tag (and, sigh, also an EMBED tag) for Quicktime or Windows Media, plus a reference to a minimal player in the case of Flash. By comparison, my Expression-generated Silverlight example has lots of moving parts — an HTML file, a XAML file, a flock of JavaScript files, and the WMV file.

The Silverlight example could of course be simplified by coalescing the JavaScript support, but that alone won’t solve another issue of deployment convenience. It’s nice to be able to embed a screencast in any arbitrary web host. From the perspective of my WordPress.com blog, that’s an issue for all four of these approaches. WordPress is always coming up with new ways to embed video from various services, but the reason that’s necessary is that WordPress.com — quite rationally — strips out most of the advanced HTML tags and JavaScript support that you might want to include in your blog postings. In general, embedded video seems to be a game of point solutions. In order to embed video flavor X in web host Y you need a specific X+Y adapter. I understand the reasons why, but it’s frustrating.

One of those adapters, by the way, will be needed for WordPress.com and Silverlight Streaming, which is the Microsoft hosting service announced at the MIX conference earlier this year. I’ve hosted another version of my Silverlight example there. It’s the same set of files as this example, minus the HTML wrapper and the core Silverlight JavaScript code, plus an XML manifest, all packaged up in a zip file. I’m not expecting my little test to attract millions of viewers, but if I were, this hosting service would be one way to handle the load.

In terms of viewing convenience, the Silverlight example exhibits a nice property that I wasn’t expecting. When you resize the window containing the player, the player scales to fit. I’m pretty sure the embeddable Quicktime and Windows Media players can’t do that. Flash-based media players are more customizable, and can respond to container resize events, but I don’t think I’ve ever seen the technique applied to a screencast. It’s a nice idea. A screencast at 1:1 resolution is guaranteed to be legible, but will also consume a lot of screen real estate. So it’s tempting to shrink its width and height in production. But by how much? Any fixed resolution will work well for some people and not others. Resizable screencasts would be great for accessibility.

Of course you can resize any standalone player. So this issue boils down to what’s possible when the player is embedded in a web page. And as we’ve seen, embedding can be problematic. In general, we need to work toward a smoother transition between embedded and standalone viewing experiences.

The ultimate test of viewing convenience is, of course: Does it play instantly, regardless of the operating system or browser I happen to be using? Flash leads the way in that regard. Silverlight aspires to the same level of plug-in ubiquity, and with the announcement of Moonlight that aspiration seems achievable.

Ultimately a screencaster wants to be able to produce one video that works well for everyone, everywhere, for various definitions of works well. That’s a hard problem. Solutions depend on the raw capabilities of media players, it’s true. But they also depend on an ecosystem of plug-ins, browsers, encoders, operating systems, and hosting services.

The style is called paperworks because these sketchcasts are made by capturing screenshots, printing out key elements, and then filming, animating, annotating, and narrating arrangements and rearrangements of these scraps of paper. The first time you watch one, you’ll be captivated: it’s cute, it’s fresh. But is this just a gimmick? After you watch a few more, and you begin to acclimate to the style, does its effectiveness wane? Not yet, for me, because these productions have more going for them than cuteness and freshness.

One of the principles at work here is the moral equivalent of cropping and zooming in the screencast medium. When you’re trying to explain software on a conceptual level, images captured from screens can be a mixed blessing. It’s valuable to show exactly what screens look like, and exactly how actions flow within and across them. But the amount of detail that’s visible in a typical screen can often distract from the story you’re trying to tell. By cropping the screen, and/or by zooming in on the active region, you can prune away a lot of visual clutter and focus on key interactions. The paperworks style is an extreme form of cropping and zooming; it prunes and focuses very aggressively.

Another principle is sketching. According to Bill Buxton, sketching goes hand in hand with what he calls design thinking. When I asked Bill how he would have used sketching in the design of a feature like the Office ribbon, he said:

You’d start with paper prototyping — quickly hand-rendered versions, and for the pulldown menus and other objects you’d have Post-It notes. So when somebody comes with a pencil and pretends it’s their stylus and they click on something, you’ve anticipated the things they’ll do, and you stick down a Post-It note.

If that’s a helpful way to imagine software interaction in the design phase, why wouldn’t it also be a helpful way to conceptualize the software in use? The paperworks style strongly suggests that it is. These sketchcasts are great visual explanations of working software. I suspect they’d be equally useful during the design of that software.

What enables CoScripter to work in the web domain is the document object model (DOM) of which every web page is a rendering. Because JavaScript code can explore and interact with the DOM’s tree of user-interface objects, the browser can be driven semantically, by object names and properties, rather than literally, by mouse clicks and keystrokes. The literal method is workable, and there many tools that make excellent use of it. The semantic method is more reliable if available, but it isn’t always. So the literal method winds up being the common denominator, because every style of application will respond to mouse clicks and keystrokes.

There is another kind of semantic technique long supported by desktop applications that define object models, notably the Mac’s AppleScript object model and Windows’ Component Object Model. These technologies enable automation scripts to reach below the user interface of applications, and to work with their internal machinery.

Using the Word object model, for example, you can automate a mail merge. If you run this program, you’ll see Word launch, you’ll see a data document written by an invisible hand, and then you’ll see a mail merge appear. What you won’t see are the user-interface actions required to produce these effects, because this level of automation bypasses the user interface.

So let’s distinguish between two flavors of semantic automation. The mail merge script does what I’ll call engine-based semantic automation. And CoScripter does what I’ll call UI-based semantic automation.

These two flavors are useful in quite different ways. With the engine-based approach, an automation script uses the application as if it (the application) were a service. In this case you don’t want windows and dialog boxes popping up all over the place, you just want to feed inputs and harvest outputs. The engine-based approach works accurately and efficiently, but it doesn’t yield a representation of task knowledge that a normal person could use, learn from, adapt, or share.

With the UI-based approach, an automation script uses the application as if it (the script) were a human being. It sees and touches exactly what the human sees and touches. This is not the optimal way to crank out a thousand mailing labels. But the UI-based approach does yield a representation of task knowledge that a normal person could use, learn from, adapt, or share.

Shareable representations of task knowledge are incredibly useful and powerful. Screencasts are one such representation, and as many people have noticed in recent years, they can radically outperform traditional forms of documentation. But you can’t interact with a screencast or concisely describe it. You can only watch and learn and imitate. Although that’s way better than not being able to watch and learn and imitate, interaction and concise description would be better still.

CoScripter delivers that superior experience of interaction and concise description. It does so by means of UI-based semantic automation which, in turn, is enabled by the browser’s document object model.

What might enable a more comprehensive flavor of UI-based semantic automation? Noodling on this question I arrived at one possible answer: the Windows UI Automation API, which is part of .NET Framework 3.0. I’d heard of it, but hadn’t connected the dots. In this June 2005 article for the ACM’s Special Interest Group on Accessible Computing, Rob Haverty lays out the rationale for this relatively new mechanism:

Windows UI Automation unifies disparate UI Frameworks such as Avalon [Windows Presentation Foundation], Trident [the browser], and Win32 so that code can be written against one API rather than several.

The basis of this unification is a tree of automation elements that is, in effect, a generic document object model. Automation providers map various specific object models, notably those of the browser and of Windows, into the generic tree. The API provides mechanisms for searching the tree and interacting with its elements.

It’s a powerful system that is also accurately described by John Robbins as “intensively fiddly.” So in this March 2007 MSDN article, he provides and illustrates the use of a set of convenience wrappers around the raw System.Windows.Automation classes. The sample program included with that article drives Notepad through a few basic operations. Could it be extended in the direction of CoScripter, in a way that realizes UI Automation’s ambition to uniformly control Windows and web applications?

I took a crack at that, and concluded that creating even a proof-of-concept will require more time and more programming chops than I can muster. But I’d be interested to hear from anyone who’s gone further down that path. I think this is potentially a very big deal. Although I suspect most programmers see UI Automation in the context of software testing, for which it is indeed well suited, Rob Haverty’s article suggests that it was primarily motivated by the need for better assistive technologies and improved accessibility.

When Tessa Lau says that accessibility guidelines are the lifeblood of CoScripter, she’s talking about affordances for people whose cannot otherwise use the full capability of their software. But consider Rob Haverty’s definition of accessible technology:

Accessible technology enables individuals to adjust their computers to meet their visual, hearing, dexterity, cognitive, and speech needs.

I like his use of the word cognitive because in some sense we are all cognitively impaired when we try to use software. For most people, most of the time, the concept count is way too high. We don’t normally think of automation as an assistive technology. But arguably it is one. And when automation yields interactive documentation that lives in shared information spaces, it becomes a really potent assistive technology.

In case it’s not obvious, I am not claiming that Windows UI Automation can realize this vision of assistive automation across the spectrum of application types. It’s currently only available by default for Vista, and optionally for Windows XP if enhanced with the .NET Framework 3.0. It is not part of Silverlight or Moonlight, though conceivably one day it might be. And it clearly has nothing to do with Mac OS X, or Java, or Flash, or the Linux desktop.

But the idea of UI-based semantic automation is something that could apply in all these domains. A proof-of-concept CoScripter-like application-plus-service spanning two major domains — Windows desktop apps and browser-based apps running on Windows — would be a big step toward that broader vision.