Automation and accessibility in Silverlight and IE8

In this interview at MIX, Mark Rideout explains how Silverlight will use the same UIA (User Interface Automation) mechanisms that make Windows apps (and will make Linux applications) accessible by way of assistive technologies like screenreaders.

If you’re not somebody who needs that kind of assistance, you may not think this matters to you. But as I’ve pointed out in a series of essays, the flip side of accessibility is automation, and that’s something we all need.

For software developers, the automation framework provides the hooks needed to test the interactive behavior of applications.

For users, it provides the hooks needed to record, exchange, and replay software interactions. In The social scripting continuum I showed how IBM’s CoScripter enables people to share their knowledge of how to use web applications. It’s fabulous, but it’s restricted to the domain of simple web apps running in Firefox. IE, Ajax, Flash, Silverlight, and desktop apps are all out of scope.

With an automation/accessibility framework common to browsers, rich runtimes in browsers, and desktop apps, you could in theory enable a common way for people to describe and share their knowledge of how to use software across the full range of application types, for any browser, any rich runtime, and any operating system.

We’re not there yet, and we may never get there, but this Silverlight announcement points toward a future that’s worth imagining.

Update: In related news, John Resig notes that IE8 supports the W3C’s ARIA (Accessible Rich Internet Applications), which makes Ajax applications accessible to screenreaders. Here’s a brief guide for the perplexed, myself included, because this stuff is a layer cake.

Native accessibility toolkits, like MSAA (Microsoft Active Accessibility) and ATK (Linux Accessibility Toolkit), are the foundation. The Mozilla implementation of ARIA rests on this layer, as does the IE8 implementation. User Interface Automation (UIA), meanwhile, is part of the .NET Framework. It can be used to automate unmanaged apps like Word, as well as managed apps on the desktop or (now) in Silverlight. How UIA will be realized on Linux is something I don’t know, but would like to find out.

I can’t formulate a unified field theory that joins all these pieces, on various platforms, but I hope one will emerge.

Pratik patel Says:

April 6, 2008 at 6:12 am

there is a complex world of accessibility that goes along with the dizzying array of web development. ARIA is one of these up and coming standards. Proper web markup with ARIA and other standards will mean that interactive behaviors will be mapped to existing standards such as MSAA, ATSPI, and UI Automation. Microsoft’s push for binding UI automation to other standards such as ATSPI means that the information that UI automation exposes will be taken by assistive technologies such as screen readers through interfaces like ATSPI to permit commonly understandable interactive experiences for users of these assistive technologies. Looks like I’ll have to write a blog post in a few days to try and demystify this (if such a thing is possible).