How shared vocabularies tie the annotated web together

I’m fired up about the work I want to share at Domains 2017 this summer. The tagline for the conference is Indie Tech and Other Curiosities, and I plan to be one of the curiosities!

I’ve long been a cheerleader for the Domain of One’s Own movement. In Reclaiming Innovation, Jim Groom wrote about the need to “understand technologies as ‘potentiality’ (to graft a concept by Anton Chekov from a literary to a technical context).” He continued:

This is the idea that within the use of every technical tool there is more than just the consciousness of that tool, there is also the possibility to spark something beyond those predefined uses. The only real way to galvanize that potentiality is to provide the conditions of possibility — that is, a toolkit for user innovation.

My recent collaboration with Mike Caulfield on the Digital Polarization Initiative has led to the creation of just such a toolkit. It supports DigiPo in the ways described and shown here. A version of the toolkit, demoed here, will support a team of investigative journalists. Now I need to show how the toolkit enables educators, scientists, investigative reporters, students — anyone who researches and writes articles or reports or papers backed by web-based evidence — to innovate in similar ways.

In tech we tend to abuse the term innovation so let me spell out exactly what I mean: Better ways to gather, organize, reason over, and cite online evidence. Web annotation, standardized this week by the W3C, is a key enabler. The web’s infinite space of addressable URLs is now augmented by a larger infinity of segments of interest within the resources pointed to by URLs. In the textual realm, paragraphs, list items, sentences, or individual words can be reliably linked to conversations — but also applications — that live in connected annotation layers.

A web of addressable segments of interest is a necessary, but not sufficient, condition of possibility. We also need tools that enable us to gather, organize, recombine, and cite those segments. And some of those tools need to be malleable in the hands of users who can shape them for their own purposes.

When I reread Vannevar Bush’s As We May Think, to prepare for a conversation about it with Gardner Campbell and Jeremy Dean (video, Gardner’s reflections), I focused on this passage:

He has dozens of possibly pertinent books and articles in his memex. First he runs through an encyclopedia, finds an interesting but sketchy article, leaves it projected. Next, in a history, he finds another pertinent item, and ties the two together.

Nowadays that first encyclopedia article lives at one URL. The pertinent item in a history is a segment of interest within another URL-addressable resource. How do we tie them together? A crucial connector is a tag that belongs to neither resource but refers to both.

When tools control the sets of tags available for resource interconnection, they enable groups of people to make such connections reliably. That’s what the DigiPo toolkit does when it offers a list of investigation pages, drawn from the namespace of a wiki, as the set of tags that connect annotation-defined evidence to investigations. You see that happening with the DigiPo toolkit shown here, and with a variant of the toolkit shown here. In both cases the tags that bind evidence to wiki pages are controlled by software that acquires a list of wiki pages and presents the names of those pages as selectable tags.

One future direction for the toolkit leads to software that acquires lists of pages from other kinds of content management systems: WordPress, Drupal, you name it. Every CMS defines a namespace that is implicitly a list of tags that can be used to bind sets of resources to the pages served by that CMS. If you’re looking to adapt a DigiPo-like tool to your CMS, I’ll be delighted to show you how.

Such adaptation, though, requires somebody to write some code. While it’s unfashionable in some circles to say so, I don’t think everyone should learn to code. There’s a more fundamental web literacy, nicely captured by Audrey Watters here:

It’s about understanding the components of the Web and knowing how to tag and then manipulate them. By thinking and developing sets of named resources, you are a Web thinker. This isn’t about programming but rather the creation of sets of resources and the identification of components that work with those resources and combine them to create solutions.

Web annotation vastly enlarges the universe of resources that can be named. But it’s on us to name them. Tags are a principal way we do that. If our naming of resources is going to be an effective way to organize and combine them, though, we need to do it reliably and consistently. Software can enforce that consistency, but not everyone can write software. So a user innovation toolkit for the annotated web needs to empower users to enforce consistent naming without writing code.

A couple of weeks ago I built a Chrome extension that enables users to define their own lists of shared tags by recording them in an Google Doc. The demonstration video prompted this query from Jim Groom:

I just got through with a workshop here demoing Hypothes.is for a European group that may be using it to annotate online legislation for data privacy set to go live in 2018. They are teaching a course on it, and this could be one of the spaces/hubs they build the open part around. I came back to this video just now, but got the sense I could already tag from within annotations/pages, so how does the tag helper change this? Just a different way at it? Is it new functionality from previous tags? I love that you can have a Google Doc list of tags, but the video example is not making sense to me for some reason. And I wanna know :)

Here’s my response. That tag helper, now incorporated into the toolkit I’m evolving for DigiPo and other uses, makes it possible for people who don’t write code to define tag namespaces that govern their gathering, organization, recombination, and citation not only of URL-addressable resources but also of annotation-addressable segments of interest within those resources. People can “tie them together” — as Vannevar Bush imagined — in the ways their interests and workflows require.

Does that answer the question? If not, please keep asking until I do so properly. User-defined tag namespaces, though admittedly still a curiosity, are one of the best ways to make collective use of a web of addressable segments.

How annotation layers define “segments of interest” for new kinds of applications

Here are some analogies we use when talking about software:

Construction: Programs are houses built on foundations called platforms.

Ecology: Programs are organisms that depend on ecosystem services provided by platforms.

Community: Programs work together in accordance with rules defined by platforms.

Architecture: Programs are planned, designed, and built according to architectural plans.

Economics: Programs are producers and consumers of services.

Computer hardware: Programs are components that attach to a shared bus.

All are valid and may be useful in one way or another. In this essay I focus on the last because it points to an important way of understanding what web annotation can enable. My claim here is that the web’s emerging annotation layer forms a shared bus for a new wave of content-oriented applications.

A computer’s bus connects devices: disk drive, keyboard, network adapter. If we think of the web in this way, we’d say that devices (your computer, mine) and also people (you, me) attach to the bus. And that the protocol for attachment has something to do with URLs.

You can, for example, follow this link to display and interact with the set of Hypothesis annotations related to this web page. You can also paste the link’s URL into a message or a document to share the view with someone else.

That same URL can behave like an API (application programming interface) that accesses the resource named and located by the URL. A page like this one, part of the DigiPo fact-checking project, uses the link that way. It derives the Hypothes search URL from its own URL, and injects the resulting Hypothesis view into the page.

Every time we create a new wiki page at digipo.io, we mint a new URL that summons the set of Hypothesis annotations specific to that page. In principle there’s no limit to the number of such pages — and associated sets of annotations — we can add. And that’s just one of an unlimited number of sites. The web of URL-addressable resources is infinitely large.

Even so, URLs address only a small part of a larger infinity of resources: words and phrases in texts, regions within images, segments of audio and video. Web annotation enables us to address that larger infinity. The DigiPo project illustrates some of the ways in which annotation expands the notion of content as a bus shared by people and computers. But first some background on how annotation works.

The proposed standard for web annotation defines an extensible set of selectors:

Many Annotations refer to part of a resource, rather than all of it, as the Target. We call that part of the resource a Segment (of Interest). A Selector is used to describe how to determine the Segment from within the Source resource.

When the segment of interest is a selection in a textual resource, one kind of selector captures the selection and its surrounding text. Another captures the position of the selection (“starts at the 347th character, ends at the 364th”). Still another captures its location in a web page (“contained in the 2nd list item in the first list in the seventh paragraph”). For reasons of both speed and reliability, Hypothesis uses all three selectors when it attaches (“anchors”) an annotation to a selection.

When a segment of interest is a clip within a podcast or a video, a selector would capture the start and stop (“starts at 1 minute, 32 seconds, ends at 3 minutes, 12 seconds”). When it’s a region in a bitmapped image, a selector would capture the coordinates (“starts at x=12,y=53, ends at x=355,y=124”). When it’s a piece of a vector image, a selector would capture the Scalable Vector Graphics (SVG) markup defining that piece of the image.

The W3C’s model of web annotation lays a foundation for other kinds of selectors in other domains: locations in maps, nodes in Jupyter notebooks, bars and trend lines and data points in charts. But let’s stick with textual annotation for now, consider how it expands the universe of addressable resources, and explore what we can do in that universe.

Here’s a picture of what’s happening in and around the above-mentioned DigiPo page:

The author has cited a Hypothesis link that refers to a piece of evidence in another web page. The link encapsulates both the URL of that page and a set of selectors that mark the selected passage within it. When you follow the link Hypothesis takes you to the page, scrolls to the passage, and highlights it. That’s a powerful interactive experience!

Now suppose you want to review all the evidence that supports this investigation. You can do it interactively but that will require a lot of context-disrupting clicks. So another program embedded in the wiki page summarizes the cited quotes for you. It uses a variant of the Hypothesis direct link that delivers the interactive experience. The variant is a Hypothesis API call that delivers the annotation in a machine-friendly format. The summarization script collects all the Hypothesis direct links on the page, gathers the annotations, extracts the URLs and quotes, injects them into the Footnotes section of the page, and rewrites the links to point to corresponding footnotes.

To enable this magic, an app that people can use to annotate regions in web pages is necessary but not sufficient. You also need an API-accessible service that enables computers to create and retrieve annotations. Even more fundamentally, you need an open web standard that defines how apps and services work not only with atomic resources named and located by URLs, but also segments of interest within them.

What else is possible on a shared content bus where segments of interest are directly addressable both by people and computers? Here’s one idea being pondered by some folks in the world of open educational resources (OER). Suppose you’re creating an open textbook that attaches quizzes to segments within the text. The quizzes live in a database. How do you connect a quiz to a segment in your book?

Because a quiz is an URL-addressable resource, you can transclude one directly into your book near the segment to which it applies. Doing that normally means encoding the segment’s location in the book’s markup so the software that attaches the quiz can put it in the right place. That works, but it entangles two editorial tasks: writing the book, and curating the quizzes. That entanglement makes it harder to provide tools that support the tasks individually. If you can annotate segments of interest, though, you can disentangle the tasks, tool them separately, build the book more efficiently, and ensure others can more cleanly repurpose your work.

Analogies are necessary but imperfect. The notion of a shared bus, formed by an annotation layer and used by applications oriented to segments of content, may or may not resonate. I’m looking for a better analogy; suggestions welcome. But however you want to think about it, the method I’m describing here works powerfully well, I’ll continue to apply it, and I’d love to discuss ways you can too.

Componentware Revisited

I’m not a scholar, nor do I play one on TV, but when I search Google Scholar I find that I’m cited there a few times, most notably for a 1994 BYTE cover story, Componentware. The details there are at best of historical interest but the topic remains evergreen: How do we package software in ways that maximize its reusability while minimizing the level of skill required to achieve reuse?

By 1996 the web had booted up and I reprised the theme in On-Line Componentware1. That’s when it dawned on me that the websites that people “surfed” to were also software components that could be woven together to meet a variety of needs. It was my first glimpse of what we later came to know as SOA (service-oriented architecture), then RESTful APIs, and most recently microservices. Ever since then, wearing one hat or another, I’ve been elaborating the theme of that column: “A powerful capability for ad hoc distributed computing arises naturally from the architecture of the Web.” (link)

That architecture has in some ways remained the same, in other ways evolved dramatically, but its generative power continues to surprise and delight me. And I keep finding new ways to package and reuse web components.

Hypothesis has been a fascinating case study. Our web annotation system has two main components. The web service, written in Python, runs on a web server. The client, written in JavaScript, runs in your browser. Both are available for reuse in many different ways.

One way to reuse the web service is to embed views in web pages, as shown in this example from the Digital Polarization (Digipo) project:

The “Matching Annotations” widget embedded in that page is just this search result wrapped in an iframe. This is one of the most common and powerful ways to reuse web components.

The Hypothesis API affords another way to reuse its server component. The Timeline widget, embedded on that same page, works that way. It searches Hypothesis for the URLs of annotations tagged with the id of the current wiki page. Then it searches the annotations on each of those URLs for another user-assigned tag that signifies the publication date, and arranges those results chronologically. (The Timeline widget could have been written in PHP to run in the wiki server, but I’m more familiar with JavaScript so instead it’s written in JS and runs in the browser.)

The Hypothesis client can also be reused in powerful ways. Most notably, you can add the client to a website by including this simple script tag in the site’s main template:

https://hypothes.is/embed.js

Or you can use the Hypothesis proxy, https://via.hypothes.is/, to inject the client into a web page, for example: https://via.hypothes.is/https://en.wikipedia.org/wiki/Proxy_server.

When you use Hypothesis to annotate a PDF file, it relies on a separate component — Mozilla’s PDF.js — to parse the PDF and render it in the browser so the Hypothesis client can operate on it. PDF.js is available natively in Firefox, the Hypothesis Chrome extension injects it when you annotate a PDF in that browser.

Another Hypothesis component, pdf.js-hypothesis, enables a web server to serve a PDF with PDF.js and Hypothesis both active. That makes PDF annotation available in any browser. We use it in our prototype Canvas app, for example, to serve annotation-enabled PDFs in the Canvas learming management system (LMS).

Still another component enables custom rendering of annotations. You can see it in action at Science in the Classroom, a collection of research papers annotated to serve as teaching materials.

Graduate students use Hypothesis to create the annotations. But Science in the Classroom prefers to display them using its own mechanism, Learning Lens. So when the page loads, it fetches annotations using the Hypothesis API and then paints them on the page using a component that’s part of the Hypothesis client but is also available as the standalone NPM module dom-anchor-text-quote.

I am deliberately blurring the definition of web component because I think it properly encompasses many different things: a web page embedded in an iframe; an API-accessible web service; a rich client application like Hypothesis (or a simple widget like the Timeline) embedded in a web page; a standalone module like dom-anchor-text-quote; a repackaging of Hypothesis as a WordPress plugin or a Canvanas external tool.

This is a rich assortment of ingredients! But there’s one that’s notably absent. We’ve seen lots of ways to use the Hypothesis client as a component that plugs into other environments and makes annotation available there. But what if you want to plug something into the Hypothesis client? There isn’t yet a mechanism for that. The code is open source and can be modified, as Marija Katic and Martin Eve have done with Annotran, a translation tool based on Hypothesis. That’s a great example of code reuse. But it isn’t, at least to my way of thinking, an example of component reuse. Although I recognize many different species of software components, they all share one piece of common DNA: reuse without internal modification.

In an essay on what I learned while building the Canvas app, I noted two critical aspects of the healthy ecosystem that Canvas and other learning management systems inhabit:

1. Standard protocols. In the LMS world, Learning Tools Interoperability (LTI) defines those protocols.

2. Frictionless component reuse. This flows from item 1. An LTI app expects to be launched from an LMS and to run embedded in an iframe there. Again, this is the most common and powerful way to reuse web components.

The question I asked there, and tried to answer: Could an iframe embed web components within a rich web client like Hypothesis? If so that might open the way for features not yet in the Hypothesis core, like controlled tagging, that would otherwise require deep surgery on the Hypothesis client, and intimate knowledge of its JavaScript framework (Angular) and the nonstandard component model dictated by that framework.

I had already tried a couple of experiments to add controlled tagging to the Hypothesis client. In this one, the tag suggestions offered in the tag editor are bound to Hypothesis groups. In this one, tag suggestions are bound to an external web service. Both experiments entailed nontrivial alteration of the Hypothesis client.

In a third experiment, I modified the Hypothesis client in a way that could enable a family of components to plug into it. This customized client embedded an iframe in the annotation editor, and launched a user-defined web application into that iframe, passing it one parameter: the id of the annotation open in the editor. Because it was configured with the credentials of a Hypothesis user, it could work as a pluggable component that communicates with the active annotation and also with the full panoply of web resources. You could, perhaps, think of it as an annotation applet. Here’s a demo.

This approach was intriguing and might serve some useful purposes, but an iframe is an ugly and awkward construct to stick into the middle of a richly-designed web client. And this approach again fails my definition of component reuse because it requires internal modification of the client.

So as I began working to integrate Hypothesis into Digipo I was still looking for a way to control Hypothesis tags without modifying the Hypothesis client. As described in A toolkit for fact checkers, we initially used bookmarklets to do that, then began developing a Chrome extension for the Digipo project.

The Chrome extension immediately solved a couple of vexing problems. It enabled us to cleanly package a growing set of Digipo tools, by making them conveniently right-click-accessible. And it got around the security constraints that increasingly make bookmarklets untenable.

Just as importantly it enabled us to blend together a Digipo-specific set of tools, some but not all of which are Hypothesis-powered. For a Digipo fact checker, Hypothesis isn’t a primary part of the experience. It’s a supporting component that’s brought into the process as and where needed. It’s infrastructure.

The Digipo workflow relies on controlled tagging to accumulate evidence into several buckets associated with each investigation. When you’re on a page that you want to put into a bucket, you can use Digipo’s Tag this Page helper to create a Hypothesis page note with the tag for that investigation. It starts here:

That leads to a page that lists the Digipo investigations.

When you choose one, the extension uses the Hypothesis API to create a page note with the investigation’s tag.

Thanks to Hypothesis direct linking, the interaction flows seamlessly from the Digipo extension to Hypothesis. You land in the annotation editor where you can do more with Hypothesis: add comments and new tags, discuss the target document with other Hypothesis users.

But this arrangement only creates Hypothesis page notes: annotations that refer to a target document but not to a selection within that document. More powerful uses of Hypothesis flow from selections within target documents. Could a selection-based annotation begin in the Digipo extension, acquire a tag, and then flow through to Hypothesis?

Happily the answer is yes. You can see that here.

The Digipo Chrome extension presents one set of helpers when you right-click on a page with nothing selected. Some of the helpers rely on Hypothesis, others just automate parts of the Digipo workflow — for example, launching advanced Google searches. When you right-click with a selection active, the Digipo Chrome extension presents another set of helpers which, again, may or may not rely on Hypothesis. One of them, Tag this Selection, works like Tag this Page in that it uses the Hypothesis API to create an annotation that includes a controlled tag. But Tag this Selection does a bit more work. It sends not only the URL of the target document, but also a Text Quote Selector that anchors the annotation within the document. In this case, too, the interaction then flows seamlessly into Hypothesis where you can edit the newly-created annotation and perhaps discuss the selected passage.

You can see more of the interplay between the Digipo and Hypothesis extensions in this screencast. I’m pretty excited by how this is turning out. The Digipo extension is Chrome-only for now, as is the Hypothesis extension, but WebExtensions should soon enable broader coverage. There’s still a need to plug packaged behavior directly into the Hypothesis client. But much can be accomplished with an extension that cooperates with Hypothesis using its existing set of affordances. The Digipo extension is one example. I can imagine many others, and I’m expanding my definition of componentware to include them.


1 I love how our copy editor insisted on hyphenating On-Line!

A toolkit for fact checkers

Update: See this post (with screencasts!)

Mike Caulfield’s Digital Polarization Initiative (DigiPo) is a template for a course that will lead students through exercises to analyze and fact-check news stories. The pedagogical approach Mike describes here is evolving; in parallel I’ve been evolving a toolkit to help students research and organize the raw materials of the analyses they’ll be asked to produce. Annotation is a key component of the toolkit. I’ve been working to integrate it into the fact-checking workflow in ways that complement the use of other tools.

We’re not done yet but I’m pleased with the results so far. This post is an interim report to summarize what we’ve learned so far about building an annotation-powered toolkit for fact checkers.

Here’s an example of a DigiPo claim to be investigated:

EPA Plans to Allow Unlimited Dumping of Fracking Wastewater in the Gulf of Mexico (see Occupy)

I start with no a priori knowledge of EPA rules governing release of fracking wastewater, and only a passing acquaintance with the cited source, occupy.com. So the first order of business is to marshal some evidence. Hypothesis is ideal for this purpose. It creates links that encapsulate both the URL of a page containing found evidence, and the evidence itself — that is, a quote selected in the page.

There’s a dedicated page for each DigiPo investigation. It’s a wiki, so you can manually include Hypothesis links as you create them. But fact-checking is tedious work, and students will benefit from any automation that helps them focus on the analysis.

The first step was to include Hypothesis as a widget that displays annotations matching the wiki id of the page. Here’s a standalone Hypothesis view that gathers all the evidence I’ve tagged with digipo:analysis:gulf_of_frackwater. From there it was an easy next step to tweak the wiki template so it embeds that view directly in the page:

That’s really helpful, but it still requires students to acquire and use the correct tag in order to populate the widget. We can do better than that, and I’ll show how later, but here’s the next thing that happened: the timeline.

While working through a different fact-checking exercise, I found myself arranging a subset of the tagged annotations in chronological order. Again that’s a thing you can do manually; again it’s tedious; again we can automate with a bit of tag discipline and some tooling.

If you do much online research, you’ll know that it’s often hard to find the publication date of a web page. It might or might not be encoded in the URL. It might or might not appear somewhere in the text of the page. If it does there’s no predictable location or format. You can, however, ask Google to report the date on which it first indexed a page, and that turns out to be a pretty good proxy for the publication date.

So I made another bookmarklet to encapsulate that query. If you were to activate it on one of my posts it would lead you to this page:

I wrote the post on Oct 30, Google indexed it on Oct 31, that’s close enough for our purposes.

I made another bookmarklet to capture that date and add it, as a Hypothesis annotation, to the target page.

With these tools in hand, we can expand the widget to include:

  • Timeline. Annotations on the target page with a googledate tag, in chronological order.

  • Related Annotations. Annotations on the target page with a tag matching the id of the wiki page.

You can see a Related Annotations view above, here’s a Timeline:

So far, so good, but as Mike rightly pointed out, this motley assortment of bookmarklets spelled trouble. We wouldn’t want students to have to install them, and in any case bookmarklets are increasingly unlikely to work. So I transplanted them into a Chrome extension. It presents the growing set of tools in our fact-checking toolkit as right-click options on Chrome’s context menu:

It also affords a nice way to stash your Hypothesis credentials, so the tools can save annotations on your behalf:

(The DigiPo extension is Chrome-only for now, as is the Hypothesis extension, but WebExtensions should soon enable broader coverage.)

With the bookmarklets now wrapped in an extension we returned to the problem of simplifying the use of tags corresponding to wiki investigation pages. Hypothesis tags are freeform. Ideally you’d be able to configure the tag editor to present controlled lists of tags in various contexts, but that isn’t yet a feature of Hypothesis.

We can, though, use the Digipo extension to add a controlled-tagging feature to the fact-checking toolkit. The Tag this Page tool does that:

You activate the tool from a page that has evidence related to a DigiPo investigation. It reads the DigiPo page that lists investigations, captures the wiki ids of those pages. and presents them in a picklist. When you choose the investigation to which the current page applies, the current page is annotated with the investigation’s wiki id and will then show up in the Related Annotations bucket on the investigation page.

While I was doing all this I committed an ironic faux pas on Facebook and shared this article. Crazy, right? I’m literally in the middle of building tools to help people evaluate stuff like this, and yet I share without checking. Why did I not take the few seconds required to vet the source, bipartisanreport.com?

When I made myself do that I realized that what should have taken a few seconds took longer. There’s a particular Google advanced query syntax you need in this situation. You are looking for the character string “bipartisanreport.com” but you want to exclude the majority of self-referential pages. You only want to know what other sites say about this one. The query goes like this:

bipartisanreport.com -site:bipartisanreport.com

Just knowing the recipe isn’t enough. Using it needs to be second nature and, even for me, it clearly wasn’t. So now there’s Google this Site:

Which produces this:

It’s ridiculously simple and powerful. I can see at a glance that bipartisanreport.com shows up on a couple of lists of questionable sites. What does the web think about the sites that host those lists? I can repeat Google this Site to zoom in on them.

Another tool in the kit, Save Facebook Share Count, supports the sort of analysis that Mike did in a post entitled Despite Zuckerberg’s Protests, Fake News Does Better on Facebook Than Real News. Here’s Data to Prove It.

How, for example, has this questionable claim propagated on Facebook? There’s a breadcrumb trail in the annotation layer. On Dec 26 I used Save Publication Date to assign the tag googledate:2016-08-31, and on the same day I used Save Facebook Share Count to record the number of shares reported by the Facebook API. On Dec 30 I again used Save Facebook Share Count. Now we can see that the article is past its sell-by date on Facebook and never was highly influential.

Finally there’s Summarize Quotes, which arose from an experiment of Mike’s to fact-check a single article exhaustively. Here’s the article he picked, along with the annotation layer he created:

Some of the annotations contain Hypothesis direct links to related annotations. If you open this annotation in the Politico article, for example, you can follow Hypothesis links to related annotations on pages at USA Today and Science.

These transitive annotations are potent but it gets to be a lot of clicking around. So the most experimental of the tools in the kit, Summarize Quotes, produces a page like this:

This approach doesn’t feel quite right yet, but I suspect there’s something there. Using these tools you can gather a lot of evidence pretty quickly and easily. It then needs to be summarized effectively so students can reason about the evidence and produce quality analysis. The toolkit embodies a few ways to do that summarization, I’m sure more will emerge.

Marshalling the evidence

In Bird-dogging the web I responded to questions raised by Mike Caulfield about how annotation can help us fact-check the web. He’s now written a definition of the political technique, called bird-dogging, we discussed in those posts. It’s a method of recording candidates’ positions on issues, but it’s recently been mis-characterized as a way to incite violence. I’ve annotated a batch of articles that conflate bird-dogging with violence:

source: https://hypothes.is/api/search?tags=bird-dogging&user=judell

Each annotation links to Mike’s definition. Collectively they form a data set that can be used to trace the provenance of the bird-dogging = violence meme. A digital humanist could write an interesting paper on how the meme flows through a network of sources, and how it morphs along the way. But how will such evidence ever make a difference?

In Annotating the wild west of information flow I sketched an idea that weaves together annotation, a proposed standard for fact-checking called ClaimReview, and Google’s plan to use that standard to add Fact Check labels to news articles. These ingredients are necessary but not sufficient. The key missing ingredient? President Obama nailed it in his remarks at the White House Frontiers Conference: “We’re going to have to rebuild, within this wild west of information flow, some sort of curating function that people agree to.”

It can sometimes seem, in this polarized era, that we can agree on nothing. But we do agree, at least tacitly, on the science behind the technologies that sustain our civilization: energy, agriculture, medicine, construction, communication, transportation. When evidence proves that cigarettes can cause lung cancer, or that buildings in some places need to be earthquake-resistant, most of us accept it. Can we learn to honor evidence about more controversial issues? If that’s possible, annotation’s role will be to help us marshal that evidence.

Bird-dogging the web

In Annotating the wild west of information flow I responded to President Obama’s appeal for “some sort of curating function that people agree to” with a Hypothes.is thought experiment. What if an annotation tool could make claims about the veracity of statements on the web, and record those claims in a standard machine-readable format such as ClaimReview? The example I gave there: a climate scientist can verify or refute an assertion about climate change in a newspaper article.

Today Mike Caulfield writes about another kind of fact-checking. At http://www.mostdamagingwikileaks.com/ he found this claim:

“Bird-dogging is a term coined by high-level Clinton staffers who openly talk about it in the video. They boast about inciting violence at Trump rallies, paying for every protest…”

Mike knows better.

Wait, what? Bird-dogging is about violence?

I was a bird-dogger for some events in 2008 and as a blogger got to know a bunch of bird-doggers in my work as a blogger. Clinton didn’t invent the term and it has nothing to do with violence.

So he annotates the statement. But he’s not just refuting a claim, he’s explaining what bird-dogging really means: you follow candidates around and film their responses to questions about your issues.

Now Mike realizes that he can’t find an authoritative definition of that practice. So, being an expert on the subject, he writes one. Which prompts this question:

Why the heck am I going to write a comment that is only visible from this one page? There are hundreds (maybe thousands) of pages on the internet making use of the fact that there is no clear explanation of this on the web.

Mike’s annotation does two things at once. It refutes a claim about bird-dogging on one specific page. That’s the sweet spot for annotation. His note also provides a reusable definition of bird-dogging that ought to be discoverable in other contexts. Here there’s nothing special about a Hypothes.is note versus a wiki page, a blog post, or any other chunk of URL-addressable content. An authoritative definition of bird-dogging could exist in any of these forms. The challenge, as Mike suggests, is to link that definition to many relevant contexts in a discoverable way.

The mechanism I sketched in Annotating the wild west of information flow lays part of the necessary foundation. Mike could write his authoritative definition, post it to his wiki, and then use Hypothes.is to link it, by way of ClaimReview-enhanced annotations, to many misleading statements about bird-dogging around the web. So far, so good. But how will readers discover those annotations?

Suppose Mike belongs to a team of political bloggers who aggregate claims they collectively make about statements on the web. Each claim links to a Hypothes.is annotation that locates the statement in its original context and to an authoritative definition that lives at some other URL.

Suppose also that Google News regards Mike’s team as a credible source of machine-readable claims for which it will surface the Fact Check label. Now we’re getting somewhere. Annotation alone doesn’t solve Mike’s problem, but it’s a key ingredient of the solution I’m describing.

If we ever get that far, of course, we’ll run into an even more difficult problem. In an era of media fragmentation, who will ever subscribe to sources that present Fact Check labels in conflict with beliefs? But given the current state of affairs, I guess that would be a good problem to have.

Towards accessible annotation: a prototype and some questions

The most basic operation in Hypothes.is — select text on a page, click the Annotate button — is not yet accessible to a visually-impaired person who is using a screenreader. I’ve done a bit of research and come up with an approach that looks like it could work, but also raises many questions. In the spirit of keystroke conservation I want to record here what I think I know, and try to find out what I don’t.

Here’s a screencast of an initial prototype that shows, with the NVDA screen reader active on my system, the following sequence of events:

  • Load the Gettysburg address.
  • Use a key to move a selection from paragraph to paragraph.
  • Hear the selected paragraph.
  • Tab to the Annotate button and hit Enter to annotate the selected paragraph.

It’s a start. Now for some questions:

1. Is this a proper use of the aria-live attribute?

The screenreader can do all sorts of fancy navigation, like skip to the next word, sentence, or paragraph. But its notion of a selection exists within a copy of the document and (so far as I can tell) is not connected to the browsers’s copy. So the prototype uses a mechanism called ARIA Live Regions.

When you use the hotkey to advance to a paragraph and select it, a JavaScript method sets the aria-live attribute on that paragraph. That alone isn’t enough to make the screenreader announce the paragraph, it just tells it to watch the element and read it aloud if it changes. To effect a change, the JS method prepends selected: to the paragraph. Then the screenreader speaks it.

2. Can JavaScript in the browser relate the screenreader’s virtual buffer to the browser’s Document Object Model?

I suspect the answer is no, but I’d love to be proven wrong. If JS in the browser can know what the screenreader knows, the accessibility story would be much better.

3. Is this a proper use of role="link"?

The first iteration of this prototype used a document that mixed paragraphs and lists. Both were selected by the hotkey, but only the list items were read aloud by the screen reader. Then I realized that’s because list items are among the set of things — links, buttons, input boxes, checkboxes, menus — that are primary navigational elements from the screenreader’s perspective. So the version shown in the screencase adds role="link" to the visited-and-selected paragraph. That smells wrong, but what’s right?

4. Is there a polyfill for Selection.modify()?

Navigating by element — paragraph, list item, etc. — is a start. But you want to be able to select the next word (or previous) word or sentence or paragraph or table cell. And you want to be able to extend a selection to include the next word or sentence or paragraph or table cell.

A non-standard technology, Selection.modify(), is headed in that direction, and works today in Firefox and Chrome. But it’s not on a standards track. So is there a library that provides that capability in a cross-browser fashion?

It’s a hard problem. A selection within a paragraph that appears to grab a string of characters is, under the covers, quite likely to cross what are called node boundaries. Here, from an answer on StackOverflow, is a picture of what’s going on:

When a selection includes a superscript3 as shown here, it’s obvious to you what the text of the selection should be: 123456790. But that sequence of characters isn’t readily available to a JavaScript program looking at the page. It has to traverse a sequence of nodes in the browser’s Document Object Model in order to extract a linear stream of text.

It’s doable, and in fact Hypothes.is does just that when you make a selection-based annotation. That gets harder, though, when you want to move or extend that selection by words and paragraphs. So is there a polyfill for Selection.modify()? The closest I’ve found is rangy, are there others?

5. What about key bindings?

The screen reader reserves lots of keystrokes for its own use. If it’s not going to be possible to access its internal representation of the document, how will there be enough keys left over for rich navigation and selection in the browser?