Celebrating Infrastructure

When cycling in forested New England countryside I sometimes wondered about the man-made forest built along the roadside — telephone poles, power lines, transformers — and thought someone should write a book about the industrial landscape. It turns out that someone did. Brian Hayes spent many years traveling around America, researching and photographing the infrastructure that sustains our civilization. The book he produced, Infrastructure: A Field Guide to the Industrial Landscape (2005, 2nd ed. 2014), is everything I imagined it would be.

(I found the book by way of a comment that Brian Hayes left here on this blog. “Couldn’t be that Brian Hayes,” I thought. But his signature led me to his blog and thence to Infrastructure‘s home on the web. I’m passing it along here in part to remind myself that my favorite books often aren’t new or well publicized. I find them serendipitously after they’ve been around for a while.)

My father and his twin brother were students of nature in a way I’ve never been. Their knowledge of plants and animals was encyclopedic and ever-expanding. But for most of us, the natural landscape is not an expanse of unnamed and unknown objects. We recognize egrets, crows, hummingbirds, oaks, pines, and maples. The same isn’t true of the industrial landscape. More often than not, driving along some industrial corridor, we’re likely to ask the question Brian Hayes’ daughter asked him: “What’s that thing?” Infrastructure answers those questions for her, and for us.

Chapters on mining, waterworks, farming, energy production and distribution, transportation, shipping, and waste management follow a plan that “traces the flow of materials, information, and energy” throughout the web of industrial networks. We learn how industrial processes work, and how to identify the structures that house and implement them. Not all of us encounter quarries, mills, dams, refineries, or power plants on a daily basis. But water towers, roads, bridges, power lines, and data cables are as much a part of our landscape as what nature put there.

Hayes invites us to know more about the names, appearances, and workings of the industrial landscape. He also challenges us to reconsider how we feel about that landscape.

I stood by the side of a highway near Gallup, New Mexico, looking on a classic vista of the American West: red sandstone buttes, rising from a valley floor. … In front of the cliffs, and towering over them, were several cylindrical spires that I recognized as petroleum fractionating columns; off to one side was a grove of gleaming white spherical tanks. … I suspect that most viewers of this scene would consider the industrial hardware to be an intrusion, a distraction, perhaps even a desecration of the landscape.

Guilty as charged. But I’m provoked by this book to reconsider. Celebrate infrastructure, don’t hide it, Stewart Brand tweeted today. “It is civilization’s metabolism and should be its pride..

Weaving the annotated web

In 1997, at the first Perl Conference, which became OSCON the following year, my friend Andrew Schulman and I both gave talks on how the web was becoming a platform not only for publishing, but also for networked software.

Here’s the slide I remember from Andrew’s talk:

http://wwwapps.ups.com/tracking/tracking.cgi?tracknum=1Z742E220310270799

The only thing on it was a UPS tracking URL. Andrew asked us to stare at it for a while and think about what it really meant. “This is amazing!” he kept saying, over and over. “Every UPS package now has its own home page on the world wide web!”

It wasn’t just that the package had a globally unique identifier. It named a particular instance of a business process. It made the context surrounding the movement of that package through the UPS system available to UPS employees and customers who accessed it in their browsers. And it made that same context available to the Perl programs that some of us were writing to scrape web pages, extract their data, and repurpose it.

As we all soon learned, URLs can point to many kinds of resources: documents, interactive forms, audio or video.

The set of URL-addressable resources has two key properties: it’s infinite, and it’s interconnected. Twenty years later we’re still figuring out all the things you can do on a web of hyperlinked resources that are accessible at well-known global addresses and manipulated by a few simple commands like GET, POST, and DELETE.

When you’re working in an infinitely large universe it can seem ungrateful to complain that it’s too small. But there’s an even larger universe of resources populated by segments of audio and video, regions of images, and most importantly, for many of us, text in web documents: paragraphs, sentences, words, table cells.

So let’s stare in amazement at another interesting URL:

https://hyp.is/LoaMFCSJEee3aAMJuXhO-w/www.ics.uci.edu/~fielding/pubs/dissertation/software_arch.htm

Here’s what it looks like to a human who follows the link: You land on a web page, in this case Roy Fielding’s dissertation on web architecture, it scrolls to the place where I’ve highlighted a phrase, and the Hypothesis sidebar displays my annotation which includes a comment and a tag.

And here’s what that resource looks like to a computer when it fetches a variant of that link:

{
    "body": [
        {
            "type": "TextualBody",
            "value": "components: web resources\n\nconnectors: links\n\ndata: data",
            "format": "text/markdown"
        },
        {
            "type": "TextualBody",
            "purpose": "tagging",
            "value": "IAnnotate2017"
        }
    ],
    "target": [
        {
            "source": "https://www.ics.uci.edu/~fielding/pubs/dissertation/software_arch.htm",
            "selector": [
                {
                    "type": "XPathSelector",
                    "value": "/table[2]/tbody[1]/tr[1]/td[1]",
                    "refinedBy": {
                        "start": 82,
                        "end": 114,
                        "type": "TextPositionSelector"
                    }
                },
                {
                    "type": "TextPositionSelector",
                    "end": 4055,
                    "start": 4023
                },
                {
                    "exact": "components, connectors, and data",
                    "prefix": "tion of architectural elements--",
                    "type": "TextQuoteSelector",
                    "suffix": "--constrained in their relations"
                }
            ]
        }
    ],
    "created": "2017-04-18T22:48:46.756821+00:00",
    "@context": "http://www.w3.org/ns/anno.jsonld",
    "creator": "acct:judell@hypothes.is",
    "type": "Annotation",
    "id": "https://hypothes.is/a/LoaMFCSJEee3aAMJuXhO-w",
    "modified": "2017-04-18T23:03:54.502857+00:00"
}

The URL, which we call a direct link, isn’t itself a standard way to address a selection of text, it’s just a link that points to a web resource. But the resource it points to, which describes the highlighted text and its coordinates within the document, is — since February of this year — a W3C standard. The way I like to think about it is that the highlighted phrase — and every possible highlighted phrase — has its own home page on the web, a place where humans and machines can jointly focus attention.

If we think of the web we’ve known as a kind of fabric woven together with links, the annotated web increases the thread count of that fabric. When we weave with pieces of URL-addressable documents, we can have conversations about those pieces, we can retrieve them, we can tag them, and we can interconnect them.

Working with our panelists and others, it’s been my privilege to build a series of annotation-powered apps that begin to show what’s possible when every piece of the web is addressable in this way.

I’ll show you some examples, then invite my collaborators — Beth Ruedi from AAAS, Mike Caulfield from Washington State University Vancouver, Anita Bandrowski from SciCrunch, and Maryann Martone from UCSD and Hypothesis — to talk about what these apps are doing for them now, and where we hope to take them next.

Science in the Classroom

First up is a AAAS project called Science in the Classroom, a collection of research papers from the Science family of journals that are annotated — by graduate students — so teachers can help younger students understand the methods and outcomes of scientific research.

Here’s one of those annotated papers. A widget called the Learning Lens toggles layers of annotation and off.

Here I’ve selected the Glossary layer, and I’ve clicked on the word “distal” to reveal the annotation attached to it.

Now lets look behind the scenes:

Hypothesis was used to annotate the word “distal”. But Learning Lens predated the use of Hypothesis, and the Science in the Classroom team wanted to keep using Learning Lens to display annotations. What they didn’t want was the workflow behind it, which required manual insertion of annotations into HTML pages.

Here’s the solution we came up with. Use Hypothesis to create annotations, then use some JavaScript in Science in the Classroom pages to retrieve Hypothesis annotations and write them into the pages, using the same format that had been applied manually. The preexisting and unmodified Learning Lens JavaScript can then do what it does: pick up the annotations, assign color-coded highlights based on tags, and show the annotations when you click on the highlights.

What made this possible was a JavaScript library that helps with the heavy lifting required to attach an annotation to its intended target in the document.

That library is part of the Hypothesis client, but it’s also available as a standalone module that can be used for other purposes. It’s a nice example of how open source components can enable an ecosystem of interoperable annotation services.

DigiPo / EIC

Next up is a toolkit for student fact-checkers and investigative journalists. You’ve already heard from Mike Caulfield about the Digital Polarization Project, or DigiPo, and from Stefan Candea about the European Investigative Collaborations network. Let’s look at how we’ve woven annotation into their investigative workflows.

These investigations are both written and displayed in a wiki. This is a DigiPo example:

I did the investigation of this claim myself, to test out the process we were developing. It required me to gather a whole lot of supporting evidence before I could begin to analyze the claim. I used a Hypothesis tag to collect annotations related to the investigation, and you can see them in this Hypothesis view:

I can be very disciplined about using tags this way, but it’s a lot to ask of students, or really almost anyone. So we created a tool that knows about the set of investigations underway in the wiki, and offers the names of those pages as selectable tags.

Here I’ve selected a piece of evidence for that investigation. I’m going to annotate it, not by using Hypothesis directly, but instead by using a function in a separate DigiPo extension. That function uses the core anchoring libraries to create annotations in the same way the Hypothesis client does.

But it leads the user through an interstitial page that asks which investigation the annotation belongs to, and assigns a corresponding tag to the annotation it creates.

Back in the wiki, the page embeds the same Hypothesis view we’ve already seen, as a Related Annotations widget pinned to that particular tag:

I had so much raw material for this article that I needed some help organizing it. So I added a Timeline widget that gathers a subset of the source annotations that are tagged with dates.

To put something onto the timeline, you select a date on a page.

Then you create an annotation with a tag corresponding to the date.

Here’s what the annotation looks like in Hypothesis.

Over in the wiki, our JavaScript finds annotations that have these date tags and arranges them on the Timeline.

Publication dates aren’t always evident on web pages, sometimes you have to do some digging to find them. When you do find one, and annotate a page with it, you’ve done more than populate the Timeline in a DigiPo page. That date annotation is now attached to the source page for anyone to discover, using Hypothesis or any other annotation-aware viewer. And that’s true for all the annotations created by DigiPo investigators. They’re woven into DigiPo pages, but they’re also available for separate reuse and aggregation.

The last and most popular annotation-related feature we added to the toolkit is called Footnotes. Once you’ve gathered your raw material into the Related Annotations bucket, and maybe organized some of it onto the Timeline, you’ll want to weave the most pertinent references into the analysis you’re writing.

To do that, you find the annotation you gathered and use Copy to clipboard to capture the direct link.

Then you wrap that link around some text in the article:

When you refresh the page, here’s what you get. The direct link does what a direct link does: it takes you to the page, scrolls you to the annotation in context. But it can take a while to review a bunch of sources that way.

So the page’s JavaScript also creates a link that points down into the Footnotes section. And there, as Ted Nelson would say, and as Nate Angell for some reason hates hearing me say, the footnote is “transcluded” into the page so all the supporting context is right there.

One final point about this toolkit. Students don’t like the writing tools available in wikis, and for good reason, they’re pretty rough around the edges. So we want to enable them to write in Google Docs. We also want them to footnote their articles using direct links because that’s the best way to do it. So here’s a solution we’re trying. From the wiki you’ll launch into Google Docs where you can do your writing in a much more robust editor that makes it really easy to include images and charts. And if you use direct links in that Google Doc, they’ll still show up as Footnotes.

We’re not yet sure this will pan out, but my colleague Maryann Martone, who uses Hypothesis to gather raw material for her scientific papers, and who writes them in Google Docs, would love to be able to flow annotations through her writing tool and into published footnotes.

SciBot

Maryann is the perfect segue to our next example. Along with Anita Bandrowski, she’s working to increase the thread count in the fabric of scientific literature. When neuroscientists write up the methods used in their experiments, the ingredients often include highly specific antibodies. These have colloquial names, and even vendor catalog numbers, but they still lacked unique identifiers. So the Neuroscience Information Framework, NIF for short, has defined a namespace called RRID (research resource identifier), built a registry for RRIDs, and convinced a growing number of authors to mention RRIDs in their papers.

Here’s an article with RRIDs in it. They’re written directly into the text because the text is the scientific record, it’s the only artifact that’s guaranteed to be preserved. So if you’re talking about a goat polyclonal antibody, you look it up in the registery, capture its ID, and write it directly into the text. And if it’s not in the registry, please add it, you’ll make Anita very happy if you do!

The first phase of a project we call SciBot was about validating those RRIDs. They’re just freetext, after all, typed in by authors. Were the identifiers spelled correctly? Did they point to actual registry entries? To find out we built a tool that automatically annotates occurrences of RRIDs.

In this example, Anita is about to click on the SciBot tool, which launches from a bookmarklet, and sends the text of the paper to a backend service. It scans the text for RRIDs, looks up each one in the registry, and uses the Hypothesis API to create an annotation — bound to the occurrence in the text — that reports the results of the registry lookup.

Here the Hypothesis realtime API is showing that SciBot has created three annotations on this page.

And here are those three annotations, anchored to their occurrences in the page, with registry entries displayed in the sidebar.

SciBot curators review these annotations and use tags to mark which are valid. When some aren’t, and need attention, the highlight focuses that attention on a specific occurrence.

This hybrid of automatic entity recognition and interactive human curation is really powerful. Here’s an example where an antibody doesn’t have an RRID but should.

Every automatic workflow needs human exception handling and error correction. Here the curator has marked an RRID that wasn’t written into the literature, but now is present in the annotation layer.

These corrections are now available to train a next-gen entity recognizer. Iterating through that kind of feedback loop will be a powerful way to mine the implicit data that’s woven into the scientific literature and make it explicit.

Here’s the Hypothesis dashboard for one of the SciBot curators. The tag cloud gives you a pretty good sense of how this process has been unfolding so far.

Publishers have begun to link RRIDs to the NIF registry. Here’s an example at PubMed.

If you follow the ZIRC_ZL1 link to the registry, you’ll find a list of other papers whose authors used the same experimental ingredient, which happens to be a particular strain of zebrafish.

This is the main purpose of RRIDs. If that zebrafish is part of my experiment, I want to find who else has used it, and what their experiences have been — not just what they reported in their papers, but ideally also what’s been discussed in the annotation layer.

Of course I can visit those papers, and search within them for ZIRC_ZLI, but with annotations we can do better. In DigiPo we saw how footnoted quotes from source documents can transclude into an article. Publishers could do the same here.

Or we could do this. It’s a little tool that offers to look up an RRID selected in text.

It just links to an instance of the Hypothesis dashboard that’s pinned to the tag for that RRID.

Those search results offer direct links that take you to each occurrence in context.

Claim Chart

Finally, and to bring us full circle, I recently reconnected with Andrew Schulman who works nowadays as a software patent attorney. There’s a tool of his trade called a claim chart. It’s a two-column table. In column one you list claims that a patent is making, which are selections of text from the claims section of the patent. And in column two you assemble bits of evidence, gathered from other sources, that bear on specific claims. Those bits of evidence are selections of text in other documents. It’s tedious to build a claim chart, it involves a lot of copying and pasting, and the evidence you gather is typically trapped in whatever document you create.

Andrew wondered if an annotation-powered app could help build claim charts, and also make the supporting evidence web-addressable for all the reasons we’ve discussed. If I’ve learned anything about annotation, it’s that when somebody asks “Can you do X with annotation?” the answer should always be: “I don’t know, should be possible, let’s find out.”

So, here’s an annotation-powered claim chart.

The daggers at top left in each cell are direct links. The ones in the first column go to patent claims in context.

The ones in the second column go to related statements in other documents.

And here’s how the columns are related. When you annotate a claim, you use a toolkit function called Add Selection as Claim.

Your selection here identifies the target document (that is, the patent), the claim chart you’re building (here, it’s a wiki page called andrew_test), and the claim itself (for example, claim 1).

Once you’ve identified the claims in this way, they’re available as targets of annotations in other documents. From a selection in another document, you use Add Selection as Claim-Related.

Here you see all the claims you’ve marked up, so it’s easy to connect the two statements.

The last time I read Vannevar Bush’s famous essay As We May Think, this was the quote that stuck with me.

When statements in documents become addressable resources on the web, we can weave them together in the way Vannevar Bush imagined.

Do Repeat Yourself, With Variations

Don’t Repeat Yourself (DRY) is a touchstone principle of software development. It’s often understood to inveigh against duplication of code. Copying a half-dozen lines from one program to another is a bad idea, DRY says, because if you change your mind about how that code works, you’ll have to revise it in several places. Better to convert those lines of code into a function that you write once and reuse.

More broadly, the DRY principle asserts:

Every piece of knowledge must have a single, unambiguous, authoritative representation within a system.

Code and data are two kinds of knowledge that ought to be represented canonically, and repeated — if at all — only by mechanical derivation, never by variation.

I often violate the DRY principle by indulging in CopyAndPasteProgramming. In my defense I point to another principle, CodeHarvesting, which defends duplication as a necessary stepping stone.

Letting a duplication of logic live for now, in order to see how to best universalize it at some later point.

For me, at least, that’s what tends to work best. A common theme doesn’t emerge until I’ve seen — and ideally others have seen and reacted to — several variations on that theme. This kind of duplication — deferred universalization — is beneficial, right?

Here’s another kind. In the JavaScript world the dominant engine of reuse is the Node Package Manager (NPM). When I first started using it a few years ago, I was shocked at the amount of duplication it entails. When you install an NPM package, the modules it depends on are copied into a subdirectory. If those modules depend on others, they are copied into yet deeper subdirectories. For even a simple JavaScript program you can end up with a forest of thousands of files.

A similar thing happens in the Python world. It’s a best practice, nowadays, to use a tool called virtualenv to create, for each Python program you run, an isolated environment with the particular Python interpreter and set of modules needed by that particular program. In practice that means, again, copying lots of files.

Arguably these duplications don’t violate DRY because they are mechanical copies that won’t vary from their originals. But they can! And here too I am prone to indulge in local variation to explore possibilities that might or might not warrant generalization.

While pondering the vices and virtues of duplicative software construction I reread Metamagical Themas, the compendium of Douglas Hofstadter’s columns in Scientific American. (The title is an anagram of Mathematical Games, the column he inherited from Martin Gardner.) In Variations on a Theme as the Crux of Creativity he states the case as plainly as anywhere. At the core of creative thought are “slippery” concepts that we develop in a virtuous cycle of innovation:

Once you have decided to try out a new way of viewing a phenomenon, you can let that view suggest a set of knobs to vary. The act of varying them will lead you down new pathways, generating new images ripe for perception in their own right.

This sets up a closed loop:

– fresh situations get unconsciously framed in terms of familiar concept;

– those familiar concepts come equipped with standard knobs to twiddle;

– twiddling those knobs carries you into fresh new conceptual territory.

We need to get DRY eventually in order to maintain stable systems. But the countervailing state needn’t be WET (“write everything twice”, “we enjoy typing” or “waste everyone’s time”). Instead I propose DRYWV: Do Repeat Yourself, With Variations.

Every piece of knowledge should have a single, unambiguous, authoritative representation within a system. But how do we arrive at such knowledge? I think we have to DRYWV our way there.

Dwelling in the zone of evidence

I’ve written plenty about the software layer that adapts the Hypothesis annotator to the needs of someone who gathers, organizes, analyzes, and then writes about evidence found online. Students in courses based on Mike Caulfield’s Digital Polarization template will, I hope, find that this software streamlines the grunt work required to find and cite the evidence that supports evaluation of a claim like this one:

Claim: The North Carolina Republican Party sent out a press release boasting about how its efforts drove down African-American turnout in the 2016 US presidential election.

That’s a lightly-edited version of something I read in the New Yorker and can send you to directly:

As we were fleshing out how a DigiPo course would work, I wrote an analysis of that claim. The investigation took me all the way back to the 1965 Voting Rights Act. Then it led to the 2013 Supreme Court decision — in Shelby vs Holder — to dilute the “strong medicine” Congress had deemed necessary “to address entrenched racial discrimination in voting.” Then to a series of legal contests as North Carolina began adjusting its voting laws. Then to the election-year controversies about voter suppression. And finally to the press release that the North Carolina GOP sent the day before the election, and the reactions to it.

Many claims don’t require this kind of deep dive. As Mike writes today, core strategies — look for fact-checking prior art, go upstream to the source, read laterally — can resolve some claims quickly.

But some claims do require a deep dive. In those cases I want students to immerse themselves in that process of discovery. I want them to suspend judgement about the claim and focus initially on marshalling evidence, evaluating sources, and laying a foundation for analyis. It’s hard work that the DigiPo toolkit can make easier, maybe even fun. That’s crucial because the longer you can comfortably dwell in that zone of evidence-gathering and suspended judgement, the stronger your critical thinking will become.

When I first read Toobin’s claim my internal narrative was: “Boasted about voter suppression? Of course those neanderthals did!” Then I entered the zone and spent many hours there. Voter suppression wasn’t a topic I’d spent much time reading about, so I learned a lot. When I returned to the claim I arrived at an interesting judgement. Yes there was voter suppression, and it was in some ways more draconian than I had thought. But had the North Carolina GOP actually boasted (Mother Jones: bragged, Salon: celebrated) the lower African-American turnout? I concluded it had not. It had reported reduced early voting, but not explicitly claimed that was a successful outcome of voter suppression.

So we rated the claim as Mixed — that is, partly true, partly false. A next step for this investigation would be to break the claim into more granular parts. (Software developers would call that “refactoring” the claim.) So:

In a press release on November 7, 2016, the North Carolina GOP reported lower African-American early voting.

That’s easy to check. True.

Here’s another:

In its 11/7/2016 press release the North Carolina GOP boasted about the success of its voter suppression efforts.

Also easy to check: False.

What about this?

In the wake of Shelby vs Holder, the North Carolina GOP pushed legislation that discriminates against African-American voters.

You need to gather and organize a lot of source material in order to even begin to evaluate that claim. My fondest hope for DigiPo is that students inclined to judge the claim, one way or the another, will delay that judgement long enough to gather evidence that all can agree is valid. That, I believe, would be a fantastic educational outcome.

How shared vocabularies tie the annotated web together

I’m fired up about the work I want to share at Domains 2017 this summer. The tagline for the conference is Indie Tech and Other Curiosities, and I plan to be one of the curiosities!

I’ve long been a cheerleader for the Domain of One’s Own movement. In Reclaiming Innovation, Jim Groom wrote about the need to “understand technologies as ‘potentiality’ (to graft a concept by Anton Chekov from a literary to a technical context).” He continued:

This is the idea that within the use of every technical tool there is more than just the consciousness of that tool, there is also the possibility to spark something beyond those predefined uses. The only real way to galvanize that potentiality is to provide the conditions of possibility — that is, a toolkit for user innovation.

My recent collaboration with Mike Caulfield on the Digital Polarization Initiative has led to the creation of just such a toolkit. It supports DigiPo in the ways described and shown here. A version of the toolkit, demoed here, will support a team of investigative journalists. Now I need to show how the toolkit enables educators, scientists, investigative reporters, students — anyone who researches and writes articles or reports or papers backed by web-based evidence — to innovate in similar ways.

In tech we tend to abuse the term innovation so let me spell out exactly what I mean: Better ways to gather, organize, reason over, and cite online evidence. Web annotation, standardized this week by the W3C, is a key enabler. The web’s infinite space of addressable URLs is now augmented by a larger infinity of segments of interest within the resources pointed to by URLs. In the textual realm, paragraphs, list items, sentences, or individual words can be reliably linked to conversations — but also applications — that live in connected annotation layers.

A web of addressable segments of interest is a necessary, but not sufficient, condition of possibility. We also need tools that enable us to gather, organize, recombine, and cite those segments. And some of those tools need to be malleable in the hands of users who can shape them for their own purposes.

When I reread Vannevar Bush’s As We May Think, to prepare for a conversation about it with Gardner Campbell and Jeremy Dean (video, Gardner’s reflections), I focused on this passage:

He has dozens of possibly pertinent books and articles in his memex. First he runs through an encyclopedia, finds an interesting but sketchy article, leaves it projected. Next, in a history, he finds another pertinent item, and ties the two together.

Nowadays that first encyclopedia article lives at one URL. The pertinent item in a history is a segment of interest within another URL-addressable resource. How do we tie them together? A crucial connector is a tag that belongs to neither resource but refers to both.

When tools control the sets of tags available for resource interconnection, they enable groups of people to make such connections reliably. That’s what the DigiPo toolkit does when it offers a list of investigation pages, drawn from the namespace of a wiki, as the set of tags that connect annotation-defined evidence to investigations. You see that happening with the DigiPo toolkit shown here, and with a variant of the toolkit shown here. In both cases the tags that bind evidence to wiki pages are controlled by software that acquires a list of wiki pages and presents the names of those pages as selectable tags.

One future direction for the toolkit leads to software that acquires lists of pages from other kinds of content management systems: WordPress, Drupal, you name it. Every CMS defines a namespace that is implicitly a list of tags that can be used to bind sets of resources to the pages served by that CMS. If you’re looking to adapt a DigiPo-like tool to your CMS, I’ll be delighted to show you how.

Such adaptation, though, requires somebody to write some code. While it’s unfashionable in some circles to say so, I don’t think everyone should learn to code. There’s a more fundamental web literacy, nicely captured by Audrey Watters here:

It’s about understanding the components of the Web and knowing how to tag and then manipulate them. By thinking and developing sets of named resources, you are a Web thinker. This isn’t about programming but rather the creation of sets of resources and the identification of components that work with those resources and combine them to create solutions.

Web annotation vastly enlarges the universe of resources that can be named. But it’s on us to name them. Tags are a principal way we do that. If our naming of resources is going to be an effective way to organize and combine them, though, we need to do it reliably and consistently. Software can enforce that consistency, but not everyone can write software. So a user innovation toolkit for the annotated web needs to empower users to enforce consistent naming without writing code.

A couple of weeks ago I built a Chrome extension that enables users to define their own lists of shared tags by recording them in an Google Doc. The demonstration video prompted this query from Jim Groom:

I just got through with a workshop here demoing Hypothes.is for a European group that may be using it to annotate online legislation for data privacy set to go live in 2018. They are teaching a course on it, and this could be one of the spaces/hubs they build the open part around. I came back to this video just now, but got the sense I could already tag from within annotations/pages, so how does the tag helper change this? Just a different way at it? Is it new functionality from previous tags? I love that you can have a Google Doc list of tags, but the video example is not making sense to me for some reason. And I wanna know :)

Here’s my response. That tag helper, now incorporated into the toolkit I’m evolving for DigiPo and other uses, makes it possible for people who don’t write code to define tag namespaces that govern their gathering, organization, recombination, and citation not only of URL-addressable resources but also of annotation-addressable segments of interest within those resources. People can “tie them together” — as Vannevar Bush imagined — in the ways their interests and workflows require.

Does that answer the question? If not, please keep asking until I do so properly. User-defined tag namespaces, though admittedly still a curiosity, are one of the best ways to make collective use of a web of addressable segments.

How annotation layers define “segments of interest” for new kinds of applications

Here are some analogies we use when talking about software:

Construction: Programs are houses built on foundations called platforms.

Ecology: Programs are organisms that depend on ecosystem services provided by platforms.

Community: Programs work together in accordance with rules defined by platforms.

Architecture: Programs are planned, designed, and built according to architectural plans.

Economics: Programs are producers and consumers of services.

Computer hardware: Programs are components that attach to a shared bus.

All are valid and may be useful in one way or another. In this essay I focus on the last because it points to an important way of understanding what web annotation can enable. My claim here is that the web’s emerging annotation layer forms a shared bus for a new wave of content-oriented applications.

A computer’s bus connects devices: disk drive, keyboard, network adapter. If we think of the web in this way, we’d say that devices (your computer, mine) and also people (you, me) attach to the bus. And that the protocol for attachment has something to do with URLs.

You can, for example, follow this link to display and interact with the set of Hypothesis annotations related to this web page. You can also paste the link’s URL into a message or a document to share the view with someone else.

That same URL can behave like an API (application programming interface) that accesses the resource named and located by the URL. A page like this one, part of the DigiPo fact-checking project, uses the link that way. It derives the Hypothes search URL from its own URL, and injects the resulting Hypothesis view into the page.

Every time we create a new wiki page at digipo.io, we mint a new URL that summons the set of Hypothesis annotations specific to that page. In principle there’s no limit to the number of such pages — and associated sets of annotations — we can add. And that’s just one of an unlimited number of sites. The web of URL-addressable resources is infinitely large.

Even so, URLs address only a small part of a larger infinity of resources: words and phrases in texts, regions within images, segments of audio and video. Web annotation enables us to address that larger infinity. The DigiPo project illustrates some of the ways in which annotation expands the notion of content as a bus shared by people and computers. But first some background on how annotation works.

The proposed standard for web annotation defines an extensible set of selectors:

Many Annotations refer to part of a resource, rather than all of it, as the Target. We call that part of the resource a Segment (of Interest). A Selector is used to describe how to determine the Segment from within the Source resource.

When the segment of interest is a selection in a textual resource, one kind of selector captures the selection and its surrounding text. Another captures the position of the selection (“starts at the 347th character, ends at the 364th”). Still another captures its location in a web page (“contained in the 2nd list item in the first list in the seventh paragraph”). For reasons of both speed and reliability, Hypothesis uses all three selectors when it attaches (“anchors”) an annotation to a selection.

When a segment of interest is a clip within a podcast or a video, a selector would capture the start and stop (“starts at 1 minute, 32 seconds, ends at 3 minutes, 12 seconds”). When it’s a region in a bitmapped image, a selector would capture the coordinates (“starts at x=12,y=53, ends at x=355,y=124”). When it’s a piece of a vector image, a selector would capture the Scalable Vector Graphics (SVG) markup defining that piece of the image.

The W3C’s model of web annotation lays a foundation for other kinds of selectors in other domains: locations in maps, nodes in Jupyter notebooks, bars and trend lines and data points in charts. But let’s stick with textual annotation for now, consider how it expands the universe of addressable resources, and explore what we can do in that universe.

Here’s a picture of what’s happening in and around the above-mentioned DigiPo page:

The author has cited a Hypothesis link that refers to a piece of evidence in another web page. The link encapsulates both the URL of that page and a set of selectors that mark the selected passage within it. When you follow the link Hypothesis takes you to the page, scrolls to the passage, and highlights it. That’s a powerful interactive experience!

Now suppose you want to review all the evidence that supports this investigation. You can do it interactively but that will require a lot of context-disrupting clicks. So another program embedded in the wiki page summarizes the cited quotes for you. It uses a variant of the Hypothesis direct link that delivers the interactive experience. The variant is a Hypothesis API call that delivers the annotation in a machine-friendly format. The summarization script collects all the Hypothesis direct links on the page, gathers the annotations, extracts the URLs and quotes, injects them into the Footnotes section of the page, and rewrites the links to point to corresponding footnotes.

To enable this magic, an app that people can use to annotate regions in web pages is necessary but not sufficient. You also need an API-accessible service that enables computers to create and retrieve annotations. Even more fundamentally, you need an open web standard that defines how apps and services work not only with atomic resources named and located by URLs, but also segments of interest within them.

What else is possible on a shared content bus where segments of interest are directly addressable both by people and computers? Here’s one idea being pondered by some folks in the world of open educational resources (OER). Suppose you’re creating an open textbook that attaches quizzes to segments within the text. The quizzes live in a database. How do you connect a quiz to a segment in your book?

Because a quiz is an URL-addressable resource, you can transclude one directly into your book near the segment to which it applies. Doing that normally means encoding the segment’s location in the book’s markup so the software that attaches the quiz can put it in the right place. That works, but it entangles two editorial tasks: writing the book, and curating the quizzes. That entanglement makes it harder to provide tools that support the tasks individually. If you can annotate segments of interest, though, you can disentangle the tasks, tool them separately, build the book more efficiently, and ensure others can more cleanly repurpose your work.

Analogies are necessary but imperfect. The notion of a shared bus, formed by an annotation layer and used by applications oriented to segments of content, may or may not resonate. I’m looking for a better analogy; suggestions welcome. But however you want to think about it, the method I’m describing here works powerfully well, I’ll continue to apply it, and I’d love to discuss ways you can too.

Componentware Revisited

I’m not a scholar, nor do I play one on TV, but when I search Google Scholar I find that I’m cited there a few times, most notably for a 1994 BYTE cover story, Componentware. The details there are at best of historical interest but the topic remains evergreen: How do we package software in ways that maximize its reusability while minimizing the level of skill required to achieve reuse?

By 1996 the web had booted up and I reprised the theme in On-Line Componentware1. That’s when it dawned on me that the websites that people “surfed” to were also software components that could be woven together to meet a variety of needs. It was my first glimpse of what we later came to know as SOA (service-oriented architecture), then RESTful APIs, and most recently microservices. Ever since then, wearing one hat or another, I’ve been elaborating the theme of that column: “A powerful capability for ad hoc distributed computing arises naturally from the architecture of the Web.” (link)

That architecture has in some ways remained the same, in other ways evolved dramatically, but its generative power continues to surprise and delight me. And I keep finding new ways to package and reuse web components.

Hypothesis has been a fascinating case study. Our web annotation system has two main components. The web service, written in Python, runs on a web server. The client, written in JavaScript, runs in your browser. Both are available for reuse in many different ways.

One way to reuse the web service is to embed views in web pages, as shown in this example from the Digital Polarization (Digipo) project:

The “Matching Annotations” widget embedded in that page is just this search result wrapped in an iframe. This is one of the most common and powerful ways to reuse web components.

The Hypothesis API affords another way to reuse its server component. The Timeline widget, embedded on that same page, works that way. It searches Hypothesis for the URLs of annotations tagged with the id of the current wiki page. Then it searches the annotations on each of those URLs for another user-assigned tag that signifies the publication date, and arranges those results chronologically. (The Timeline widget could have been written in PHP to run in the wiki server, but I’m more familiar with JavaScript so instead it’s written in JS and runs in the browser.)

The Hypothesis client can also be reused in powerful ways. Most notably, you can add the client to a website by including this simple script tag in the site’s main template:

https://hypothes.is/embed.js

Or you can use the Hypothesis proxy, https://via.hypothes.is/, to inject the client into a web page, for example: https://via.hypothes.is/https://en.wikipedia.org/wiki/Proxy_server.

When you use Hypothesis to annotate a PDF file, it relies on a separate component — Mozilla’s PDF.js — to parse the PDF and render it in the browser so the Hypothesis client can operate on it. PDF.js is available natively in Firefox, the Hypothesis Chrome extension injects it when you annotate a PDF in that browser.

Another Hypothesis component, pdf.js-hypothesis, enables a web server to serve a PDF with PDF.js and Hypothesis both active. That makes PDF annotation available in any browser. We use it in our prototype Canvas app, for example, to serve annotation-enabled PDFs in the Canvas learming management system (LMS).

Still another component enables custom rendering of annotations. You can see it in action at Science in the Classroom, a collection of research papers annotated to serve as teaching materials.

Graduate students use Hypothesis to create the annotations. But Science in the Classroom prefers to display them using its own mechanism, Learning Lens. So when the page loads, it fetches annotations using the Hypothesis API and then paints them on the page using a component that’s part of the Hypothesis client but is also available as the standalone NPM module dom-anchor-text-quote.

I am deliberately blurring the definition of web component because I think it properly encompasses many different things: a web page embedded in an iframe; an API-accessible web service; a rich client application like Hypothesis (or a simple widget like the Timeline) embedded in a web page; a standalone module like dom-anchor-text-quote; a repackaging of Hypothesis as a WordPress plugin or a Canvanas external tool.

This is a rich assortment of ingredients! But there’s one that’s notably absent. We’ve seen lots of ways to use the Hypothesis client as a component that plugs into other environments and makes annotation available there. But what if you want to plug something into the Hypothesis client? There isn’t yet a mechanism for that. The code is open source and can be modified, as Marija Katic and Martin Eve have done with Annotran, a translation tool based on Hypothesis. That’s a great example of code reuse. But it isn’t, at least to my way of thinking, an example of component reuse. Although I recognize many different species of software components, they all share one piece of common DNA: reuse without internal modification.

In an essay on what I learned while building the Canvas app, I noted two critical aspects of the healthy ecosystem that Canvas and other learning management systems inhabit:

1. Standard protocols. In the LMS world, Learning Tools Interoperability (LTI) defines those protocols.

2. Frictionless component reuse. This flows from item 1. An LTI app expects to be launched from an LMS and to run embedded in an iframe there. Again, this is the most common and powerful way to reuse web components.

The question I asked there, and tried to answer: Could an iframe embed web components within a rich web client like Hypothesis? If so that might open the way for features not yet in the Hypothesis core, like controlled tagging, that would otherwise require deep surgery on the Hypothesis client, and intimate knowledge of its JavaScript framework (Angular) and the nonstandard component model dictated by that framework.

I had already tried a couple of experiments to add controlled tagging to the Hypothesis client. In this one, the tag suggestions offered in the tag editor are bound to Hypothesis groups. In this one, tag suggestions are bound to an external web service. Both experiments entailed nontrivial alteration of the Hypothesis client.

In a third experiment, I modified the Hypothesis client in a way that could enable a family of components to plug into it. This customized client embedded an iframe in the annotation editor, and launched a user-defined web application into that iframe, passing it one parameter: the id of the annotation open in the editor. Because it was configured with the credentials of a Hypothesis user, it could work as a pluggable component that communicates with the active annotation and also with the full panoply of web resources. You could, perhaps, think of it as an annotation applet. Here’s a demo.

This approach was intriguing and might serve some useful purposes, but an iframe is an ugly and awkward construct to stick into the middle of a richly-designed web client. And this approach again fails my definition of component reuse because it requires internal modification of the client.

So as I began working to integrate Hypothesis into Digipo I was still looking for a way to control Hypothesis tags without modifying the Hypothesis client. As described in A toolkit for fact checkers, we initially used bookmarklets to do that, then began developing a Chrome extension for the Digipo project.

The Chrome extension immediately solved a couple of vexing problems. It enabled us to cleanly package a growing set of Digipo tools, by making them conveniently right-click-accessible. And it got around the security constraints that increasingly make bookmarklets untenable.

Just as importantly it enabled us to blend together a Digipo-specific set of tools, some but not all of which are Hypothesis-powered. For a Digipo fact checker, Hypothesis isn’t a primary part of the experience. It’s a supporting component that’s brought into the process as and where needed. It’s infrastructure.

The Digipo workflow relies on controlled tagging to accumulate evidence into several buckets associated with each investigation. When you’re on a page that you want to put into a bucket, you can use Digipo’s Tag this Page helper to create a Hypothesis page note with the tag for that investigation. It starts here:

That leads to a page that lists the Digipo investigations.

When you choose one, the extension uses the Hypothesis API to create a page note with the investigation’s tag.

Thanks to Hypothesis direct linking, the interaction flows seamlessly from the Digipo extension to Hypothesis. You land in the annotation editor where you can do more with Hypothesis: add comments and new tags, discuss the target document with other Hypothesis users.

But this arrangement only creates Hypothesis page notes: annotations that refer to a target document but not to a selection within that document. More powerful uses of Hypothesis flow from selections within target documents. Could a selection-based annotation begin in the Digipo extension, acquire a tag, and then flow through to Hypothesis?

Happily the answer is yes. You can see that here.

The Digipo Chrome extension presents one set of helpers when you right-click on a page with nothing selected. Some of the helpers rely on Hypothesis, others just automate parts of the Digipo workflow — for example, launching advanced Google searches. When you right-click with a selection active, the Digipo Chrome extension presents another set of helpers which, again, may or may not rely on Hypothesis. One of them, Tag this Selection, works like Tag this Page in that it uses the Hypothesis API to create an annotation that includes a controlled tag. But Tag this Selection does a bit more work. It sends not only the URL of the target document, but also a Text Quote Selector that anchors the annotation within the document. In this case, too, the interaction then flows seamlessly into Hypothesis where you can edit the newly-created annotation and perhaps discuss the selected passage.

You can see more of the interplay between the Digipo and Hypothesis extensions in this screencast. I’m pretty excited by how this is turning out. The Digipo extension is Chrome-only for now, as is the Hypothesis extension, but WebExtensions should soon enable broader coverage. There’s still a need to plug packaged behavior directly into the Hypothesis client. But much can be accomplished with an extension that cooperates with Hypothesis using its existing set of affordances. The Digipo extension is one example. I can imagine many others, and I’m expanding my definition of componentware to include them.


1 I love how our copy editor insisted on hyphenating On-Line!