Gov2.0 transparency: An enabler for collaborative sense-making

Recently my town has adopted two innovative web services that I’ve featured on my podcast:, which does what its name suggests, and, which delivers video of city council meetings along with synchronized documents.

You can see the Keene instance of CrimeReports here, and our Granicus instance here.

I’m delighted to finally become a user of these systems that I’ve advocated for, written about, and podcasted. I’m also eager to move forward. We’re still only scratching the surface of what Net-mediated democracy can and should become.

In the case of CrimeReports, the next step is clear: Publish the data. It’s nice to see pushpins on a map, but when you’re trying to answer questions — like “Are we having a crime wave?” — you need access to the information that drives the map. Greg Whisenant, the founder of, says he’d be happy to publish feeds. But so far the cities that hire him to do canned visualizations of crime data aren’t asking him to do so, because most people aren’t yet asking their city governments to provide source data. So a few intrepid hackers, like Ben Caulfield here in Keene, are reverse-engineering PDF files to get at the information. Check out Ben’s remixed police blotter — it’s awesome. Now imagine what Ben might accomplish if he hadn’t needed to move mountains to uncover the data.

In the case of Granicus, I’m reminded of this item from last year: Net-enhanced democracy: Amazing progress, solvable challenges. The gist of that item was that:

  • It’s amazing to be able to observe the processes of government.

  • It’s still a challenge to make sense of them.

  • Tools that we know how to build and use can help us meet that challenge.

Check out, for example, last week’s Keene city council meeting. Scroll down to an item labeled 2. Ordinance O-2009-21. In this clip, the council agrees to amend the city code for residential real estate tax exemptions. I wish I could link you directly to that portion of the video, which begins at 34:11, in the same way that I can link you to the associated document. But more broadly, I wish that a citizen who tunes in could understand — and help establish — the context for this amendment.

Here’s the new language:

Sec. 86-29 Residential real estate tax exemptions and credits

With regard to property tax exemptions, the city hereby adopts the provisions of RSA 72:37 (Blind); RSA 72:37-b (Disabled); RSA 72:38-b (Deaf or Severely Hearing Impaired); RSA 72:39-a (Elderly); RSA 72:62 (Solar); RSA 72:66 (Wind); and RSA 72:70 (Wood).

With regard to property tax credits, the city hereby adopts the provisions of RSA 72:28, II, (Optional Veterans’ tax credit); RSA 72:29-a , II, (Surviving Spouse); and RSA 72:35, I-a, (Optional Tax Credit for Service-Connected total disability).

In this case, I just happen to know a bit of this amendment’s backstory. Earlier this year I found out — only thanks to a serendipitous encounter with a city councilor at a social event — that my wood gasifier qualified me for an exemption. This was the first such exemption, and to my knowledge is still the only one granted.

If I hadn’t gone through that experience, though, the video clip and its associated document would mean nothing to me. There would be no way to make a connection between state law on the one hand, and a documented case study on the other.

On the next turn of the crank, I hope that services like Granicus will enable us to make those connections. Seeing the process of government in action is a great step forward. Now we need to be able to use links and annotations to help one another make sense of that process.

Where is the money going?

Over the weekend I was poking around in the recipient-reported data at I filtered the New Hampshire spreadsheet down to items for my town, Keene, and was a bit surprised to find no descriptions in many cases. Here’s the breakdown:

# of awards 25
# of awards with descriptions 05 20%
# of awards without descriptions 20 80%
$ of awards 10,940,770
$ of awards with descriptions 1,260,719 12%
$ of awards without descriptions 9,680,053 88%

In this case, the half-dozen largest awards aren’t described:

award amount funding agency recipient description
EE00161 2,601,788 Sothwestern Community Services Inc
S394A090030 1,471,540 Keene School District
AIP #3-33-SBGP-06-2009 1,298,500 City of Keene
2W-33000209-0 1,129,608 City of Keene
2F-96102301-0 666,379 City of Keene
2F-96102301-0 655,395 City of Keene
0901NHCOS2 600,930 Sothwestern Community Services Inc
2009RKWX0608 459,850 Department of Justice KEENE, CITY OF The COPS Hiring Recovery Program (CHRP) provides funding directly to law enforcement agencies to hire and/or rehire career law enforcement officers in an effort to create and preserve jobs, and to increase their community policing capacity and crime prevention efforts.
NH36S01050109 413,394 Department of Housing and Urban Development KEENE HOUSING AUTHORITY ARRA Capital Fund Grant. Replacement of roofing, siding, and repair of exterior storage sheds on 29 public housing units at a family complex

That got me wondering: Where does the money go? So I built a little app that explores ARRA awards for any city or town: For most places, it seems, the ratio of awards with descriptions to awards without isn’t quite so bad. In the case of Philadelphia, for example, “only” 27% of the dollars awarded ($280 million!) are not described.

But even when the description field is filled in, how much does that tell us about what’s actually being done with the money? We can’t expect to find that information in a spreadsheet at The knowledge is held collectively by the many people who are involved in the projects funded by these awards.

If we want to materialize a view of that collective knowledge, the ARRA data provides a useful starting point. Every award is identified by an award number. These are, effectively, webscale identifiers — that is, more-or-less unique tags we could use to collate newspaper articles, blog entries, tweets, or any other online chatter about awards.

To promote this idea, the app reports award numbers as search strings. In Keene, for example, the school district got an award for $1.47 million. The award number is S394A090030. If you search for that you’ll find nothing but a link back to a page entitled Where is the Money Going? can’t bootstrap itself out of this circular trap. But if we use the tags that it has helpfully provided, we might be able to find out a lot more about where the money is going.

Purple Numbers for PDF documents?

My contribution to Silona Bonewald week was an interview about her new project Silona proposes two new features for government websites. First, change tracking. Second, permalinks for documents, sections, and paragraphs.

Nobody will dispute the need for, or utility of, these features. The question is how to implement them across a sprawling landscape of content management systems and publishing procedures that still, in many cases, regard print as canonical and the web as an afterthought.

In a follow-on discussion with Silona, on the citability wiki, I recalled a little-known and rarely-used feature of PDF documents. You can form URLs that point to specific pages. And with the right preparation, you can even form URLs that point to named destinations within pages.

Those of us fluent in web-friendly document formats like HTML and XML will tend to recommend that these become canonical. But having recently observed what happened when the old-fashioned non-XML method of math typesetting was supported by, I have to ask: How much more mileage might we be getting out of the existing print-oriented systems?

I am not an expert user of PDF authoring tools, nor an expert user of software libraries that enable programmatic manipulation of PDF files. But some of you are. What would it take, I wonder, to post-process the kinds of PDF files that governments typically produce, in order to add Purple Numbers?

The civic dashboard

On Friday my local paper ran a story entitled Keene crime rates steady over years. Because that link will go dark soon, I’m going to assert fair use for the part of the story that cites statistics:

Strings of vehicle break-ins and vandalism and the occasional vicious beating or stabbing may lead some to believe that Keene’s streets are getting meaner, but crime statistics show little change over the last six years.

Even in light of rough economic times, which typically parallel a spike in shoplifting — people begin stealing groceries or other necessities they can no longer afford — the Elm City’s property crime rate remains stable.

The city’s social programs, such as The Community Kitchen, which provides food to area residents in need, play a significant role in curbing crime, Keene police Lt. Jay U. Duguay said.

“We’re behind the nation when it comes to economic issues. People are still losing their homes and jobs, but overall we haven’t felt the effects of it yet,” he said. “Right now it’s wait-and-see.”

During the last six years, Keene police have received an average of 490 reports dealing with larceny or theft. Last year they took 667 reports of larceny or theft, the highest number of those types of crimes since 2002, which saw 604 reports.

From the beginning of this year to the end of April, there were 202 reports of larceny and theft, slightly higher than the 147 during the same period last year, and 33 burglaries, which is on par with previous years.

“There’s going to be periods with a little influx, but for the most part it’s steady,” Duguay said. “I was actually kind of surprised at how consistent the numbers were.”

In 2004 and 2005, property crime rates dipped dramatically. While 2003 saw 557 larcenies and thefts, that number hit 272 the following year and then slightly increased to 286 the next year before rising to 455 in 2006.

“We didn’t change our patrol procedures during those times (2004 and 2005) and we weren’t up to full staff. So I don’t know why those years are lower,” Duguay said. “I think the more consistent number is the high number, but thank goodness for the lows.”

Violent crime reports in Keene have also remained steady over the last several years, with an average of 366 assaults annually.

Between 20 and 30 sex assaults are reported in the city each year, though only a small fraction of those cases result in arrests because the others lack sufficient evidence, Duguay said.

Statistics only tell part of the story, though. For the crime victims, the numbers hold little meaning.

The story concludes with anecdotes from townsfolk who either do, or don’t, believe that tough economic times are making Keene’s streets meaner.

I quote at length from it here because I think it captures a moment in time. The story seems to be data-driven, but not in the way that many of us now realize such stories can be. The reporter got some numbers from the police department, and the story quotes a lieutenant’s interpretation of those numbers, but there’s nothing available for an interested citizen to verify or falsify. And there’s no reference to an alternative source — from the US Bureau of Justice — that could confirm, challenge, or otherwise contextualize the numbers.

I hope that my response, below, also marks a moment in time — one in which people didn’t demand, governments didn’t provide, journalists didn’t exploit, and all these groups didn’t collaboratively engage with more and better evidence than informs most civic dialogue today.

From time to time, communities ask: Are we having a crime wave? A couple of summers ago it seemed that way. The Sentinel invited TalkBalk comments, and one person wrote: “Keene has gone downhill. Once a peaceful, quaint city that was safe, it is no more.”

We shouldn’t have to just speculate about these trends though. We should be able to look at the data and draw reasonable conclusions. Increasingly, we can.

In 2007 I looked, and the first source I found was the data reported by the Keene police department (and every other police department around the country) to the US Bureau of Justice. I noticed a couple of things. First, the numbers showed no uptick in violent crime. But since they stopped in 2005, they didn’t address concern about events in the then-current 2006-2007 period.

Second, because the numbers went back to 1985, they revealed a remarkable anomaly. There was a huge spike in violent crime — assaults and rapes — from 1990 to 1994. You can see the trend plainly in the charts and data I’ve posted at What happened then? How should this historical context influence our perception of current trends? I’d love to see the Sentinel ask, and try to answer, these questions.

Since the Bureau of Justice data wasn’t current enough to address the 2006-2007 concerns about crime, I asked the police department to provide me with more recent data. In the end, after multiple requests and some nudging by an attorney, they complied. The snapshot I received, with numbers through July 2007, showed no evidence of a recent uptick in either violent crime or property crime. That was

It was also enlightening to compare the raw data in the police spreadsheet to the numbers reported to the Bureau of Justice. They don’t exactly line up. This isn’t nefarious, it’s just what happens when local systems try to mesh with national systems. There is a lot of local variation in the classification of different types of crimes, and room for interpretation when you bundle them into larger

Fast forward to summer 2009. The economy has tanked, and people are again wondering whether we’re having a crime wave. The Sentinel gathered some data, talked to the police, and concluded — I suspect correctly — that as before, the perception of a crime wave is not the reality.

For the reasons I’ve explained, the police department numbers reported in the Sentinel don’t quite line up with those reported to the Bureau of Justice. Consider larceny-theft, for example:

          2003   2004   2005   2006
Sentinel   557    272    286    455
Justice    534    245    235    622

But I do wonder about this:

“Violent crime reports in Keene have also remained steady over the last several years, with an average of 366 assaults annually.”

I hope that’s an error. According to the Bureau of Justice there were at most about 100 violent crimes per year, back in the dark ages of 1990-1994, and we’ve averaged between 40 and 60 per year from then until 2007.

In any case, here’s the larger point. Cities around the country have begun to realize that the operational data of city government can be made available to everyone — citizens as well as journalists — so that we can all monitor the health of our cities in a collaborative way. Crime statistics are one popular category of data, others include restaurant inspections, infrastructure repairs, and licensing.

Nowadays it costs about $100/month to augment a police department’s information system with software that reports current crime statistics online, and also displays the locations of crimes on a map. In New Hampshire, one such system ( has been installed in Exeter, Hampton, Laconia, and Rochester.

I’d love to see the Keene police department join that club. A civic dashboard is part of what I proposed during the Community Visioning Process. But there’s no need to wait until 2028. Cities around the country are creating their dashboards now, and we can too.

Influencing the production of public data

In the latest installment of my Innovators podcast, which ran while I was away on vacation, I spoke with Steven Willmott of 3scale, one of several companies in the emerging business of third-party API management. As more organizations get into the game of providing APIs to their online data, there’s a growing need for help in the design and management of those APIs.

By way of demonstration, 3scale is providing an unofficial API to some of the datasets offered by the United Nations. The UN data at, while browseable and downloadable, is not programmatically accessible. If you visit 3scale’s demo at you can sign up for an access key, ask for available datasets — mostly, so far, from the World Health Organization (see below) — and then query them.

The query capability is rather limited. For a given measure, like Births by caesarean section (percent), you can select subsets by country or by year, but you can’t query or order by values. And you can’t make correlations across tables in one query.

It’s just a demo, of course. If 3scale wanted to invest more effort, a more robust query system could be built. The fact that such a system can be built by an unofficial intermediary, rather than by the provider of the data, is quite interesting.

As I watch this data publication meme spread, here’s something that interests me even more. These efforts don’t really reflect the Web 2.0 values of engagement and participation to the extent they could. We’re now very focused on opening up flexible means of access to data. But the conversation is still framed in terms of a producer/consumer relationship that isn’t itself much discussed.

At the end of this entry you’ll find a list of WHO datasets. Here’s one: Community and traditional health workers density (per 10,000 population). What kinds of questions do we think we might try to answer by counting this category of worker? What kinds of questions can’t we try to answer using the datasets WHO is collecting? How might we therefore want to try to influence the WHO’s data-gathering efforts, and those of other public health organizations?

“Give us the data” is an easy slogan to chant. And there’s no doubt that much good will come from poking through what we are given. But we also need to have ideas about what we want the data for, and communicate those ideas to the providers who are gathering it on our behalf.

Adolescent fertility rate
Adult literacy rate (percent)
Gross national income per capita (PPP international $)
Net primary school enrolment ratio female (percent)
Net primary school enrolment ratio male (percent)
Population (in thousands) total
Population annual growth rate (percent)
Population in urban areas (percent)
Population living below the poverty line (percent living on less than US$1 per day)
Population median age (years)
Population proportion over 60 (percent)
Population proportion under 15 (percent)
Registration coverage of births (percent)
Registration coverage of deaths (percent)
Total fertility rate (per woman)
Antenatal care coverage – at least four visits (percent)
Antiretroviral therapy coverage among HIV-infected pregnant women for PMTCT (percent)
Antiretroviral therapy coverage among people with advanced HIV infections (percent)
Births attended by skilled health personnel (percent)
Births by caesarean section (percent)
Children aged 6-59 months who received vitamin A supplementation (percent)
Children aged less than 5 years sleeping under insecticide-treated nets (percent)
Children aged less than 5 years who received any antimalarial treatment for fever (percent)
Children aged less than 5 years with ARI symptoms taken to facility (percent)
Children aged less than 5 years with diarrhoea receiving ORT (percent)
Contraceptive prevalence (percent)
Neonates protected at birth against neonatal tetanus (PAB) (percent)
One-year-olds immunized with MCV
One-year-olds immunized with three doses of Hepatitis B (HepB3) (percent)
One-year-olds immunized with three doses of Hib (Hib3) vaccine (percent)
One-year-olds immunized with three doses of diphtheria tetanus toxoid and pertussis (DTP3) (percent)
Tuberculosis detection rate under DOTS (percent)
Tuberculosis treatment success under DOTS (percent)
Women who have had PAP smear (percent)
Women who have had mammography (percent)
Community and traditional health workers density (per 10 000 population)
Dentistry personnel density (per 10 000 population)
Environment and public health workers density (per 10 000 population)
External resources for health as percentage of total expenditure on health
General government expenditure on health as percentage of total expenditure on health
General government expenditure on health as percentage of total government expenditure
Hospital beds (per 10 000 population)
Laboratory health workers density (per 10 000 population)
Number of community and traditional health workers
Number of dentistry personnel
Number of environment and public health workers
Number of laboratory health workers
Number of nursing and midwifery personnel
Number of other health service providers
Number of pharmaceutical personnel
Nursing and midwifery personnel density (per 10 000 population)
Other health service providers density (per 10 000 population)
Out-of-pocket expenditure as percentage of private expenditure on health
Per capita total expenditure on health (PPP int. $)
Per capita total expenditure on health at average exchange rate (US$
Pharmaceutical personnel density (per 10 000 population)
Physicians density (per 10 000 population)
Private expenditure on health as percentage of total expenditure on health
Private prepaid plans as percentage of private expenditure on health
Ratio of health management and support workers to health service providers
Ratio of nurses and midwives to physicians
Social security expenditure on health as percentage of general government expenditure on health
Total expenditure on health as percentage of gross domestic product