New expectations (and new opportunities) for stewards of public data

Here’s an update for those who’ve been following the story of my quest for local crime statistics[1,2]. This morning I met with the police chief and some other officials. Given that I began asking for this data in late April or early May, and went through four rounds of telephone and then email contact, it shouldn’t have taken so long to convene the meeting. And it would have taken longer had I not engaged my friend Ted Parent, who is a lawyer and a great champion of democracy, to write a letter to the city attorney. The magic incantation in New Hampshire, by the way, is not Freedom of Information Act (FOIA), but rather Right to Know. It wasn’t enough for me to utter those magic words, though. Ted had to do that, in a letter that went on to describe in great detail my reputation, qualifications, and seriousness of purpose. That description is true, but shouldn’t have been necessary, nor should Ted’s services have been.

In any event, we had a productive discussion and will meet again soon to discuss logistics: what’s unavailable and why, what’s available and how to get it. What will likely be available is an update to this data set, which might or might not reveal trends since 2005. That would be of interest locally because, while there’s a strong sense that crime is worse lately, nobody seems to be clear about the details.

But here’s why that might not help. The feds only gather and report on certain categories of crime. Among those not included, the chief told me, are drunk driving incidents, which he’s been seeing a lot more of lately. Another systematic omission: rapes only count as rapes when inflicted on females by males.

Then there’s the fact that state participation in the National Incident-Based Reporting System (NIBRS) — which is apparently the new name for what used to be called UCR (Uniform Crime Reporting) — is voluntary and spotty.

So it’s unclear what questions can even be answered — in local, state, or national context — by the UCR/NIBRS data that the city’s software can and does report to the feds.

But other questions are entirely outside the scope of that dataset. It includes no location information, for example. I was surprised to learn that while the city does of course collect street addresses when entering crime reports into its database, they’re unaware of any straightforward way to get the location data back out in order to visualize geographic patterns. My hunch is that I can help them with that, if I can get hold of a raw export, so that’s something we’re going to explore at our next meeting.

This has been an interesting process to observe. Today the assistant city attorney said something that crystallized, for me, an insight about the stewardship of public data. Although the city has so far received very few Right to Know requests, one of them, she said, could have proved very costly in terms of the software and consulting services that would have been needed in order to comply. That insight won’t rewrite the legacy system, but it certainly imposes an important new requirement on its successor.

The folks I met with today aren’t familiar with ChicagoCrime.org or CAPStat, but I didn’t get the impression they’re opposed to the idea of citizen participation in the interpretation of government data. On the contrary, I think they may conclude that deploying systems to enable that participation would be as useful to them as it would be to the public.

We have a long way to go at all levels: local, state, national, international. But expectations are being reset, up and down the line, and I’m hopeful that we’ll get where we need to go.