Category: visualization

The Internet’s secret sauce: surfacing coincidence

What is it that makes my favorite online services so compelling? I’m talking about the whole family of services that includes Dopplr, Wesabe, Twitter, Flickr, and del.icio.us among others.

I find it interesting that people don’t generally refer to any of these as “web sites”. They are “services”.

I was fortunate enough to spend some time with Dopplr’s Matt Biddulph and Matt Jones last week while in London where they described the architecture of what they’ve built in terms of connected data keys. The job of Dopplr, Mr. Jones said, was to “surface coincidence”.

I think that term slipped out accidentally, but I love it. What does it mean to “surface coincidence”?

It starts by enabling people to manufacture the circumstances by which coincidence becomes at least meaningful if not actually useful. Or, as Jon Udell put it years ago now when comparing Internet data signals to cellular biology:

“It looks like serendipity, and in a way it is, but it’s manufactured serendipity.”

All these services allow me to manage fragments of my life without requiring burdensome tasks. They all let me take my data wherever I want. They all enhance my data by connecting it to more data. They all make my data relevant in the context of a larger community.

When my life fragments are managed by an intelligent service, then that service can make observations about my data on my behalf.

Dopplr can show me when a distant friend will be near and vice versa. Twitter can show me what my friends are doing right now. Wesabe can show me what others have learned about saving money at the places where I spend my money. Among many other things Flickr can show me how to look differently at the things I see when I take photos. And del.icio.us can show me things that my friends are reading every day.

There are many many behaviors both implicit and explicit that could be managed using this formula or what is starting to look like a successful formula, anyhow. Someone could capture, manage and enhance the things that I find funny, the things I hate, the things at home I’m trying to get rid of, the things I accomplished at work today, the political issues I support, etc.

But just collecting, managing and enhancing my life fragments isn’t enough. And I think what Matt Jones said is a really important part of how you make data come to life.

You can make information accessible and even fun. You can make the vast pool feel manageable and usable. You can make people feel connected.

And when you can create meaning in people’s lives, you create deep loyalty. That loyalty can be the foundation of larger businesses powered by advertising or subscriptions or affiliate networks or whatever.

The result of surfacing coincidence is a meaningful action. And those actions are where business value is created.

Wikipedia defines coincidence as follows:

“Coincidence is the noteworthy alignment of two or more events or circumstances without obvious causal connection.”

This is, of course, similar and related to the definition of serendipity:

“Serendipity is the effect by which one accidentally discovers something fortunate, especially while looking for something else entirely.”

You might say that this is a criteria against which any new online service should be measured. Though it’s probably so core to getting things right that every other consideration in building a new online service needs to support it.

It’s probably THE criteria.

Oakland Trib’s Not-Just-A-Number improves on crime data visualization

OJR’s Jim Wayne dives into Oakland Tribune’s “Not Just A Number” web site. The service won the Service Journalism Award from ONA for an amazingly powerful view of crime data.

The basic premise was to create a data visualization for Oakland homicide crime data that made the victims and, more importantly, the people in their lives real participants in the story rather than pure statistics (or just plain ignored entirely).

It’s a very powerful site and a model for all local newspapers to follow. It’s disappointing but no surprise the media creates these kinds of community services before local governments do. At least we’re getting more access to crime data.

Wayne also points to a crime data visualization from the Los Angeles Times called The Homicide Map that I wasn’t aware of.

They have a nice map mashup that takes a more statistical approach, yet they also include things like images of the victims.

Unfortunately, as Oakland Tribune producers Katy Newton and Sean Connelley point out, a mug shot is not a fair image to use for a violent crime victim in a statistical map. But I’m glad to see them exposing data that needs to be shared.

732 homicides in Los Angeles so far in 2007! Unbelievable.

Ikea ruined my floors

My second child is due next week. I intended to reconfigure our 1-bedroom house to create 2-bedrooms so that all 4 of us and the dog could spread out a bit.

Some amazing software from Google and Ikea made me feel more ambitious.

Now I have 2 unfinished bedrooms, a new but incomplete kitchen, dust in places I didn’t think dust could find, large and somewhat dangerous gaps in the floors and a couple of contractor battle scars. The new baby won’t remember the state of things, but no doubt my wife will keep the memory alive for years to come.


It all started with SketchUp. I spent several hours mapping out our house trying to assess what was possible. I was able to move walls around in a 3D model and imagine with some accuracy what it would be like to live in our remodelled house.

This visualization gave me language and vision to communicate with contractors and helped me budget the work. It also gave me the confidence to make some more dramatic changes than what we conceived in our heads.

What could have been a day or two of demolition and some simple framing work turned into major structural work that altered the feel of the house entirely.


Then, as we were closing down on the scope of the project and looking to finish in early August with plenty of time to spare, I started playing with Ikea’s downloadable kitchen planner software. Despite our time and budget constraints I couldn’t resist the idea of planning ahead a little.

After you design the space and create your grid, you choose cabinets from Ikea’s collection. You drag and drop them on the canvas and fit them together the way you want. It’s actually a lot of fun despite being very buggy.

When you give buyers power and easy onramps to services you turn them into valuable customers instead of just drive-by shoppers. Here’s how they hook you into buying from their shop…

Once you can see your kitchen in 3D and move it around and pretend to cook in it, you can then click to see a price sheet for your plan. Ikea, as you know, is very reasonably priced. So, suddenly you feel like you can afford an awesome new kitchen.

Now all I could think of was how to adjust our plans so that we could afford a new kitchen. I was also anxious to see if the kitchens looked ok or if they looked like the other prefab swedish lego blocks you often get from Ikea.

Ikea’s software is brilliant on several levels…

First, I’m much less likely to try another vendor once I’ve already perfected my plan. Second, they have enough range in their designs that you can’t help but think that one of the choices there will work. Third, you can essentially go out and get your kitchen now. They have to help you work out a few details, but you could, in theory, have your kitchen parts in hand same-day.

Whereas, we might have only considered Ikea for some handy paper towel hooks and cutlery organizers previously, we ended up buying everything but the countertop and appliances from Ikea.


The best part of doing your kitchen with Ikea’s planner, in my mind, is the fact that you can print out the plan and march into the building inspection office to get your plan approved on your own. My wife did just that without having any building experience. She was back home after an hour, permit approved. I really wish we had done that with the overall job, but instead we payed exorbitant fees for professional drawings. I’ll never make that mistake again.

There’s a ton of work left to do on the house that we don’t know exactly how to fund, yet.

I imagine smart retailers like Ikea are hoping that budget planning software doesn’t evolve fast enough to help people like me realize that my cash is probably better spent on things like dealing with the nails sticking out of my floors before I buy nice new cabinets.

Crime data stories

My Potrero Hill neighbors tell me that the sweet song of crackling firearms in the evening always begins again in May as the days get longer, hotter and schoolless.

Recently, I witnessed a sample of the gun play happening in the nearby projects, and I decided to do some of my own research to understand what’s going on. The first thing I found was that I wasn’t the only witness to this particular incident:

“Two of the bullets hit our daughters bedroom– one went through the wall and crossed a small portion of the room and lodged in another wall near her sliding glass door.

[The Police] told us that based on the 24 bullet shells they found up the hill on Missouri St. near the public housing, there were two guns involved, one of which was an AK47 the other was probably a 9mm pistol. The police have no idea who was firing the guns and given that there are not witnesses, there is not likely to be any resolution to the incident. The officers were confident that the two bullets that hit the condo were random and not targeted at us.”

There are lots of factors behind violent neghborhoods, and the San Francisco projects are pretty densely representative of many of those factors. But it really irritates me that guns are so prevalent in the area, and, in general, so prevalent in America.

So, I started my journey at the old PotreroHillSF Crime Mashup which apparently doesn’t work any more. There is an ongoing “Police Blotter” on the site, though, with some good reporting.


I then found the official San Francisco Police Department Crime Map. Of course, the data is wrapped in their own heavy-handed user interface and unavailable in common shareable web data formats. The tool is burdened with legal trappings and strangely fails to acknowledge homicides, though they offer an explanation:

“A homicide may not appear correctly on the map because:

  1. The incident was initially reported as an assault and the victim died some time later from the injuries.
  2. The incident was reported as an arson, and the body was not found until a later time.
  3. A body was found and the cause of death was not obvious to the officer making the incident report.”

I’m hoping that the City has more advanced reporting capabilities internally, as it seems pretty obvious that we have a data visualization failure going on here. I can see some data around assaults, robberies, larceny, vandalism, drug incidents, etc.

But the compelling visual storytelling is missing.

I want to know how many crime incidents in the projects this year involved guns. How many guns in these events are registered/unregistered? How many of the gun incidents were or became homicides vs non-gun related incidents? Where did the guns come from? What kinds are being used?

I suspect most guns aren’t registered which is an argument used by those who think a gun ban would be useless. People who want guns will find them, legal or not. But I also suspect that the victims aren’t carrying guns. Thus, the argument that people should have the right to own a gun to protect themselves isn’t a counterbalancing force. People who avoid violence won’t carry guns, legal or not.

As I progressed with this research I realized that somewhere in between raw data and overt campaigning is an interesting space. Data can help us learn and make more intelligent and informed decisions about how to manage and evolve our society and its rules.

Unfortunately, that space seems more difficult to find than it should be. I should be able to download data for myself or at least be able to visualize the stories behind the data in relevant pictures and charts.

Of course, there’s the fantastic ChicagoCrime.org web site which has done a lot to raise awareness about crime data. Despite the lack of available data from the local government, site owner Andrian Holovaty found a way to collect what he needed to make this site through an automated script:

“Each weekday, my computer program goes to the Chicago Police Department’s website and gathers all crimes reported in Chicago.”

The site has some great info (such as this screenshot of “Armed Robbery: Handgun Incidents”), though I still want to see an editorial lens on this data that puts a bit more meaning behind it.

For example, it only takes a glance to see in this series of Census images of San Francisco that the City is incredibly segregated, something I think many residents choose to ignore under the mask of open-mindedness. Even here, though, the story is incomplete without some intelligence wrapped around the data. What’s the trend? Is it becoming whiter? Where are people going who are leaving?

This same question punctures my happy place every time I exit onto Palo Alto’s University Avenue from Highway 101 and pass what is now a high end office park where one of the most dangerous areas in the country used to exist only a decade or so ago. I’m very pleased it’s a safer place, but do we understand the cost of that transition? Where did those people go? Are they better off?

Yahoo! colleague Micah Laaker pointed me to an interesting project he worked on back in 2002 and 2003 called the Denver Census Tract Animation Project. He worked with Citizen Mapmakers to trend movement of the African-American population in Denver from 1960 to 2000. Here’s a snapshot of their work:

I really like the way they visualized data to tell a story here. We need similar visualizations for crime data.

The InfoPlease “School Shootings” site gets closer to telling a story about guns just by focusing on a type of statistic and representing it. What a powerful domain name! However, the data here is still pretty raw and limited. This is hugely important information, but there’s an implicit argument here that should be made much more explicit with actionable information and analysis. In its current state it’s just telling us that there are a lot of school shootings (a surprising number in Europe, actually).


The Citizen Crime Watch site for New Orleans gets even closer to what I want to see. Similar to ChicagoCrime.org, they visualize with your standard data-on-a-map mashup, but the hover links point to coverage in the local media. I’m suddenly given a much more human window into the crime scene, and I can read about each event. For example, on April 9, 2007, there was a homicide in a trailer park:

“…Officers found Williams lying on the floor of the trailer with blunt-force trauma to her head. Emergency medical technicians declared her dead at the scene. An autopsy shows she had been beaten to death, said John Gagliano, chief investigator for the Orleans Parish coroner’s office.

The trailer is in a trailer park at 6801 Press Drive run by the Federal Emergency Management Agency. Although the trailer park is near the campus of Southern University, the chancellor, Victor Ukpolo, said neither faculty nor students live there.

The murder is being investigated by Detective Harold Wischan, who can be reached at (504) 658-5300.”

I’m very thankful for local reporting from sites like Nola.com, The Times Picayune, and community leaders such as Mike Lin of PotreroHillSF and the increasingly active Yahoo! Group Potrero Hill Parents Association who all help surface this kind of information, but it’s not enough. The City needs make it easier for its residents to both report on things that matter to us and to collect the data, filter it, and act on it.

People will always want greater access to information. This is particularly true in communities where poor decision-making creates mistrust:

“Under pressure from constituents who say New Orleans police stonewall requests for crime data, the City Council’s criminal justice subcommittee took police representatives to task Wednesday, calling for a faster, freer flow of public information…When asked for a written breakdown of policy and procedures relating to the release of public information, Maj. Michael Sauter, the head of technology, told the council most of that information was ‘not meant for the public.'”

Similarly, Rick Klau has begun experimenting with this kind of thing in response to the Magnetix toy recall incident. He calls it “Open source parenting” and observes that bottom-up community-driven politics is likely to be more successful than anything a politician can enable:

“If the government is under-staffed and under-funded to help parents avoid harmful toys, then why can’t we help ourselves?…Give thousands of parents the tools to easily identify harmful products, leverage the community’s ability to provide visibility to legitimate threats while minimizing less serious risks, and quickly disseminate information that could be instrumental in avoiding a serious accident.”

I’m suddenly wondering what role politicans will play if communities are able to form solutions to issues locally, nationally and internationally on their own. Maybe instead of legislators (or merely professional campaigners/marketers), politicians will become community managers.

I also start wondering what politicians do all day if they can’t sort out ways to curb violence in our neighborhoods. I don’t see why anyone living in this country or any other should have to worry about whether their child will be shot accidentally in his or her bedroom by stray AK47 bullets or intentionally while at school.

I’m convinced the answer is in the data that is already being collected in various government crime databases. And I’m sure the answer is related to gun access.

Where is Tufte when you need him?