Author: Matt McAlister

Breaking through the attention barrier

For some reason I get a bit annoyed when people write about our information overload limits. This happened the other day when I saw Seth Godin’s piece titled “Warning: The internet is almost full.” He writes:

“The decentralized nature of the net means that it will never be physically full. As long as we can keep making hard drives, we won’t run out of space to store those inane videos of your Aunt Sally. What is full is our attention.”

I just refuse to believe that we’ve hit the ceiling of what the human brain can deal with.

There is no doubt that we have a lot of useless information available to us, much of it pushed at us, cluttering our lives in really irritating ways. But information overload is a symptom of some bigger issues that we can and should resolve.

I think it’s about better linguistics, technologies and education, to begin with. More broadly, it’s about how we collectively understand and apply abstraction layers to manage a more complex world.

Like everyone, I hit my attention limit nearly every day. Seth is right when he says “You can’t read every important blog… you can’t even read all the blogs that tell you what the important blogs are saying.”

That’s a reason to explore some more, not to give up. We shouldn’t become fatalistic about the future of information or look down our noses at all that messy stuff strewn about the Internet. I never want the flow of information to slow down or, worse, retract, no matter how much mess gets in the way of finding the stuff that matters to me.

What we may need are more dramatic changes in our language, more effective information discovery services, more experience-based education programs both for kids and adults, and, perhaps even more important than all that, an altered world view that can accommodate and make the most of the vast resources that are now part of our culture forever.

Hacking BNP data

Less than a week after trying out some new data mapping concepts at Guardian Hack Day, a big pile of data appeared on the Internet begging to be mapped. Using some of their new skills and a convenient constituency data tool, a small team of innovators got to work to produce some really interesting data-driven journalism.

Mat Wall details what happened behind the scenes:

“[Simon Willison] wrote a piece of code to extract the 12,000 BNP member’s postcodes through the They Work For You constituency API. Now he had a voting constitency for each person on the list. He then injected this data back into his hack day project to plot this information onto the map obtained from Wikipedia. This took about an hour.”

Update: The project didn’t end there, it turns out. The infographics guys used the concept for the newspaper the next day. Here’s a picture of what was printed:
Hack Day Map in The Guardian Newspaper

Notes from Hack Day at The Guardian

We hosted our first Hack Day last week at The Guardian. Amazing fun.

Here’s a 15min highlight reel:

We did a lot of the standard stuff that makes Hack Day so interesting, but there were a few innovations to the event format itself that I thought worked really well, too:

  1. DabbleDB. Simon Willison setup a simple hack submission queue using DabbleDB, a handy online database tool. It’s as if the software was designed for this purpose. Two nice benefits: 1) you can upload a screenshot with your submission which it displays nicely, and 2) it prints beautifully. I handed out a hardcopy of the hack demo queue for each judge who then used the list to take notes.
  2. Double Screens. We setup 2 projectors so we could jump back and forth between presentation locations and save some time. While one person was presenting, the next person was setting up on the other screen. I was a little worried it would be distracting, but that wasn’t a problem at all.

    I think this is primarily what kept the pace up. We got through 37 hacks in just about an hour. At that pace you couldn’t really afford to look away. Oh, and Simon’s lightning timer was hugely helpful, too.

    This then had the nice effect of giving the judges more time to deliberate…

  3. Comprehensive recognition. The judges went through every single hack and found a way to acknowledge each participant. Emily Bell did a sort of improv act dishing out the jokes. She first went through all the hacks that “we would have given an award to”. Then she handed out the trophies…
  4. The Guardian Hack Day TrophySilly trophies. These worked perfectly. You can keep it on your desk. It makes no sense to anyone else. And it reinforces the idea that the recognition is for the work itself, not for winning a competition. We did hand out a couple of Flip cameras and Make Magazine generously offered some free subscriptions for the hardware hacks, but the emphasis was clearly on the hackers and their hacks, not the idea of ‘winning’.

Otherwise, it seemed to operate much like other Hack Days, except for the refreshing focus on hacks that mean something. I wasn’t sure what kind of hack quality to expect which was in fact very high, but I loved the fact that most of the hacks had the added dimension of context.

Many times a Hack Day results in a lot of amazing technology solutions for problems that don’t exist. I would never challenge the value of creativity for creativity sake, as that’s a big part of what Hack Day is about. But I was really happy to see that in addition to the impressive technical hacks things like Ben Griffiths’, Rob McKinnon’s and Simon Willison’s hacks (to name a few) presented data and information in new ways that could influence the way people think about what they are reading or interacting with.

Anyhow, the event was fantastic, and I’m really looking forward to doing it again.

Celebrating in the streets of San Francisco

Off topic here, but this video is worth sharing. My wife and I decided to take a stroll after the election speeches. We were heading for a pub but got sidetracked by all the noise in the streets of San Francisco. Then we saw a crowd forming and cops smiling, not a recognizable combination.

This is what we saw (1 min):

Some other accounts of the event:

Neil Girling: “Extra Action Marching Band led the crowd in jubilation, and there were many cheers and chants of “Obama” and “U S A.” The mood was ecstatic, and the cops were polite and extremely hands-off; a little after midnight, when they finally asked Extra Action Marching Band — who have a reputation for chaos and noise — to start to shut down, they did so with smiles and were met with the same.”

Tim Redmond, SFBay Guardian: “San Francisco is going crazy. I haven’t seen this much excitement in the streets since we shut down the city when the Iraq war began. But this time, we actually have something to celebrate.”

Sean Bonner, SF Metblogs: “My neighborhood went 9 kinds of insane last night.”

Strange Things Will Happen: “It was like Italy had won the World Cup, only with less mopeds and more high-fiving.”

Why the open strategy is a good idea

The giant dotcoms and innovative Internet startups have been testing and proving approaches to openness as a strategy over the last few years. The ideas seem to have permeated some of the farther corners of the Internet now, and I think we’re on the cusp of an explosion of open strategies and concepts across all markets, not least of which is the publishing world.

The most recent notable examples came out of The New York Times and BBC.

The New York Times took Federal Elections Commission data and repurposed it for people who build web applications. The FEC site is good and the data is mildly useful there, but The New York Times is able to shine a spotlight on what they are doing and make a more useful interface to their data. Given the power of their brand they can also help pull together people from around the Internet who understand data visualization and how to build interactive web sites.

The New York Times is embracing raw data in ways that give context and meaning to the mashup world that has grown so much in the last few years.

Jeff Jarvis’ NYC startup DayLife is another interesting but different type of example:

“Our technology collects content from thousands of high-quality online sources, deeply analyzes and parses it, and creates a trove of data that can then be reused in an infinite number of ways by publishers of all sizes.”

The DayLife platform gives them the ability to not only create a pretty good destination site but also to power other people’s web sites with interesting content they have identified and classified. We have used it on guardian.co.uk to enhance our Olympics coverage and US Elections coverage, among other things.

But DayLife went the extra step recently in making it possible for any of the content owners in their system to open up and release their content as an API via the DayLife platform, the same way The New York Times released data from the FEC via nytimes.com.

DayLife is playing data aggregator and distributor for publishers who haven’t figured out how to do this themselves:

“Using Enterprise API, you can launch your OWN API to your OWN CONTENT, all powered by Daylife’s proven technology platform. And you can do this in a matter of days.

You’ll get to choose whether to make your API available to the world at large, or limit access to your in-house developers. In either event, Enterprise API lets you break out of the bottleneck of your in-house CMS to start building new stuff with your content, immediately.”

What starts to become clear when you look at these kinds of services is that media organizations should improve how they facilitate information flow. This is actually what journalism has always been all about, releasing important information to people in a useful way. The media businesses that figure out how to make information flow more fluidly across the entire Internet are going to win over time.

That’s one of the reasons we released full content RSS feeds this week on guardian.co.uk. Rather than require people to visit the web site to read articles we post, they can get what they want when and where they want it from us.

We want to make guardian.co.uk useful to people in whatever context matters to them wherever they are on the Internet.

This is a must-have feature of publishing today even though Forrester’s recent study shows that very few people understand RSS. Mark Hopkins of Mashable.com noted that opening out is not just about serving end-users. It’s about being a part of the activity streams happening all across the Internet:

“If you turn your attention to the most popular sites on the web, sites like Facebook, MySpace and Google all have syndicated content strewn all through them. Let alone sites like FriendFeed, Plaxo and and thousands of blogs and news sites out there that rely on aggregation of content via RSS.”

But the open strategy is not just about opening out. It’s also about opening in.

BBC.co.uk/music is a really good example. They recently began integrating data from other sites and then releasing the raw information in ways that developers would be able to use elsewhere, as well.

“By adopting [the MusicBrainz] open standard, our pages are able to benefit from public domain content linked from MusicBrainz such as biographies from Wikipedia and discographies from MusicBrainz. But MusicBrainz IDs also make it straightforward for third parties to work with our data and automate links to our pages.”

BBC is linking both literally and figuratively to the open web.

The benefits to doing this are obvious and immediately tangible on some levels and then much more strategic and a bit abstract on others.

First, Google’s search engine will reward sites that reference and get referenced by authoritative and relevant sources. BBC’s Matthew Shorter adds,

“Having a persistent, and increasingly rich, resource to link to for each artist on the BBC should also help those pages appear higher up the search rankings”

Second, being the essential data source for a particular topic creates lots of opportunity as the number and size of the customers of that data increases.

For example, you can be sure that Tele Atlas made a lot of money when Google committed to a 5-year deal for their location information:

“The agreement spans Google’s current and future map-based services and navigation offerings across mobile, online and desktop environments. These include the Google Mapsâ„¢ and Google Earthâ„¢ services and mobile applications such as Google Maps for Mobileâ„¢. The agreement also gives Tele Atlas access to edits for its maps from Google’s community of users, whose suggested changes can help the company further increase the quality and richness of Tele Atlas maps.”

But no single source of data is ever guaranteed to be the most important or relevant source on the Internet forever.

Actually, it’s not inconceivable that OpenStreetMap will be more valuable than NavTeq and Tele Atlas very soon, as it is has the added benefit of being enhanced by humans every day. You can think of it like a Wikipedia for geography.

Plus, it’s free to use.

The business of openness is about being a part of the wider ecosystem in addition to creating your own. When you’re open you can be relevant to people wherever they are on the Internet in addition to when they make the effort to visit you. And when that relationship becomes meaningful the revenue streams will find their way to you, too.

eBay, Salesforce.com, Flickr, and Amazon owe much of their success to their approach to openness. They’ve opened up their platforms in ways that allow other sites and participants in the Internet ecosystem to both take from and add to their services.

If you need clarity on the revenue model, look no further than Google’s AdSense. AdSense accounts for well over 30% of Google’s total revenue. They wrote checks totalling about $3 billion in the first half of 2008 to all the partners who carry the ‘Ads by Google’ ad unit out there. That’s a pretty good incentive to participate in their ecosystem.

With a little creativity, the open strategy will create a lot of opportunity for any media organization. And sometimes I even wonder if it will be the only model that works at some point in the not too distant future.

The opportunity cost of noiselessness

At minute 4:30 in LCD Soundsystem’s ‘Yr City’s A Sucker‘ a tightly bound harmonic ‘Aaah’ and the shout that arises from beneath it speaks volumes about what the media business often means to people.

“What we want is what you want. What you want is a case of the hah hah hah’s.”

It’s like casinos and gamblers, drug dealers and addicts — a sort of degenerative codependence.

The ‘Aaah’ is brief. It sounds nice and pleasant and makes you think everything is happy here, production and consumption joined for the good of all.

Behind it, however, is an angry or perhaps exhausted shout that breaks through just for a moment at the end. It’s the musical representation of The Narrator’s struggle in Fight Club:

“We’re consumers. We are by-products of a lifestyle obsession. Murder, crime, poverty, these things don’t concern me. What concerns me are celebrity magazines, television with 500 channels, some guy’s name on my underwear.”

The media business is not the only space in which creativity wants to escape the dominant and manipulative incumbent power structures. Finance. Healthcare. Government. Law.

Product design is a great example. In a recent TechTalk at The Guardian Matt Webb argued that people want to use physical objects to connect with people. Social networking is a feature that can be unlocked from within everything we touch.

Nearly as interesting as his philosophy and his prototypes is the fact that Matt was able to build these things on his own. With very little cash he circumvented traditional product manufacturing to make these new ideas real.

Matt Webb's Olinda Radio

After his talk one of the staff asked whether the entire last century was a weird anomaly in the history of mankind. He wondered whether the means of production had become so sophisticated and expensive that they ultimately crushed man’s natural tendency and desire to build things. That’s a good question.

Did the machines and systems and processes and subsequently their impact set the bar for success so high that many people stopped trying?

Lucas Gonze explores this thinking in terms of music. Clay Shirky sees similar patterns in television.

You could argue that the second industrial revolution was the beginning of the great numbing of humanity. While we changed the world at unprecedented scales, the consequence was a systematic reduction of the noise that comes from wider participation.

Human messiness was suppressed to pursue perfect design.

But maybe the noise of participation has music within it that our ears don’t yet understand.

Jeremy Keith got me thinking about some of this further in his dConstruct keynote. He talked a bit about evolution, randomness and power. Once a power law is established it puts a large percentage of the participants in the system at a disadvantage.

Those systems undoubtedly created huge opportunities and changed lives dramatically, frequently for the better, but the indirect costs are becoming more clear today.

In another song called ‘Yeah‘ on the LCD Soundsystem album, James Murphy challenges people to wake up. He repeats ‘Yeah’ over and over like a mindless chant. He doesn’t sound interested or excited. He just agrees as if he lost the will to disagree. And then he breaks out of the trance for a moment and says:

“Everybody keeps on listening in. Nobody listening up.
Everybody keeps on talking about it. Nobody’s getting it done.
I’m tired, tired, tired now of listening, listening… knowing that the ship’s gotta run.”

The optimist hopes forces such as the Internet will reengage the world and empower people to do things they felt excluded from doing before, to participate again and to add to the collective experience of humanity.

The pessimist fears that people are being played by markets designed to take from them and that they are failing to recognize the impact of consuming the goods sold to them…that we are complicit in the growth of the forces that hurt us.

Fred Wilson sees opportunity across all markets to use the Internet as a way to empower people again. He writes about the wider strategy in the investments he makes at Union Square Ventures:

“[The Internet] is taking power away from existing large institutions and pushing it out to smaller entities and often all the way to individuals. In the process it is building up new institutions (such as Google), but the net result appears to be a distinct shift of ‘power to the people.'”

In Adam Greenfield’s insightful essay ‘Waiting for the Whirlwind‘ (via Tim O’Reilly) he articulates some of the deeper fears in America that cause and result in the nomination of someone like Sarah Palin to such incredible political heights.

“The things that endear this onetime nowhere-burg mayor to Americans are indicators that a whole lot of people think tomorrow came too soon…Where they were once called to dream and to believe that their best days as a community still lay ahead, [mainstream Americans] are now at war with the future.”

If Palin was a surprise to anyone then they weren’t paying attention when California voted in a seemingly unqualified feel-good entertainer to govern the region….twice. We elected one of them President…twice.

Palin simply reflects a common desire to maintain the co-dependence of our current political structures.

I have to agree with one of the comments posted in response to Mr. Greenfield’s essay where Daniel Erwin challenged people to participate usefully in whatever this trend is about rather than run away and point fingers.

“Don’t be one of those people who tries to stop the train because most of the world is “crazy” and values things you (and I) don’t understand. Just keep doing your work and you’ll do much better than all this fearful complaining.”

The test for the next century may play out in the way we engage our diverse natures to advance as a whole or rather as many parts of a whole. What do we advance toward? I’m not sure it matters…as long as lots of people are participating and that we recognize how to make corrections dynamically along the way.

James McLurkin studies and builds swarming robotics which are inspired by an obsession with insects. Ants, for example, make lots of mistakes, dropping food and stepping on eachother, but they recognize signals around them that give them clues as to what decisions to make and thus accomplish amazing things as a whole.

The mathematical theory behind his swarming robots is derived from research on distributed algorithms — the idea that problem solving can be tackled by different systems on a network.

It’s about zooming out, abstracting beyond our differences so that individual uniqueness becomes a point of leverage for something bigger collectively instead of worrying too much about the future which is guaranteed to be full of errors in judgment no matter what we do.

Imagine if Google chose not to index less popular web sites or perhaps only the left-leaning political sites or maybe only scientific research documents. They could have focused on eliminating the noise entirely.

Instead, a gigantic market grew around Google’s search engine because they delivered on the expectation that everything on the Internet can be found there, no matter how uninteresting or useless or potentially offensive the unique individual parts of the whole collection seemed to be.

Google embraced the noise. They didn’t care to be perfect. And now the opportunity before them seems nearly boundless.

The future for expensive TV is bright

Clay Shirky gave a fantastic keynote at MediaGuardian Edinburgh International Television Festival about how the TV business needs to adapt to the community behaviors happening around their programming online with or without their knowledge and consent.

He was very well received and undoubtedly inspired some new thinking, though I’m sure he wouldn’t have been embraced so wholeheartedly by this crowd had he done the same talk he delivered at Web 2.0 Expo earlier this year. In that talk, he gave a harsh view of the TV landscape and the blinders keeping people in that business from embracing and feeding the ‘cognitive surplus‘ happening out there as a result of the new social dynamics online.

I was able to catch him at a cocktail party later, and we spoke about the power laws that are starting to show cliff-like shapes in the media business. He emphasized that at the peak of the power curve there’s a much higher endpoint than what we’ve seen in the past. And the tail may have gotten longer at the expense of the curve in the middle.

power law long tail
In other words, in order to have sweeping success in the mass market, production is going to cost a lot more in the future. The tail just gets longer and longer. But the people in the middle who have enjoyed healthy margins on the past on things that are somewhat costly and have somewhat interesting customer numbers are getting squeezed from both sides.

I asked if this could actually be an argument supporting what many TV businesses are doing — searching endlesly for the next hit, the mass market play first and foremost. Could the increase in the cost of mass market success make it easier for the leading channels to break away from the pack? Could a leading position today actually be the only way to secure a successful future?

Clay added that the market for dominance in television may only be 2 or 3 national channels. And if those channels don’t spend enough to make truly exceptional programming worthy of appealing to the masses, then they will be competing with the millions upon millions of participants producing material somewhere to the right of the cliff.

This got me thinking about one of the important panels of the day…the News discussion. The panelists there discussed diferent approaches to prime time news shows, among other things. And I couldn’t help but wonder if Michael Rosenblum spoke truth when he predicted the end of the prime time news show:

“The notion of linear television news is antithetical to the web – a distinctly non-linear, VOD environment. The notion of waiting until 6:30PM or 10PM to get the ‘breaking news story’ is simply a non-starter in the web world…technology has now consigned that model to the trash bin. But people still cling to it in the fear that the tidal wave of news and information uncontrolled and unedited is far too overwhelming for the average person.”

That’s not to say that very very difficult and costly journalism wouldn’t come and take its place. I hope it does. And after talking to Clay I feel more confident that the market would in fact reward the producers for exceptional journalism for the mass market. Paying top dollar for important work seems real and justified.

Pivoting on data with Freebase

This screencast from David Huynh describes a data browsing interface he built using Freebase. It’s a fantastic data visualization tool that allows you to pivot and filter results that can then be rendered in lists, on maps or on a timeline. He calls it Parallax:

Stefano Mazzocchi explains the inspiration for the tool:

“David’s idea was to organically combine the ability of faceted browsing to drill down on a set of given items, but then to use the faceted values as the new set of items, thus ’sliding’ the faceted browsing window onto the selected set and make that the new point of view. This would create a way to browse “sideways” from a particular set of items, following items of different type that are connected to the currently browsed ones.”

Very clever.

(A bit more on Freebase here.)

Using fantasy football to drive network effects

Network effects accelerate when services are accessible wherever the user is engaged. That leap has been made in many different contexts in online media from advertising (AdSense) to participation and personal publishing (Flickr and Twitter).

More mainstream publishers got close to this when they began publishing RSS feeds, but the effects of the RSS reading experience don’t come back to the publisher and add value to the wider network like they should.

A click back to the article on the source domain does not improve that article for everyone else who reads it, for example.

It may seem difficult to create network effects around content except in the form of things like reader comments and social bookmarking. But now there are some new ways to create network effects in the publishing business.

Most publishers have found some kind of social tool that makes sense as part of what they offer. It may be a forum, a friends network, or in some cases a game or contest. All those things can capture activity and engage the participants from anywhere on the Internet.

We recently launched a new fantasy football application at The Guardian (when I say ‘football’ I mean ‘soccer’), and we immediately began thinking about where else people might enjoy playing the game. The developers and product manager cranked out a very rudimentary iGoogle Gadget version of the app so that you can stay on top of what’s happening in the game directly from your browser start page.

The gadget is not yet fully functional, but when we start reflecting your activity in the game back to you through the gadget then network effects will be possible. I haven’t been a huge fan of most of the social apps out there, but I can definitely see myself using this one a lot.

In many ways, it also makes me a more frequent user of Google than I already was, but that’s a topic for another post.

At this point in the evolution of the Internet, the online product launch checklist probably dictates that a portable version of a service is a minimum requirement, must-have feature. In that model, the domain can serve as a rules engine, storage and a transaction hub, but the activity of an application needs only a lightweight container and an end-user who’s happy with the experience wherever it may exist.