Data pattern-hunting makes everything boring

Digital platforms have finally become the economic powerhouses many of us whose careers have spanned a few technology waves always knew was possible.

Among other things I’m quite interested in how their use of statistics and cohort analysis is driving conformity in society at scale. Admittedly, this is a subjective chicken and egg question, really, but it seems to me that being unique and different today is harder and riskier than ever before while the rewards for being normal become greater and greater.

There was a weird moment somewhere around peak Simon Cowell when everyone realized we were complicit in pop music’s boringness. His show eliminates outliers to form a cohort of pretty good but mostly forgettable performers. Then we vote for the least worst one.

This is happening everywhere. Cars all look the same. News outlets report the same news. Even fashion brands who trade on being different are literally losing their edge.

Is there a better sign of the times than the full embrace of the color gray?!

Guitar solos, roadsters, columnists and fashion icons reminded us that we can step out of the machine if we want to. Now we don’t even try. Instead we blend everything interesting until the mass has no distinctive color at all.

Normalizing everything didn’t happen because of Internet platforms. It started decades before they were even invented. But the Internet platforms mastered the art and became commercial juggernauts, as a result.

What makes them good at it? It’s too simple to say they’re good at computers. That may be true, but it’s really all about how they manage the information flowing in and out of their computers.

Marc Andreesen was right. Software ate the world.

The people who are looking at that data and making software that feeds the computers are assessing patterns. They’re not looking for exceptional data or data that sits outside the norm. There’s so much data flowing through these computers that they can only handle data that looks the same. They literally throw away unusual signals. They call it noise. Sometimes they call a weird data signal an ‘error’.

It’s much easier to make sense of large clusters of common behaviors and then to focus on those clusters. If you can drop people doing the same thing through a conversion funnel and get transactions at the bottom then you have a business model.

Startups have a lot of pressure to scale quickly, so there’s really no time to waste on the anomalies. They have no incentive for handling unusual activity other than to find a way to shove anomalies or ‘errors’ back through the funnel somehow. They are looking for normal patterns and doing everything they can to make all the data they collect fit in the same bucket.

To be honest, I find it unfair to blame the platforms for this market dynamic.

I remember very clearly the dismissive tones from the non-techies in the late ’90s every time an amazing Internet startup would appear, “Yeah, but nobody’s making money, yet. It’s just hype.” That went on for years.

Then things started to work. But let’s be clear. The platforms weren’t intentionally employing nefarious data manipulation to exploit us. They were optimising ads. That’s all it was in the beginning.

Equally, the dotcom leaders need to acknowledge their moral obligations today. Their businesses have become part of our lives. They sold it that way and we bought it. That deal needs to change now, just like their terms and conditions change all the time.

The problem is that outliers and anomalies are easy to ignore. And until trolling or misinformation or abuse or fraud or whatever else infiltrates and distorts the normal patterns of behavior in large enough quantities these digital spaces that want our time and attention really just don’t care.

It’s not just the bad behavior they don’t care about. They don’t care about good behavior that is non-normal, either.

Journalism is a great example. Performance of news on Facebook declined for several consecutive months in 2017 and instead of looking at how to embrace news they turned it off. Large numbers of people valued getting news via Facebook, but the nuances of trust, a very human value, by the way, were considered too hard to address.

I refuse to believe that these non-normal patterns are too expensive to identify and serve in a sensible way. The same machines and statistics that are so good at finding normal patterns are just as good at finding things that are not normal. But rather than build the tooling and reporting and insights that value the non-normal they build error handling systems and quantify them with negative terms.

Can a digital business succeed by serving the outliers? Of course they can! It’s crazy to think they don’t have the creativity, manpower or computing resources to value the things that are different. They choose not to because it requires some thinking. It’s easier to encourage the outliers to behave like everything else and just ignore those that don’t.

There’s another question about whether this trend is bad. It would be easy to argue that more people are more educated, safer, suffering less than in pre-platform history. They bring people together and create shared understanding which probably increases peace in the world on balance.

But let’s be honest about the cost of normalizing everything and failing to value the outliers in society at large. If we want magical music like Stevie Ray Vaughan’s version of “Little Wing” and beautiful cars like the 1962 Aston Martin and intelligent perspective like Edward R. Murrow commentary then we owe it to ourselves to ensure the outliers have room to explore and push the boundaries.

The value of a cohort should not be measured exclusively by its relative proximity to the mean. We’ll keep losing the good stuff in this world if we do that.

Now that software has won and platforms drive the economy (and lots of other things, too) they must look at their role in the world with a wider field of vision. They need to be serious about diversity from the boardroom all the way down to the simplest line of code.