The Attention Index — An open source algorithm for ranking premium media

At Kaleida we’ve been developing a new method for measuring media — an algorithm that values quality and impact. We’re opening up the project with a Creative Commons license today so we can start collaborating with other orgs interested in using the data.

The Attention Index algorithm considers editorial decisions and social activity to derive attention scores.

The idea for Kaleida when we started the company a year ago was to reinvent the way media is measured.

The media ecosystem has been defined by reach, impressions, and clicks for about 20 years. From our experience much of the value that goes into producing quality journalism gets lost by those types of metrics, and the effect has been disastrous for the industry.

Yet it seems completely counterintuitive that media orgs would struggle given the massive opportunity the Internet offers. Over 3.5B people are connected now. Fast mobile networks reach 84% of the global population. And digital ad spending is expected to reach $224 billion globally in 2017.

If quality journalism outlets could demonstrate how much impact they have in the world then lucrative business models would surely find them.

So, we collected tons of data about the media and began analysing it. You can see our machine doing its magic throughout the day every day on kaleida.com, creating a map of interests in the world. The data shows what matters to whom, when, where and how much. That was the first phase of our plan to reinvent media measurement.

The second phase is beginning to unfold now — The Attention Index.

The Attention Index puts all this activity into context. It scores and ranks behaviours in relative terms so it’s easy to compare what’s working vs what’s not working. For example, it considers whether a publisher thinks something is particularly important and uses reader response to see if the public agrees.

https://kaleida.github.io/attention-index/

We’ve started publishing raw data for people to download and explore. The May 2017 CSV includes 128,000 article headlines, URLs, editorial promotion data, social data and attention scores. Every month we’ll publish the data we’ve collected and scored with the algorithm.

The math and methodology is public, too. Kaleida CTO Graham Tackley has documented step-by-step how he applied various statistics and arrived at these results.

And, of course, everyone is invited to join the community and to contribute. The Attention Index is published with a Creative Commons license with support from Google’s DNI fund.

The Attention Index algo — Full exploration and sample data is available via a Jupyter Notebook.

There’s a lot more to do. We started by sharing our work with several media academics in the US and the UK. The data and the algorithm will require more scrutiny, more contributors, more data and more use cases.

Opening it up for public use will help us get that kind of engagement and, hopefully, make the Attention Index an industry standard.


Everyone we talk to in the media business is frustrated with many of the same things. Fighting to lead on metrics you don’t believe in and that don’t really serve your business goals is right near the top of the list.

The lower-end commodity metrics will always be useful, but it’s time to recognize things that matter and to talk about them with shared language.

Keep an eye on the project and be sure to get in touch if you want to get involved or if you have questions about it.

UK election attention was far stronger on the left than the right

The data suggests mainstream and alt-media coverage and the amount of sharing of that coverage on Facebook may have played a role in the outcome.

We’re finding some fascinating connections between media attention and realworld events. That doesn’t mean one causes the other, necessarily, but sometimes it might.

Last week’s UK election outcome was yet another example.

Source: Kaleida

Early campaign reports from Conservative-led voices that Theresa May would win a landslide victory were not apparent in our data. Actually, the race looked very close in terms of media attention.

After an initial focus on May when she announced the snap election in mid-April the amount of coverage and the volume of social activity in response to that coverage appeared to even out. Some days Corbyn took the lead. Some days May was winning.

The interesting trend we found in our data was the scale of attention across the political spectrum. Attention tipped to the left far more than the right.

Source: Kaleida Data, 2017

In some research we did with The Guardian we scanned a selection of alt-media sources and compared how well they performed against mainstream publishers. Articles posted by activist blogs like Another Angry Voice often outperformed UK election coverage by the BBC, The Guardian, The Telegraph and even the big US publishers like NYT, CNN and Fox News.

They must have had an impact. But there’s another interesting story in the data.

The Independent and The Guardian influenced significantly more attention than The Telegraph, the Daily Mail and The Times. They even beat out the BBC, Buzzfeed and HuffPost. It was the left-leaning mainstream national news orgs that produced the most articles. And it was those articles that received the most shares on Facebook compared to coverage by right-leaning publishers.

Similarly, the left-leaning alt-media sources dominated the right. Westmonster, Breitbart, Guido Fawkes and Your News Wire were not as successful in their UK election coverage compared to their left-leaning counterparts, by these measures.

Now, we didn’t make a prediction prior to the UK election. We learned from our French election predictions that we needed more data, particularly when a race is close. In that case we overestimated the power of English-language media attention on the outcome of the French vote.

In the case of the UK election we not only needed more alt-media sources to measure attention, we knew nuance in how we measure attention was required, too. There are things that need to be taken into account such as the impact of a publisher’s coverage in proportion to the size of their readership and the number of likes they have on Facebook.

If you’re interested in this data we will make it all public soon, or you can get in touch. We’re happy to share it.

The impact of alt-media on elections

In a collaboration with The Guardian published today we looked at several alt-media sources that are having an impact on the UK’s general election debate.


We found some remarkable figures. Many of the articles published by individuals with strong left or right political views are being shared considerably more than articles by the biggest national news orgs in the UK, including The Guardian, BBC, and The Independent. Sources ranged across the spectrum from The Canary and Skwawkbox to Westmonster and Breitbart.

Source: Kaleida Data, 2017

Another Angry Voice is a Yorkshire-based site published by Thomas Clark. He holds 3 of the Top 10 most-shared articles about the election. The Independent and indy100 perform well compared to others, but most of their most successful articles have earned less than half of Clark’s top-performing post at over 100,000 shares.

Guardian journalist Robert Booth spoke to the university student behind Evolve Politics, Matthew Turner. He captured what we’re seeing brilliantly,

“This kind of dynamic activism is new to the media,” Turner said. “Stories that go viral are stories that you can rally around. I think the vast majority of readers like us because we light a fire in their belly.

Readers of the mainstream media tend to not get that nowadays. We are the ones offering the fight.”

That sentiment was echoed by both the left and the right alt-media sources.

The data behind The Guardian’s report can be found in our UK election coverage explorer. If you are interested in a full set of results to explore on your own, please get in touch. We would be happy to share more data.

How news orgs manage their home pages

Professional news orgs have a lot of insight about what’s important and what’s not. Sometimes it goes unnoticed with all the noise in today’s crowded media market, but the signals are there in the data for us to see if we look in the right places.

Smaller publishers including fake news sources have learned how to make web sites that look as good as larger professional web sites. As a result much of the work that goes into producing important journalism and providing it to people goes unnoticed.

So, how do we know the difference? How does anyone know that a large group of hardworking and intelligent media professionals have carefully crafted an important story and that they think it matters to the public?

One signal is the way stories get prioritized.

Source: Kaleida Data, 2017

Kaleida CTO Graham Tackley looked into the data and found consistently that about 2% of all the articles from the publishers we track make it to the lead position on their home pages. If 98% of their output doesn’t get the top featured slot then these featured pieces obviously matter a lot.

We didn’t see consistent behaviour around how long featured stories remain there. The majority of Fox News’ featured stories hold that position for a few hours. The New York Times varies from story to story, but none last in the featured slot for more than 24 hours.

The publishing strategy for what else goes on the home page varies even more. Stories can last somewhere on the home page for minutes or days. The mean average for a story on the home page is 12 hours, about 5% last on the home page for up to 2 days.

The Daily Mail, for example, has two parallel publishing strategies. The main news column is rotated every 24 hours, while the long right-hand column moves at a much slower pace. Nothing on the Buzzfeed home page lasts more than 24 hours, most cycling through in about 4 to 6 hours.

There are many other channels such as Facebook where publishers prioritise their stories. Sometimes stories are published there without offering an equivalent piece on their own web site. We also know from looking at each publisher’s Google sitemap file that there are many stories that never make it to the home page or any of the social channels.

While we don’t yet know the impact of these various strategies on each publisher’s business we can use this data to build a clearer picture of what matters from their perspective.

The data gets particularly interesting when you look at it in aggregate.

If all the people in the news business from both left and right-leaning politics think something matters then that’s worth knowing. That’s a very loud signal. The decision not to feature a story can be equally loud.

Of course, identifying what matters is a two-way street. Just because an editor thinks something matters doesn’t mean readers always feel the same way. But it’s our belief that professional news orgs provide more value than most of us realise. It’s harder to see it with all the noise out there now, but the amount of work that goes into the editorial decisions made every day can be useful in setting some standards of value across the media ecosystem.

Maybe the business model problem for media has more to do with the process of media production than its distribution. Maybe the two-way street is precisely where the value lies.

More on that soon.

How much is a Facebook partnership worth?

At Kaleida we found correlations that prove sharing patterns on Facebook are a good proxy for referral traffic. That means publishers can work out the real value of all the effort they put into doing things with (and for) the company.

News orgs have struggled with platforms for a long time. They don’t know how to find a fair working relationship because it’s hard to quantify the benefits for each side. What is the unit of value to ground the negotiation?

Facebook, Google, Twitter and all the rest are always selling the idea that they drive a lot of traffic to publishers’ web sites and, therefore, publishers can’t afford not to work with them. If that’s the starting point then there’s a very important question to ask:

How much is that traffic worth? And how do you know?

This has been a particularly slippery question when it comes to Facebook, but it doesn’t have to be.

Kaleida CTO Graham Tackley decided it was time to find some answers. What he found confirmed his suspicions that organic shares on Facebook are a reasonable proxy for real web site traffic.

The sample data here is pretty small, but it’s enough to get started. It includes one day of data from a publisher we work with who will remain anonymous. We have total page views and referral data for hundreds of articles published in early April, 2017 by “Publisher X”.


In this chart you can see a band of dots suggesting a correlation between organic sharing and Facebook referral traffic. The Pearson correlation coefficient is 0.86.

To put that in context, a perfect correlation is 1.0, meaning every point is an exact 1-to-1 relationship between the x and the y axis. A correlation of 0.0 means there is no relationship, that the figures are all totally random. So, a correlation of 0.86 is actually pretty strong, good enough to confirm the idea that Facebook sharing patterns are a reasonable proxy for Facebook referral traffic patterns on a web site.

The correlation between organic sharing and total web site traffic is much lower at 0.65. Facebook is not the sole source of traffic, of course. But neither is Google which scores 0.89. Google scores higher overall because Google provides a more even spread of traffic to publishers’ web sites, driving people to home pages and the like in addition to articles.

Interestingly, while Google traffic may be more reliable and evenly spread Facebook pushed more referrals to article pages for Publisher X than Google did.


So, if there’s a strong correlation between sharing and traffic then we can work out how much value Facebook adds to the bottom line. Crucially, we can trust that the number will convert into real income unlike some of the things Facebook wants publishers to do.

What is a ‘share’ on Facebook worth?

This data can help us paint that picture. First, let’s improve the precision of the answer by focusing on the higher value data.

We know, for example, that most articles get very few shares, less than 100. Let’s throw those out of this sample, because those shares aren’t really helping either the publisher or Facebook. Equally, the 90th percentile have gone viral or they got a huge boost in some unpredictable way. Facebook can’t guarantee those will happen. According to Kurt Gessler of Chicago Tribune, the algorithm may not surface a third of all brand page posts anyhow. So, let’s throw those out, too.

That leaves us with articles that have earned between 80 and 10,000 organic shares. Publisher X here had 1.1M Facebook referrals for that sample from over 240K shares.

So, in this case, one share on Facebook counts for about 4.5 page views.

If we use a generous average eCPM of $10 to make the math easier then it’s reasonable to say Facebook referrals have driven about $10k worth of advertising revenue opportunity that day for Publisher X.

Revenue divided by transactions = unit value. So, how much is a share on Facebook worth?

An organic share on Facebook is worth about $0.04 to the publisher.

If that was a typical day for this publisher and all the other assumptions are in the right ballpark then Facebook referrals probably drive between 3% and 5% of total digital revenues for Publisher X.

It’s true that being present on a platform has value to it far beyond referral traffic. It’s also true that publishers could be wiser about their publishing strategies which would improve that figure.

But to accept the sales pitch from the platforms without also knowing where the tangible value lies and how much is on the table is a mistake nobody should make. Strategic value is really important, but given the internal cost of many publishers’ Facebook initiatives, including Publisher X, it’s likely that much of the strategic value in partnering is being lost. And as digital ad revenue market share moves away from news orgs this trade off feels less and less compelling.

Now that we know sharing and referral traffic have a significant and reliable relationship those discussions should feel different. Negotiating that partnership should start with a number: $0.04.


Facebook share value will vary per publisher based on total referrals, shares and CPM. We could use more data from more publishers to refine these figures, so if you want us to do this analysis for you, please get in touch: matt@kaleida.com and tackers@kaleida.com.

Macron’s attention spike just prior to winning the election

On Thursday the attention levels for Macron and Le Pen were neck and neck. But then attention swung dramatically in favour of Macron, mostly because everyone was talking about the email hack. The next day he won the election.

Source: Kaleida Data, 2017

In this chart we’ve multiplied the number of articles about Macron and Le Pen published by major media sources in the US and the UK by the number of organic shares the articles earned on Facebook.

The major peak in the chart happened around the final debate. Attention for both fell before Macron got the big boost that put him in the lead at the end.

The remarkable insight here is that yet again we find the combination of media coverage and the response to that coverage reflecting real world outcomes. Correlations are not the same as causation, but we are finding more and more reasons to pay close attention to this relationship.

Macron’s lead is narrower than the polls suggest

If attention matters in the French election as much as it did for Trump, Brexit and Round One then the polls need to consider how people are responding to coverage of the candidates.

Source: Kaleida Data, 2017

Source: FT Research, 2017

The polls currently show Emmanuel Macron leading Marine Le Pen in the French Presidential Election. The FT has a nice poll tracker mapping a steady position of voting intentions for the more centrist candidate.

Le Pen has closed in noticeably but not threateningly.

The polls were were strikingly accurate in Round One, so there’s no reason to believe that they will be wrong now. However, recent history in other parts of the world suggest people’s voting intentions can’t always be trusted on the big day.

While projecting American and British experiences onto the French electorate is bound to be inaccurate, the data tells a story that should concern anyone who expects Macron to take 60% of the vote. The polls aren’t considering the impact of the media, the attention Le Pen is capturing and the momentum moving in her direction these last few days.

Monitoring attention data to predict the French Election

Attention density favoured Le Pen up until the first round of voting in the French Election. Now Macron has momentum with the media and, therefore, a good chance of winning.

Source: Kaleida Data, 2017

Up until the weekend our data suggested that Le Pen’s lead in the French Election was insurmountable.

The pattern looked similar to what we’ve seen elsewhere this year — as if Le Pen had acquired a critical mass of attention density, a position where a sort of natural gravitational force pulls all media in around something.

Attention from mainstream media alone is not enough to create this force. Lots of things get heavy coverage and no response. And neither is viral distribution of things people say on Facebook. A viral story may have no value to it other than its own reflection.

Rather it’s the intersection of those two things that creates a sense of density and weight and real world impact.

Le Pen had that kind of gravity, even after a tumultuous February that included a ‘fake jobs’ scandal, a fraud probe, and a police raid. *

Since the beginning of 2017 the leading news orgs have been covering Marine Le Pen a lot more than all the other candidates. 40% have been about her, 30% about François Fillon and 21% about Emmanuel Macron with the rest spread out amongst the others.

With so much media attention focused on Le Pen she is obviously going to dominate how coverage about the candidates is shared on Facebook. In fact, prior to Round One articles about Le Pen earned 70% of all shares of coverage about the candidates with 26% split between Fillon and Macron.


In mid-April that attention market became destabilised. It was Jean-Luc Mélenchon who really changed things. It seemed to come from a big rally in Lille on the 12th.

Stories began resonating in the market about his increasing fame, his plans for taxing the rich, pumping €100 billion into France’s green economy, and his euroscepticism. Suddenly, shares of coverage about Mélenchon actually eclipsed Le Pen for the first time since we began tracking coverage of the candidates.

The whole thing seemed up for grabs.

Now that we’re down to two candidates things really have changed. And Macron is actually leading Le Pen in terms of attention. We’re tracking nearly 200 articles about Macron over the last 24 hours which have earned over 150k shares on Facebook. Le Pen no longer has the same attention density with just 150 articles in the same period and about 140k shares on Facebook.

Both candidates are probably outperforming expectations given neither represents the mainstream political parties, and it’s plausible that Macron is outperforming more than Le Pen given the many years she has already spent in the media spotlight.

Based purely on attention density as we’ve described here, Kaleida data now suggests that Macron is leading the race. A lot of news and sharing of that coverage will happen between now and election day. If the past is an indicator of what happens next then the daily shifts in attention will surely affect voting.

The side that can build the most coverage by mainstream media that is shared the most on Facebook is likely to come out on top.

* The significant caveat in what the data tells us here is that Kaleida is looking at English-language articles from major news orgs in the US and UK only. Without tracking coverage of the French media market interpretations of data about French issues have to be taken with many grains of salt.

Why Kaleida’s data says Le Pen will win


If sharing of media coverage is in fact an indicator of real effects in the real world then either Mélenchon or Macron will face Le Pen in the 2nd round. And they will lose.

We noticed interesting patterns in our data after Brexit that were reflected again in the US election. It’s not media coverage that affects change. And it’s not what people say on Facebook. It’s the intersection of the two — how people respond to media coverage.

We saw similar correlations when we looked at coverage of public companies and changes in stock price, though, of course, there is a long list of caveats propping up this idea still. The biggest caveat in terms of the French Election is that we’re not tracking French news sources.

But the data tells a fascinating story even though the science is not yet fully formed.


Since the beginning of 2017 the leading US and UK news orgs have been covering Marine Le Pen the most. 636 articles (40%) have been about her, 470 articles (30%) about François Fillon and 330 (21%) about Emmanuel Macron.

With so much media attention focused on Le Pen she is obviously going to dominate how coverage about the candidates is shared on Facebook. In fact, articles about Le Pen have earned 70% of all shares of coverage about the candidates with 26% split between Fillon and Macron.

The crazy February spike that included a ‘Fake Jobs’ scandal, a fraud probe, and police raiding the Front National offices increased the volume for all values but didn’t seem to skew the percentages all that much.


However, there are two signals that indicate a window is open for a challenger. The rate of sharing and sentiment both put a qualitative lens on these numbers.

Stories about Jean-Luc Mélenchon have recently been shared at a significantly higher rate than all the other candidates combined. He’s only received about 6% of all the coverage from the US and UK media, but those stories have travelled very successfully. At 1,200 shares per article on average vs 730 for all stories about the candidates the Mélenchon stories have been performing 50% better than Le Pen’s coverage since the beginning of March.

Another interesting fact: Mélenchon coverage has been promoted on the front pages of those publishers’ web sites only about 1,000 hours vs Le Pen’s 13,500 hours. In other words, readers have disproportionately embraced Mélenchon vs the amount of coverage and the effort used to promote that coverage.

In contrast, Fillon has been on a steady decline since February in all the areas we track. The amount of negative coverage about Fillon is striking, and the more that coverage gets shared the worse Fillon’s position becomes.



Le Pen, on the other hand, is getting more and more coverage with slightly more positive than negative sentiment.

Clearly, Le Pen is winning in the US and UK media markets for reasons that have nothing to do with the French.

Mainstream media is stuck between voicing the concerns of the people and amplifying the messages of the populists. That and the addictive nature of web site traffic that populist voices hand to them on a plate make it hard for publishers to resist being a player in the game. Le Pen is Brexit. Le Pen is Trump.

It’s as if a formula has emerged or a kind of perfect storm.

There are indicators that people see through this now. Macron and Mélenchon have both been climbing in recent weeks at astonishing rates. Shares of coverage about Mélenchon actually eclipsed Le Pen for the first time since we began tracking coverage of the candidates.

If the patterns we’re tracking are in fact interrelated with real world events then those trends are too little too late. The volume of both the coverage and the amount of sharing of that coverage by readers supports a win by Le Pen no matter who challenges her in the 2nd round. By these measures Mélenchon is the only one who stands a chance.

The size of United’s PR crisis

The negative response to the latest blunder at the airline is up there with some of the biggest stories of the year. It’s not impossible to fix this, but they’ve got a tremendous hill to climb now.

Late last week I flew United from San Francisco to London. As we got on the plane one of the other passengers gave a box of chocolates to the flight staff as an act of kindness and support for the awful week they’ve had.

The staff seemed to appreciate it. Everyone who works there is truly in the media spotlight. This level of attention is not what members of the United staff who didn’t forcibly remove a passenger asked for.

It may not be totally surprising that it happened. I can imagine what it felt like on that plane that day from many years flying with them. What’s really surprising is the scale of attention this story has achieved.


Nearly 300 stories have been published by the leading news orgs in the US and UK about the incident. Unsurprisingly, our analysis shows sentiment of the coverage is almost exclusively negative.

Source: Kaleida Data, April 2017

Those stories have earned nearly 1M shares on Facebook in aggregate. That’s a lot of unwanted exposure for the company. It surpassed United’s trouble with leggings from the previous week about four times over.

By comparison, coverage of Trump’s Muslim travel ban in early February ultimately earned over 7 million shares. This story isn’t at that level, but it ranks among the biggest this year, so far.

Fox News was first to publish a story that day. One of the passengers who recorded the incident posted the video via Twitter and mentioned @FoxNews, @CNN and local ABC affiliate @WHAS11 to get their attention. It worked. Though being first didn’t seem to give Fox the edge, as several others including CNN and The Huffington Post achieved more total shares for their articles.

The biggest story was the Washington Post’s. They promoted it immediately via their Facebook page and within an hour it was getting over 30 shares per minute. After that it rolled on at a steady pace of at least 20 shares per minute for about two days. The article is up to about 65k shares in total now.

Washington Post published about 50 articles on the topic. Fox News has done about 35. CNN and HuffPo have published about 20 each.

Articles about the incident earned about 3.1k shares on average vs a recent average of about 2k shares per article for all articles by all the publishers we track.

It was a big story for everyone except the Daily Mail. They published 30 stories and only earned about 1k shares in total for all that work.

Coverage of United Airlines is shared mostly by young, educated women with left-leaning politics. That demographic may not be the same one trading their stock, but the message is clear — shareholders didn’t like it.


United Airlines posted a comment from CEO Oscar Munoz to their Facebook page a few hours later to soften the blow. Over 100k comments followed, and 140k shares. 90% of the reactions to that post are angry faces.

Suffice to say that didn’t help matters. Maybe it made things worse.

This might be one of those death spiral moments where a company’s PR efforts only amplify their troubles. As if things couldn’t get any worse (though one should never say that about airplanes) in a separate incident a United passenger apparently got stung by a scorpion on a plane.

United would be smart to use this moment to do something really bold. Free tickets for passengers on flight #3411 and apologizing on morning television isn’t going to cut it.

It seems they need a cultural change at the company. If they start by valuing their customers more that may inspire them to do things really differently.

Maybe they should get political. Everything is political in 2017.

Reversing the tide of negative stories getting shared on social media means filling the Internet with positive ones about something people care about. How about randomly offering doctors, women and Muslim passengers free flights, upgrades and airport lounge access? Doing that every day would chip away at the problem in front of them externally and change the culture internally.

They can’t swing the conversation back with one story, but if our hypothesis at Kaleida is right, that sharing of coverage about a subject has a real impact on that subject, then United has a lot of media to generate that people care enough about to share on Facebook. They better get to work soon.