Posts Tagged ‘metrics that matter’

It’s easy to get excited about metrics, measuring content effectively, and the related tools and features that accompany these advancements in thinking and strategy. That said, it’s always nice to take a step back and think about how journalism in the context of analytics is evolving in a broader sense. Tow Fellow Caitlin Petre frames some interesting ideas about quantifying journalism in her blog post for the Tow Center blog, and we’re reposting it here for you all to enjoy. Throw your ideas in the Comments section – we’d love to hear what you’re thinking.

This post originally appeared on the Tow Center for Digital Journalism Blog on October 30th, 2013.

Journalists are seeing an explosion of quantitative data about how readers interact with what they write. To date, much of the conversation about metrics and news has focused on the dangers of using metrics to guide news judgment or, on the other hand, the risks of ignoring metrics completely. But crucial empirical questions about how metrics are produced and put to use remain largely unanswered. How do analytics firms measure complex qualities like engagement, make predictions about the future performance of content, and communicate with journalists about the value of metrics? And how do journalists at different types of news organizations use analytics in their day-to-day work? Are increasingly sophisticated measures of stories’ performance shaping journalists’ ideas about what is important, interesting, or newsworthy? Has the availability of such data changed the internal dynamics of news organizations?

These are some of the questions I aim to tackle in my Tow project, using qualitative methods, such as ethnographic observation and in-depth interviews, to better understand the development and use of metrics in an analytics firm and two news organizations. But before I can answer these questions, I have to ask a different one – one that is as dreaded in my field (sociology) as it is common: What is this a case of? Even though researchers have a tendency to become infatuated with the most minute details of our subjects, what we’re ultimately trying to do is identify and account for patterns in the social world. The classic “what is this a case of?” question prods us to zoom out, to put things in context, to consider the broader implications of whatever it is we’re studying.

So, what are metrics a case of? While there is a burgeoning movement to measure journalism – especially non-profit and investigative pieces – more qualitatively, most metrics categorize and count – page views, unique visitors, time on page, drop-off rate – with the aim of comparing things: pieces of content, news organizations, authors, readers. Metrics, and the big data trend of which they are a part, represent what philosopher Ian Hacking calls “an avalanche of numbers” made possible by astonishing advances in the ability of computers to collect, store, and process huge amounts of data. And metrics aren’t the only numbers in the avalanche – more and more journalists are now deriving stories from their analysis of quantitative datasets.

Sociologists Wendy Espeland and Mitchell Stevens have argued that quantification, like speech, is “a social action that…can have many purposes and meanings” that arise and shift through use. Scholars who study quantification have sought to uncover these purposes, meanings, and uses: here are two ideas from their research that can guide our thinking about the role of numbers in the production of contemporary news. The first concerns metrics, the second concerns the growing prevalence of data journalism.

  • Numbers can discipline, even in cases where they aren’t intended to. Michel Foucault famously argued that statistical measures have been used in attempts to control and “normalize” populations that were considered deviant. But sometimes numbers that were meant merely to measure inadvertently serve a disciplining function. U.S. News and World Report’s law school rankings were designed to provide prospective students better information about schools they were considering, but Wendy Espeland and Michael Sauder discovered that law school personnel internalized these measures, and began to change their admissions and financial aid policies to better conform to them. In other cases, however, the implementation of quantitative accountability measures faces considerable resistance and resentment, such as in Tim Hallett’s ethnographic study of the faculty at an urban elementary school. My preliminary research (as well as work by C.W. AndersonPablo Boczkowski, and others) suggests that both internalization and resistance are present in journalists’ response to metrics.
  • Numbers can establish trust where it is lacking. Historian Theodore Porter has argued that because of numbers’ longstanding association with rationality and objectivity, quantification can be a useful “strategy for overcoming…distrust,” especially in professional fields that are susceptible to outside criticism. At a time when public trust in journalism has dropped precipitously, then, we might expect the standards of journalistic evidence to become increasingly quantitative. As Tim Berners-Lee puts it in The Data Journalism Handbook, “it used to be that you would get stories by chatting to people in bars, and it still might be that you’ll do it that way sometimes. But now it’s also going to be about poring over data and equipping yourself with the tools to analyze it and picking out what’s interesting.”

Contemplating the increasingly important role of quantitative data in journalism also leads us to interesting questions about the idea of “objectivity,” and specifically about the relationship between scientific and journalistic definitions of this term: How do they overlap? Where do they conflict? If we are indeed seeing a quantitative turn in journalism, will it push these two conceptions of objectivity to be reconciled? These are questions I’ll address in future posts.

Caitlin Petre is a Tow Fellow working on a project on Metrics:Production and Consumption for the Tow Center for Digital Journalism.  The Metrics:Production and Consumption project is made possible by generous funding from both The Tow Foundation and the John S. and James L. Knight Foundation.  To learn more about the Tow Center Fellowship Program, please contact the Tow Center’s Research Director Taylor

salon logo

“It’s been the feeling that following metrics too closely is corruptive to good quality journalism,” Haile said. “I think if you’re following the wrong metrics that’s true.”

Salon has a big story today about Chartbeat and our CEO Tony Haile. Writer Alex Halperin discusses online journalism in the era of listicles, click-based metrics, and frequent debate regarding what topics merit coverage, and how Chartbeat may continue to shake things up in the industry – for the better, we think(!). The article features real talk with Tony about measuring content quality and value through audience engagement, click-bait journalism, and where he thinks online publishing is heading.

Here’s the whole article and enjoy the excerpt below. If you have questions or comments, tweet at Tony – he’d love to hear from you.

But as Haile presents it, Chartbeat wants to change the data editors and, more importantly, advertisers care about. He thinks this could improve journalism’s quality by reducing the incentive to write click-bait headlines, produce unnecessary slideshows, pointlessly paginate articles and indulge in other chicanery to inflate page views.

Raising page views for its own sake, “Doesn’t help the audience,” Haile said. “The advertiser doesn’t get anything more from it. It’s just a way of gaming the numbers.”

“If [a headline reads] ‘Prince William caught in love triangle,’ it doesn’t matter what the story says,” Haile said. “I’ve got that click, I’ve got that page view. So it lends itself to lower quality.” But in a media climate where every post is judged on its own terms — whether it’s a war zone dispatch or a curated list of tweets about “Mad Men” – how can quality be measured?

Haile thinks the crucial metric should be time, how long a page captures readers’ attention. He believes that articles that engage readers, and are therefore more likely to create a loyal audience, should be worth more to advertisers. That might sound simple, but almost two decades into the era of online media, the industry hasn’t been able to make that happen.

– From “This man decides what you read”, Salon

I wish every article contained the epic Oliver Stone-like phrase “This is the Chartbeat era”, as author Josh Sternberg utters in this Digiday piece that went live yesterday. But that shield-thumping sentence aside, this Digiday piece gets into interesting new territory in its dissection of  publishers embracing post-pageview mentalities when it comes to paying writers.

We talk a lot about the need for new, quality-focused metrics here at Chartbeat. Our CEO Tony talks about it. So does Josh, our data scientist. But it’s always great to get new voices into that conversation – voices that pose new questions or present distinct angles as Digiday does here.

Digiday’s examination of the new ways publishers are measuring editorial success and rewarding strong journalistic performance reveals the exciting direction in which many top sites are heading: “Publications like Complex and Forbes are trying to come up with performance criteria for writers that guard against abuses yet still reward writers for attracting audiences and moving the business forward.”

Of course, Complex and Forbes are by no means alone in their quest for better metrics. And while I think we’re asking the right questions, we’re still a ways away from having all the answers we need. We think our Engaged Time metric starts to push us in the right direction in measuring the un-gameable attention of a site’s audience, but we all know there’s no single silver-bullet metric that will solve all our problems — so we’ll keep working until we’ve found all the right ones.

In the meantime, it’s cool to see topics being explored more in depth. Hope you’ll check out the article and let us know what you think.


Page views are a pain in the ass. It’s the metric that lies behind so many of the bad design decisions on the web. Time on page is too. The way it’s been measured has often been more akin to a finger in the air than the solid metrics we expect. Neither really take into consideration how users are actually viewing information – from social to mobile – or at least not well.

As Lewis DVorkin states in his latest Forbes piece, “the page view engagement metric of the last decade served its purpose.” He’s right. It did. But now, RIP Page Views. It’s time for us to move on and start using metrics that truly matter to your site’s performance – those are the metrics that tell the full story of your content.

That full story is not told in a page view. Sure, it might tell you that you’re awesome at driving traffic to your site, but beyond that…? What kind of traffic? The right kind, the kind what will come back, the kind that will genuinely engage, share, and chat about you?

We, the Chartbeat crew, are pretty certain the future lies in engagement. Engagement – or specifically a new metric we’re rolling out called engaged minutes – is all about how and how long someone is actually, actively engaged with your content. It’s about knowing precisely what that user is engaging with and how. For instance, is this page linkbait, driving clicks but no engagement? Or is this piece of content a slow burner that’s hidden behind a bad headline, so readers have a tough time finding it, but when they do, they love and engage with it? If so, it might be time to change how you position and promote those stories as a result.

And that’s the point, right? To take action and make decisions. That’s why you’re using real-time data, to adapt and react to who is engaging with your content and how, whether those actions are immediate or longer-term. And you can’t take meaningful action on page views.

So, please pack up your things, page views. Engaged minutes is on its way to usurp your throne.

When people initially think about “real-time analytics” they often think of it as “like traditional analytics, only faster.” But what’s often missed is the transformative way chartbeat measures visitor activity: it gives you the data you need faster, but it also makes that data much richer.

With a traditional service like Google Analytics, a user sends a “ping” to the service when they first arrive on the page, and, sometimes, whey they click on links. From this data, it’s possible to count the number of page views in a period of time. But measures of how long users remain on the page are just guesses.

In contrast, with chartbeat, one user sends repeated pings, as frequently as every 15 seconds, to say “I’m still here, and this is what I’m doing.” Are they actively reading your site, or do they have it open in a browser tab for later? Are they writing something, maybe a comment or search? Are they scrolling down and engaging with the content or not getting much further than the headline?

The chartbeat method of counting provides a richer sense of how many qualified and retained views you’re getting, rather just raw visits to your web site. Our data gives a more accurate accounting of time on page and a better sense of how much of that time was actually spent looking at and interacting with your page. This is information that that can be valuable in optimizing your website, improving your marketing, or selling more advertising.

Why your chartbeat numbers don’t match your traditional analytics numbers

All analytics services measure data in different ways and reconciling them can be difficult, but doing so can be even more difficult when trying to reconcile real-time analytics with traditional analytics. Knowing how many users are on your site at any given time is qualitatively different from knowing how many page loads occurred during a given period of time. The same 5 page views in an hour could be 1 concurrent view if users do not remain engaged and leave your site, or 5 concurrent views if those users remain engaged and stick around.

This is illustrated in the diagram above. While traditional measurement only knows that there were 5 visits between 10:00 and 11:00, chartbeat knows which sessions are active at any given moment and what they’re doing. When you load the dashboard or view your history, you see slices in time, some with 2 users present at once (10:05), some with 4 users present (10:00).

So what’s better, page views or concurrent views? We think they both have their place. But it’s important to understand that these are two very different perspectives on visitor behavior, and there can be valuable insight in the differences. If chartbeat shows a larger percent of traffic on some page than Google does, that page may be holding users’ attention for longer. If chartbeat shows more traffic coming from direct than search, it may mean that users coming via Twitter are more likely to linger than hits from web search. Viewing this behavior from the angles of both traditional and real-time analytics will give you far greater insight into what’s really happening on your site and how to improve it than any one tool alone.