Archive for the ‘Data Science’ Category

Traffic During the Facebook Outage

August 4th, 2014 by Josh

As you’ve all but certainly heard, Facebook had a major outage midday on Friday. Overall traffic on news sites dropped by 3%, thousands took to Twitter to voice their frustration, and, apparently, a select few called the LA Sheriff's Department. Most interestingly for us, the Facebook outage provided a natural experiment to look at what the world of web traffic looks like without Facebook. Here, I’ll delve into two issues that are particularly interesting to look at through the lens of the outage.

Facebook and dark social

So-called “dark social” traffic — traffic to articles that lacks a referrer because it comes via HTTPS or apps — is subject to endless speculation. What portion of it comes from emailed links? From links sent via instant messaging? From standard social sources like Facebook and Twitter but with the referrer obscured? From search sites that use HTTPS? By virtue of the fact that no explicit referrer is sent, it’s impossible to tell for sure. Since Facebook makes up a huge portion of non-dark traffic, one might guess that a big chunk of dark traffic is actually Facebook traffic in disguise.

Of course, during the outage virtually all Facebook traffic was stopped, so we can use that data to ask how much dark traffic was definitely not coming from Facebook. The answer? Very little of it was coming from Facebook directly. Take a look at the graph below.

traffic-fb-ds

Facebook referrals dropped by almost 70% during the outage (note that traffic didn’t drop to 0, presumably because some number of people had Facebook pages open before the outage). There’s certainly a drop in dark social, but it's not nearly as stark, and dark social traffic just before the outage was only 11% higher than at its low point during the outage. Since 70% of Facebook traffic dropped off, that would imply that at most 16% (11% / 70%) of traffic could’ve been directly attributable to Facebook.

Now, of course, we’d expect some other social sharing might be negatively impacted — if people aren’t discovering articles on Facebook, they might not be sharing them in other ways. So, that doesn’t mean that 16% of dark social traffic is from Facebook, but it does provide strong evidence that 84% of dark social traffic is something other than Facebook traffic in disguise.

Where people go in an outage

As I discussed in my last post, a huge percentage of mobile traffic comes from Facebook. Given that, we’d probably expect mobile traffic to be hardest hit during the outage. And, indeed, entrances to sites on mobile devices were down 8.5%, when comparing the minute before the outage to the lowest point while Facebook was down.

Interestingly, though, we see the opposite effect on desktops: a 3.5% overall increase in desktop traffic after the beginning of the outage. That increase was largely fueled by a 9% increase in homepage direct traffic on sites with loyal homepage followings. We saw no increases in traffic via other referrers, including Twitter and Google News, during the outage. While we certainly can't claim that the outage was the cause of that uptick in desktop traffic, the timing is certainly notable.

traffic-desktop-direct

In short, then: our brief world without Facebook looked a bit different, albeit in predictable ways. Significantly less news was consumed on phones, slightly more homepages were visited on desktops, and 30 minutes later, when Facebook came back online, traffic returned to normal.

The Homepage, Social, and the Rise of Mobile

July 28th, 2014 by Josh

In the much-circulated New York Times Innovation Report, perhaps the most discussed graph was this one, showing a roughly 40% decline in homepage audience over the past three years.

nytimes-innovation-homepage

With that graph, innumerable articles announcing the “death of the homepage” were written, in The Atlantic, Poynter, and on numerous blogs. Most hinged on the relationship between the rise of social traffic and the decrease in homepage traffic. One thing that isn’t mentioned in most of these articles, though, is that the rise in social traffic was contemporaneous with a rise in mobile traffic, and that mobile is as much a principal part of the story as social is. Here, I’d like to explore the three-way interaction between mobile traffic, social, and homepage visitation.

Social traffic and mobile devices

The importance of social sharing on mobile devices is much discussed. (Take for example, the recent ShareThis report, which reported that 63% of Twitter activity and 44% of Facebook activity happens on mobile.) People aren’t just using social media on mobile to share articles, of course, they’re also clicking to those articles. Below, we break down the share of traffic coming from Facebook and Twitter by device across a random sample of our sites. (Note: We specifically chose sites without separate mobile sites and without mobile apps, to ensure that we’re making fair comparisons across devices.)

traffic-device

Facebook’s share of overall mobile referrals is nearly 2.7x larger than its share on desktop. Twitter’s share is 2.5x larger on mobile than on desktop. And, if anything, those numbers likely undercount the significance of social referrals, since many apps don’t forward referrer information and get thrown into the bucket of “dark social.” In some sense, then, it’s fair to say that—for most sites—mobile traffic more-or-less is social traffic.

Mobile and homepage traffic

Setting aside where visitors come from, mobile visitors are substantially less likely to interact with a site’s homepage. Below we plot, for the same collection of sites as above, the fraction of visitors that have visited any landing page (e.g. the homepage, a section front) over a month.

homepage-all

What we see is dramatic: Desktop visitors are over 4x more likely to visit landing pages than those on phones.

Is that because mobile visitors come from social sources, and social visitors are less likely to visit landing pages—a fact that’s often cited when discussing the state of homepage traffic? Or is it not an issue of referrer at all—are mobile visitors intrinsically less likely to visit landing pages? To move toward an answer, we can control for referrer and ask the same question. Below, we plot the fraction of visitors who come to the site from Facebook and then and during the same month (but not necessarily on the same visit) visit a landing page.

homepage-facebook

Comparing this graph to the previous one, three things are clear:

  1. As discussed above, mobile visitors are significantly less likely to ever visit landing pages than desktop and tablet visitors.
  2. Similarly, visitors who come from Facebook are significantly less likely to ever visit landing pages than those who come from other sources. On average, only 6% of visitors who come from Facebook ever visit a landing page, compared to nearly 14% of overall visitors.
  3. These two phenomena are to some degree independent—desktop-based Facebook visitors are half as likely to visit landing pages as other desktop-based visitors, while mobile Facebook visitors are one-third as likely to visit homepages as other mobile visitors.

It’s also worth a quick note that, in all of these respects, tablet traffic is much closer to desktop traffic than it is to mobile traffic.

Overall, this seems to be cause for substantial concern to publishers—increases in social and mobile traffic are the two most significant traffic trends of the past few years, and both are strongly associated with drops in homepage traffic. Since, as we’ve seen before, homepage visitors are typically a site’s most loyal audience, potential drops in homepage visitors should be concerning. In the short term, it’s safe to assume that a successful mobile strategy will hinge upon a steady stream of social links—that visitors won’t return unless we reach out to them directly. In the longer term, there’s a lot of work for all of us in determining how best to build an audience in a post-desktop (and potentially post-homepage) world.

Attention Web World Cup Wrap-Up: Sample Size and Variability

July 17th, 2014 by Dan

After a month of exciting matches, the Attention Web World Cup has come to a close. In a time-honored tradition (pun intended) Ghana defeated the US with a score of 30 to 25. Congratulations to everyone from Ghana who was consuming content on the web during World Cup matches; you all contributed to this amazing achievement! And to my fellow Americans: next time around, let’s spend more time reading, okay?

To wrap up the festivities, one of our designers made these awesome animations of the time course of each tournament game based on the data I pulled. These plots show the median Engaged Time for users from each country as each match progresses.

When you view these animations, you’ll likely notice that some of these countries have incredibly stable Engaged Times while others have Engaged Times that are incredibly erratic. The U.S., for instance shows a very small amount of variance in median Engaged Time, while Cote d’Ivoire and Cameroon have median Engaged Times that jump all over the place.

This behavior is a consequence of sample size. At any particular time during a match, users from many of the African countries and other smaller countries were a much smaller sample size than, say, users from the US or Australia. In statistics and data analysis, we’re always concerned about sample size for exactly the reason illustrated in many of these graphs. The variability in the sampled statistic can mask the “true” value. We can try to capture this with a distribution, but if the width of that distribution is large, then we can’t be very confident in the value of whatever measure of central tendency we choose (mean, median, mode, etc.). And sample variance depends on the inverse of the sample size, so only as the number of points we’ve sampled gets large do we have a hope that the confidence in our estimate will rise.

I’m actually quite surprised the U.S. made it so far in my scoring scheme here. I knew going into the #AWWC that some countries were sorely underrepresented in our sample. I expected a fair chance that these countries would show a falsely high median Engaged Time. If enough of the small sample of users just so happened to be long-engagement users, this would skew their results. In the Group Round this was okay, because I performed a statistical test that tried to account for this variability. There, I asked a very common statistical question: Assuming these two teams actually have the same median Engaged Time, what is the probability that I’d observe a difference in medians at least as extreme as the one I’ve observed? If that probability was low enough, then I declared Team A and Team B to have different medians, and took the higher one as the winner. But in the bracket round, we needed clear winners (no draws were allowed), so we left it up to sampling variance. For the small-sample-size teams, this was a double edged sword. They only needed a few users spending an inordinate time engaged with content to edge above the higher-sample-size teams. But, conversely, if the users they had spent very short times, that would skew towards losing. We can see, though, that this seemed to work out well for these counties—they made a great showing all the way through the AWWC.

Thinking about variability is my job, so I might be biased here (yes, a statistics pun), but I hope you enjoyed this fun exploration of our data. I hope it got you thinking about international variability in engagement, and variability of metrics in general. Tweet me @dpvalente or email me at dan@chartbeat if you want to continue the discussion.

Revisiting Return Rates

July 14th, 2014 by Josh

Starting today, we’ve updated our definition of return rate in both our Weekly Perspectives and in the Chartbeat Publishing dashboard. Consequently, you’re likely to see a shift in the numbers in your dashboard — so we wanted to write a quick note explaining the change, why we made it, and what you can expect to see.

Defining return rate

Return rate, if you’re not familiar with it, is a metric designed to capture the quality of traffic that typically comes from a referrer. It measures the fraction of visitors coming from a given referrer who return to a site later — if 1,000 people come to a site from, say, Facebook, should we expect 10 of them to come back or 500? Depending on the answer, we might interpret and respond to a spike from Facebook quite differently. While the intuition behind return rate is straightforward, the actual formula used to calculate it is a bit more up for grabs. Up until now, we’ve calculated return rates using the following formula: CodeCogsEqn (3) That formula roughly captures a notion of “how likely is it, for a given visit from Facebook, that that visit will be ‘converted’ into a return?”   As we’ve talked through that definition over the past year, we’ve come to realize that it’s more natural to phrase returns in terms of people, not visits — to ask “how likely is it, for a given visitor from Facebook, that that person will be ‘converted’ into a return?” Hence, we’re now using the following calculation: CodeCogsEqn (4) So, rather than speaking in units of “visits,” this definition speaks in units of “visitors” — a seemingly small (but significant) change. In addition, we’re now only counting a return if it’s at least an hour after the initial entrance, which corrects for a pattern we sometimes see where visitors enter a site and then re-enter a few minutes later.    

What's changing?

It’s likely that the return rate numbers in your dashboard and Weekly Perspectives will drop under this new definition. To help you sort out whether your numbers are trending up or down, we’ve gone back and recalculated reports using the new methodology, going back to the beginning of June. We hope that the transition to the new definition is painless, but if you have any questions, feel free to comment or get in touch with me at josh@chartbeat.com

Attention Web World Cup: Follow Along with Our Bracket for the Round of 16

June 27th, 2014 by Dan

AWWC

After two weeks of intense international web engagement, our bracket for the Attention Web World Cup is set. Many of the groups came down to the very last game, and if you’ve been following along, you witnessed the excitement of Honduras narrowly edging out Switzerland by one second and the USA keeping their two-second halftime lead to defeat Germany and advance to the knockout stage of the Attention Web World Cup.

The first round looks to have some really exciting matchups, if the scores from the Group Round any indicator. Nigeria, however, appears to be the clear favorite going into the next round.

There will be a few small rule changes for the Attention Web World Cup from here on out. First, there will be no draws allowed, so we’re throwing statistical significance out the door and determining the winner only by the team with the highest median Engaged Time. Second, although the teams in the #AWWC are different than in the real World Cup, each match will still be played at the same time as the corresponding match.

Keep on checking back for bracket updates and blog posts. Boa Sorte e divirta-se!