AMP traffic in aggregate is increasing, but how widespread is the impact for publishers? Chartbeat, together with The Daily Beast, collaborated on a two-part research study to rigorously quantify the effect of adopting the Google-backed Accelerated Mobile Pages (AMP) format on publisher traffic. This study is the first formal statistical analysis of the effects of AMP on website traffic. The overall result of this study is a methodology for analysis that we hope will be useful to other data scientists.

Our overarching finding is that AMP boosts traffic for publishers on average, but most publishers are not average. Only 1 in 3 we analyzed could see clear statistical evidence of a traffic increase. Though it may be possible to optimize AMP implementation to improve monetization, publishers seeing lower revenue on the platform will have a hard time making the case that a traffic boost will make up for it.

The study includes a pair of related analyses. Both are documented in full in the complete study. Here’s a summary of our findings:

The first study

The first is an A/B test run by The Daily Beast in which, at random, half of newly published articles were published in the AMP format and half were published in the standard format. This test failed to show clear statistical evidence of higher Google-driven page views on the AMP than non-AMP content. Meanwhile, The Daily Beast observed clearly lower revenue for the AMP format.

The second study

The second is a time series analysis, which was designed jointly but conducted by Chartbeat. This was run over a sample of data from 159 publishers, including The Daily Beast and 158 Chartbeat-enabled publishers that adopted the AMP format in 2017. The cohort represents a mix of local and national publishers of news, sports, and lifestyle content. A clear majority of these publishers are US-based, but 10 countries are represented.

In this analysis, traffic in the months leading up to and following AMP implementation was used to determine how much the introduction of the format affected Google-driven traffic relative to other sources of traffic. We found that the average traffic effect of AMP introduction was +22% (+-4%), but there was wide variability between publishers. Only 34% of individual publishers saw statistically significant increases in traffic from AMP (at the 95% confidence level).

We did further investigations to see if we could characterize publishers that saw the biggest increases. However, we found no statistically significant correlation between AMP effect sizes and a number of potentially interesting characterizations, including implementation date, overall traffic to site, share of search traffic, and broad type of content. It is possible that the primary source of variability lies outside our ability to measure; one possibility is that AMP implementation varies in quality, though anecdotally The Daily Beast executed the recommended high quality implementation and saw no clear evidence of a traffic increase.

Summary

In sum, the prospect of AMP may not be positive for all publishers. Though the technology offers rightly lauded fast page loads, and potential opportunities in new products, with only 34% of publishers seeing a clear boost in traffic and some facing substantial monetization challenges, implementing AMP may come at a high cost for publishers. Those publishers facing revenue challenges might be better served by optimizing their implementation setup on AMP rather than relying on a traffic boost to solve these monetization challenges.

We welcome you to read the complete study and let us know your take.

Have you seen similar findings within your media company? Feel free to reach out to us at brad.doll@thedailybeast.com and christopher@chartbeat.com with thoughts or questions. We’d love to hear from you, and we’d be happy to discuss how you could put this methodology to work on your own site.