Our guest blogger today is Michael Lovitt, VP of Engineering at Vox Media.

Last month, Vox engineers spent three days at Chartbeat’s NY office hacking on digital ad measurement. I want to share Vox’s take on the current state of online advertising and tell you how and why we’re working with Chartbeat to make things better.

The current state of online advertising

The state of digital advertising — and brand advertising, in particular — is suboptimal. In short, as an industry, we’re showing ads that readers don’t want to see and measuring using techniques that don’t tell us enough about whether ads are successful.

The good news, from our perspective at Vox Media, is that digital advertising has enormous room for improvement. We believe that by designing beautiful ads that delight instead of annoy, elegantly integrating these beautifully-designed ads alongside premium brands and content and for a premium audience, and employing measurement techniques that reveal a complete picture about who viewed an ad and to what degree readers engaged with and were influenced by it, that we can provide an advertising experience that does a better job of pleasing everyone involved: publishers, advertisers, and readers alike.

We’re making progress on all of the above, but solving these problems isn’t easy. It helps to have great partners.

Solving problems with Chartbeat

On measurement in particular, we were excited to work directly with Chartbeat because, like us, they’re convinced that the current state of digital ad measurement is not good enough and are actively working to make it better. Of course, Chartbeat is in the business of knowing how users engage with websites, and they’ve started to put that expertise to work tracking ads. (We had nodded our heads all the way through reading their recent blog post on the superiority of Engaged Time over impressions and clicks and awesome study showing a strong correlation between Engaged Time and brand recall.)

We’re also passionate (OK, addicted) users of Chartbeat’s products. Screenshots of important Vox milestones as visualized by Chartbeat litter our Campfire transcripts. These people do awesome work and it’s fun to hack with folks whose work you admire.

For all of these reasons, when Chartbeat invited us to attend one of their Hack Weeks, we immediately said yes.

Pre-hack planning

The Hack Week invitation was made at an opportune time, as Vox had just released an early alpha version of a new ad product metrics dashboard. The system was still in its early stages, but it was functioning in at least a basic way at every layer, and it served as a foundation for us to hack on.

We arrived at Chartbeat with a mock of a few changes to the dashboard — we knew we wanted to start reporting, for every ad, both the Average and Total Engaged Time. As Chartbeat has shown, Engaged Time correlates well with brand recall, so being able to provide this level of insight to our advertisers and to ourselves would be one important step forward in moving beyond standard clicks and impressions

So we came with one concrete thing that we wanted to accomplish (having validated the idea with Chartbeat beforehand to make sure what we had in mind wasn’t crazy); beyond that, we were ready to play it by ear.

Hack, hack, hack

Here’s a quick rundown of what happened during our three days at Chartbeat:

Trei, Niv, Pablo, Aaron, and I arrived late Wednesday morning and kicked off with a short presentation to the Chartbeat team about Vox, and then settled into Chartbeat’s Stark Tower conference room and got to work.

We sat down with some of the Chartbeat team — Harry, Matt, Wes, Shaun, and Alex — and reviewed the dashboard mock. It turned out that Chartbeat’s existing ad product would be sufficient to record all the data we needed, but that a new API would need to be built to return that data back to us. Vox engineers got to work adding Chartbeat tracking to Vox ads, and Chartbeat engineers started building out the new API.

By the end of the first day, we had a Chartbeat-instrumented Vox ad running in production and a new Chartbeat API running on a laptop and returning data from Chartbeat’s production data store. From there, we could run our metrics dashboard in our dev environment and start to see real data flow in. Hurray!

On the second day, we solidified what we had built the day before. We wrote the front-end code to beautifully display metrics on the dashboard and got the Chartbeat API running on a server on the web so that we could push our metrics dashboard changes live.

Since we were making good progress on our primary goal, we also took some time to hack on Chorus, Vox’s publishing platform. We used an existing production Chartbeat API to integrate real-time Chartbeat metrics into the Chorus layout editor — so that, for example, an editor on Polygon, when deciding what stories should be placed on the home page, can easily see the current visitor count on each currently-placed story.

The morning of the third and final day, we asked ourselves if we could get one more metric, Engaged Concurrents, integrated into the dashboard before the 1pm demo. It seemed feasible so we tried and got it done.

We showed off what we had all built to the Chartbeat team over pizza. After declaring total victory, the Vox crew said goodbye and stopped for celebratory drinks while we waited for our train back to DC.

Closing thoughts

We had a blast hacking with Chartbeat. It’s fun to make things with smart people, especially when they’re focused on the same problems as you are. Beyond ad measurement problem-solving, we enjoyed getting to know the team and talking shop: approaches to team organization, benefits and trade-offs of remote versus centralized workers, respective merits of modern JavaScript frameworks, and on and on.

Finally, we’ve been able to maintain post-hack momentum and have made great progress taking these projects to completion — an important final step in any hackathon, and especially a cross-company hackathon like this one. Chartbeat has already released the APIs they built during our visit into production and we’re working this week at Vox to update the API client code in our metrics dashboard.

Now that we have this ad engagement data, what’s next? To start, we’ll be using this data internally to better understand how well our ads are performing — by feeding it into our ad design process, measuring ad performance, and iterating. We’ll consider this data alongside other metrics that reveal how users are viewing and interacting with our ads. As we continue to take on the hard problem of optimally measuring digital ads, we look forward to having partners like Chartbeat at our side.

 

PS- Check out Chartbeat CTO Wes Chow’s guest post for the Vox Product blog.