Headquartered in Salt Lake City, Deseret News is among Utah’s most widely-circulated papers. Its newsroom has partnered with Chartbeat since 2013, but hadn’t yet made full use of the suite’s powerful headline-testing capabilities. In 2017, testing was still an under-the-radar area of interest, with just a handful of headline tests on live stories per day, as Web Producer Ginny Romney explained.

Content Director Aaron Shill saw a clear opportunity.

“Our team had seen traffic start to stagnate, and since it’s our job to grow traffic, we talked about which cultural and procedural changes we should make,” he said. “We all believed it was improving headlines—and to improve headlines, we had to test them.”

The result? A CTR increase of over 45% a month— and a newsroom even more engaged in a culture of collaboration.

(For further reading on this topic: How the South China Morning Post approaches their digital transformation)

How Deseret put headlines to the test

The initiative came out of a larger, company-wide effort, Shill said. Previously, Deseret’s digital operations and traditional newsroom worked separately: undoing that divide by integrating the teams’ workflow and operations was a necessary first step to bring about larger cultural and procedural changes.

With leadership working to emphasize the value of cohesion and collaboration, it wasn’t hard to then begin popularizing Headline Testing. The team routinely used Chartbeat’s Heads Up Display in homepage management, and the tool’s seamless integration of Headline Testing made it easy to start testing.

The new challenge, then, was building momentum. Shill and his team needed to make people want to test every headline, thus optimizing homepage engagement and growing Deseret’s audience of loyal readers. To make testing compelling, they made it fun—through a game that engaged the entire organization.

(Related: Test headlines, work smarter: How MLB.com’s culture of experimentation improves reader engagement)

The rules of the game

The setup was simple: they announced a month-long contest around headline testing, splitting the entire organization into two competing teams. One was captained by Editor Doug Wilks, the other by Head Digital Officer Burke Olsen. Then, five people on each team were designated as “headline testers.” When a story was published, each team would compile and submit headline options to the testers, and the game would begin.

“That’s why original headlines were given more weight—we wanted the traffic, but we wanted original headlines to be rewarded more for getting that traffic.”

To avoid a popularity contest, the game’s organizers developed a weighting system that rewarded winning original, innovative, or creative headlines with more points than alternative variants.

“That’s why original headlines were given more weight—we wanted the traffic, but we wanted original headlines to be rewarded more for getting that traffic,” he said.

(For further information: Watch our video on Headline Testing)

Immediate and long-term impact

The initiative was hugely popular. Participants from editorial to marketing and sales engaged in competition, while “actually having fun,” Shill said. Given that each published story represented a chance to run a headline test, and thus a chance to earn points, the teams ended up testing nearly every story published to the homepage—running more than a thousand tests over the course of the contest.

The game also revealed a cultural openness to digital change.

“Everyone knew that writing headlines for the website was a different process than for print editions, but there was no telling who would be able to develop that skill, or how quickly they’d take to it,” Shill said.

Case in point: the repeat winner, a print reporter and career journalist, had worked for Deseret News for 30 years—but one of the paper’s interns was another frequent champion.

After the monthlong competition that began in August came to a close, the Deseret team looked through their testing data and noticed excellent results compared to July and September. In July, 180 tests produced a 45.5% average increase in CTR, while in August, 587 tests yielded more than 49% in average CTR increase. In September, there was a 48.7% average CTR increase for 298 tests.

Additionally, the contest validated hunches held by leadership and newsroom end-users—using Chartbeat Headline Testing should be daily practice. Running occasional tests wouldn’t make an impact—they needed to drastically increase the number of tests to see great results and ROI.

It’s proven that optimizing headlines at scale helps grow engagement, which was certainly the case for the Deseret team. But perhaps the most significant impact was in the infrastructure and communication channels they had created, and the legacy of how to use a good tool to transform the workings of a legacy organization. “It’s simple,” Shill said. “We’ve operationalized testing headlines and made it part of our jobs.”

Want to see the impact of Headline Testing in your organization? Reach out to us at headlines@chartbeat.com.