Regardless of your newsroom’s size or how many articles you publish every day, chances are you’ve got a Twitter account.

What’s more, you’ve likely tried, to greater or lesser success, to leverage the social network for the distribution and promotion of your content. But once your thought-provoking, 140-characters-or-less message is dispatched, what happens next? Will the time and effort you spent pitching your editor pay off? Will you draw in readers who will actively engage with the content? Will you manage to convince readers to explore additional articles? Could you even convince users to come back over and over again? Or, is all that effort lost in the Twitterverse, drawing in a few readers who come and leave, never to be seen again?

These are just a few of the questions we’ve been trying to answer here at Chartbeat. But rather than placing all visitors who come from Twitter into a single class and making the assumption that they all behave the same way, we decided to take a deeper dive with an eye toward nuance. We examined the behavior of readers who come from tweets published by content owners (first parties) versus those coming from independent agents (third parties).

To test our assumption and measure different forms of engagement, we decided on four metrics: average Engaged Time; average number of other pages a viewer visits on a site within two hours of their first visit; percent of users who return after initially visiting; and of those users who return, the number of times on average they will return in the next 30 days. Roughly broken down, this gives us two metrics to look at short-term reader value (engagement and redirects), and two to look at long-term reader value (percent retained and rate of user return).

From previous experience and assumptions we made, it seemed to us that readers coming directly from a content owners’ tweet would probably already be a member of that publisher’s loyal audience. It would therefore seem logical that these users show qualities similar to that of loyal readers—chiefly that they exhibit a higher than average return rate, and read more pages when visiting.

So it wasn’t surprising that when observing the percent of users that came back, readers from first-party sources showed returning rates about 15% higher than readers from third parties. During their initial visit, readers coming from Twitter also tend to stick around longer, with first-party consumers reading on average three pages during a visit, compared to non-social traffic’s one page. The difference between first-party and third-party social consumers, however, does not differ significantly.

The number of times returning users came back however was surprising. Of those users who came back at all, users coming from a first party returned on average 8 to 10 times. A similar user, though, who came from a third party came back 11 to 13 times. This may suggest that after passing that retention barrier and convincing a reader to come back, the users you receive turn out to be much more valuable, as they return more often, and help bolster your current loyal population.

When looking at the time a reader engaged with a page, we found that readers from third parties actually engaged with content significantly longer than first-party readers. While first-party consumers engaged with content about the same amount as any other user, regardless of where they came from (averaging between 37 and 39 seconds), third-party readers engaged on average between 42 to 45 seconds. (Calculated with a p-value of p < 0.01) These differences, while seeming small, can lead to practical differences in engagement from a few seconds to nearly a 40% difference in Engaged Time.

Though there are many reasons that these differences may be occurring, one possible conclusion lies in what attracts readers to engage with content. Users who follow publishers on Twitter are apt to know more about the publisher’s content; consequently having a greater sense of what type of content they want to read. Loyal readers, as opposed to new readers, may therefore be skimming through content, knowing they will come back later for follow-up stories, or to learn more. Non-loyal readers, however, who are generally the readers coming from third-party tweets, come due to the referral of a friend. These readers may engage deeply with content due to the personal connection with the recommender of that content.

So, to answer the initial questions of whether your tweets actually matter to the health of your site: Of course they do—you already knew that. Your tweets are vital to your loyal audience, and bring in readers who return more often and consume higher quantities of content than readers coming from anywhere else. Don’t forget about the importance of making content people want to tweet about, though! Because it turns out people actually listen to the recommendations of their friends, deeply engage in the content before them, and if enamored enough to return in the future, turn into fiercely loyal members of your site’s virtual population.