When it comes to audience diversity, newsrooms are asking the wrong questions

News organizations often miss what their audiences actually care about because they rely too often on a flawed approach to surveying and ask the wrong questions.

Right now, many news organizations’ surveys dramatically over-represent older white people in their audience and fail to ask about what researchers identify as the foundations of a real relationship with a news organization. The Institute for Nonprofit News’ audience team, of which we are a part, has developed a new set of free survey tools with leading researchers and news leaders that asks smarter questions and explains how to pull actionable insights from that data.

In developing these tools, it was not uncommon for us to hear stories like we heard from Cityside, which was invested in holding itself accountable to serving a broad swath of residents in Berkeley and Oakland, Calif.

“Last year, we shared the findings of our annual survey with the entire newsroom, and reporters and editors felt discouraged because our audience was represented as mainly older and white — it was so different from what they were expecting,” said Alejandra Armstrong, audience engagement editor for Cityside.

Cityside had invested heavily in building an audience for Oaklandside and had seen some major wins with its quarterly Culture Makers event series that had brought out lots of young people, who anecdotally expressed a strong affinity for the news organization and showed up when it was in their community. The editorial staff expected that a willingness to read a story or to show up for an event translated to a willingness to take a survey. But it didn’t.

Many of the organizations we talked to had similar goals to Cityside — they wanted an audience that looked like the larger community they hoped to serve in the same ways they wanted their staff to reflect the community. But their surveys were telling them something else.

“We really had to emphasize that this is not a good representation of our audience as a whole — this is who answered the survey,” said Armstrong, who contributed to INN’s audience survey template.

We found the initial misunderstanding of surveys in Cityside’s newsroom was incredibly common. Surveys are very strong when used to understand the motivations and behaviors of different constituencies. But when they’re used in a narrow way as a census to count up the number of people in different groups, they may lead to disappointment.

At its worst, the census approach looks like scorekeeping, with people tallied up in categories that are largely divorced from what researchers have found actually drive their news habits — such as whether readers are talking about what they’ve read with other people, or what role the organization plays in their daily routines. Rather than showing opportunities, the survey ends up becoming demoralizing to journalists and the data gets shelved and ignored.

A survey like the one INN has designed allows news organizations to see that there are many times where a person may minimize a political identity when engaging in specific news topics — while a small aspect of their identity, like being a suburban commuter, has an outsized impact on when and how they access news.

Researchers studying news audiences have found that asking a question like “How often do you talk with others about things you read in this publication?” or “Do you agree or disagree with the following statement: This publication cares about people like me” can be much more instructive about how the organization is forming relationships with different constituencies.

But why do surveys keep showing news organizations so many older white voices if that’s not who the audience actually is?

By far the most effective way to field a survey is by email. Richer, older, white audiences have a preference for email newsletters and are likely to dominate news organization’s largest email lists. They also have more time to take surveys. This is made more pronounced by the industry practice of scrubbing newsletters’ lists of anyone who doesn’t act like the most devoted readers with high open or click-through rates.

These are well-known issues among news audience researchers, said Stephanie Lynn Edgerly, associate dean for research at the Medill School at Northwestern University.

“It does not necessarily invalidate the findings, but it’s good to know who you are hearing from and who you are not,” said Edgerly who contributed to creating INN’s audience survey template. She added that news organizations should strategize ways to research audience segments that are not in their email databases or that aren’t taking their surveys.

News organizations should strongly limit the census approach to surveys, because there are no standard adjustments offering a simple fix to these issues. INN’s approach to surveys focuses instead on understanding how relationships are changing over time with different constituencies. The complaint that young people may provide only 10% of the responses but are 20% of the population falls into the failures of the census approach. INN’s survey instead looks to see what attitudes and habits are unique to that constituency and judges the organization’s work on how that changes over time. The goal is to change relationships to the organization and to news, not to taking surveys.

“A good survey can help a news organization better understand their audience — their attitudes, needs and habits,” Edgerly said. “Surveys are also useful for tracking audience patterns over time, in response to newsroom shifts or major events.”

The whole industry benefits if we better understand what our journalism is doing in people’s lives. INN is making these tools available for free to its members and extending additional support to our network to get the most out of it.