Separating the wheat from the chaff in health studies

By Megan Myscofski, California Health Fellowship

Covering health-related studies is usually a large part of a health reporter's job. It can also be one of the most fraught because studies can be difficult to vet for meaningfulness or accuracy, especially on a tight deadline.

"We as journalists are the last chance to get it right for the public," said Regina Nuzzo, who teaches statistics and is also a reporter.

It's not uncommon for a public information officer or even a scientist who worked on a particular study to misspeak about statistics related to it, she noted, and for that reason, she advised journalists to check press releases and quotes against a study rather than take them at face value.

She also said that language often trips people up, from readers to researchers. People tend to understand the risk or chance of a particular outcome better than they understand odds, so when possible, it is best to find and cite the former.

"Always ask about the risk," Nuzzo said. "Don't get swayed by the odds."

Independent journalist Tara Haelle, who co-led the workshop at Health Journalism 2024 in New York City, also warned attendees to watch for researchers who collect data while looking for a particular outcome.

"If you torture the data long enough, it will confess," she said.

But that or other poor research practices does not mean a study is not newsworthy. Haelle said it is important for journalists to be cautious about giving bad work airtime, but also that sometimes these stories help readers be better skeptics of questionable work.

"Sometimes I cover a study because it is a bad study," she said.

Haelle also explained the basic structure of a study but said she never reads one in the order it is written. She suggested that other journalists should read and dissect a study in whatever way makes the most sense to them.

Both Haelle and Nuzzo stressed in the workshop that journalists should check in with experts in a study's subject matter about a study as part of reporting on it.

"There's no substitute for understanding a subject deeply," said Nick Milazzo, lead researcher for Examine.com, a company that collects and vets scientific research related to health and wellness.

He said the field of nutrition research is particularly prone to bad research - many studies fail to control for confounding variables, spin data to sell goods and feature research primarily done by people who work for the companies selling the product being studied.

Megan Myscofski is a statehouse reporter for Capital Public Radio in Sacramento. She is a 2024 AHCJ-California Health Journalism Fellow.