AHCJ – Association of Health Care Journalists

07/23/2024 | News release | Distributed by Public on 07/23/2024 14:34

Writing about health care interventions? The HealthNewsReview criteria are as relevant as ever

Photo by Ivan Samkov via Pexels

By Gary Schwitzer, founder & publisher of HealthNewsReview.org

Editor's note: AHCJ is republishing the HealthNewsReview criteria because they are as useful as ever if you want to assess your own work by applying the questions as a quality checklist. That process seems likely to help your audiences better navigate health news so they can make the best possible decision for their own health.

In 2004, researchers at the University of Newcastle in Australia launched a pioneering website - Media Doctor Australia - aimed at providing an objective analysis of the strengths and weaknesses of health news stories.

At the core of their work were 10 standardized criteria that were used to review the stories. A year later, as I began building the now-defunct U.S.-based website, HealthNewsReview.org, I gained permission from the lead Australian researcher to adopt those criteria.

Nearly 20 years later, I've still never seen a better set of criteria for reviewing stories about health care interventions. It was a patient-centered approach, laying out questions that patients need to have answered when they encounter news about treatments, tests, products and procedures.

Eventually, projects in Canada, Hong Kong, Japan and Germany adopted the same Australian criteria. Ours was the best-funded ($800,000 per year at its peak), with the largest team of contributors (50 at peak), the greatest output (a dozen articles per week at peak), and the biggest audience reach (about 1 million unique visitors per year at peak), including a broad international following. I was invited to lecture in Australia, Austria, Brazil, Canada, China, England, Finland, Germany, India, Mexico, Peru, Puerto Rico and Spain.

During our 16-year run, our team published detailed reviews of more than 2,600 news stories, giving the grades "satisfactory," "unsatisfactory" or "not applicable" for each of the criteria. We also reviewed 600 news releases, using slightly different criteria.

Overall, our grading showed a clear tendency for health news to emphasize or exaggerate the potential benefits of an intervention while minimizing or ignoring potential harms.

Here is how we described the criteria and their importance.

#1 Does the story adequately discuss the costs of the intervention? Our long-term experience with thousands of stories shows that the cost criterion received the most unsatisfactory scores. News stories don't help patients and consumers if they don't address costs.

#2 Does the story adequately quantify the benefits of the intervention? New health care treatments should work well. When reading a story, people want to know: How effective is the intervention? Do the numbers back it up?

#3 Does the story adequately explain/quantify the harms of the intervention? In the news, if you're only hearing about the potential benefits of a test or treatment, run for the hills. What are the harms, and how often do they occur?

#4 Does the story seem to grasp the quality of the evidence? Ideally, health care interventions are subjected to rigorous testing to prove they work. When reading about a new intervention, device or diagnostic tool, people should come away with a sense of how rigorous the evidence is for the intervention.

#5 Does the story commit disease-mongering? Sometimes the naming of new conditions or diseases actually may just be opportunities to medicalize (and monetize) normal states of health. Readers aren't well-served by stories that exaggerate or oversell conditions. [Note: Examples of this include turning risk factors into diseases, with the implication that, then, these must be treated (e.g., low bone mineral density becomes osteoporosis); or medicalization of normal states (aging, baldness, wrinkles, shyness).]

#6 Does the story use independent sources and identify conflicts of interest? Health care is rife with conflicts of interest. Readers deserve to know if the sources in the story have any conflicts of interest. For example, do researchers stand to benefit financially from the treatment they are promoting? Are they paid consultants for the company that manufactures a relevant pharmaceutical? Readers also deserve to hear the opinion of objective, independent third-party experts, who are vital to backstopping exaggerated claims.

#7 Does the story compare the new approach with existing alternatives? Just because something is new in health care doesn't mean it's better. When reading about a new drug, test, surgery, etc., readers should come away informed that there may be other options on the table. These options may have a longer, more-proven track record.

#8 Does the story note the availability of the treatment/test/product/procedure? New medical treatments often get lots of attention before they're ever made available to the public. After reading a story about a new intervention, a person should be able to answer: Is it actually available to me?

#9 Does the story establish the true novelty of the approach? Often something that's lauded as new or novel really isn't. Or it may be new, but it's not better. So, when reading about a "new" medical intervention, people deserve to know: What is truly new and different about this idea?

#10 Does the story appear to rely solely or largely on a news release? Sometimes news stories rely solely or largely on a PR news release and never disclose that fact. Readers deserve to know if a story includes no original reporting.

I never thought that all 10 criteria had equal weight or importance. The five that I list below, and the grades for them, are the most important in my view. They are also the categories that gave journalists and PR professionals the most trouble. Our reviewers frequently pointed to incomplete news stories (which scored poorly) that relied solely or largely on PR news releases (which also scored poorly).

Here are some of the most common reporting flaws that contributed to those low percentages.

  • Framing study results in the most positive light, often using relative, not absolute, risk reduction figures.
  • Writing about study results on surrogate endpoints or surrogate markers, without explaining the limitations of such results.
  • Writing about observational study results while inappropriately using causal language and without explaining the limitations of such results.
  • Reporting based on input from a single source, often someone with a conflict of interest that is not disclosed. Worse yet, sometimes the reporter or their company has accepted something of value from that source.
  • The tyranny of the anecdote: Beware of single-anecdote stories in which the patient quoted may be used as a public relations tool.

A few observations jumped out from our news release reviews.

  • News releases on nutrition, nutritional supplements or "natural" interventions tended to be consistently more promotional than evidence-based - providing little or questionable data.
  • Another category that stands out for hype was news releases on various new cancer tests. We felt that many did a disservice to patients and the general public with misleading claims. We noted PR releases that didn't disclose conflicts of interest, misled readers, gave no justification for claims made, or generally oversold an idea beyond the boundaries of what the evidence showed.

One of the most gratifying types of feedback we received was from journalists who would bring our 10 criteria to their editors to push for inclusion of parts of the story that the editors wanted to delete. In some cases, the reporters were more experienced on these topics than their editors, so this pushback was important.

We were funded for eight and a half years by the Foundation for Informed Medical Decision Making (now defunct) and for four years by the Laura and John Arnold Foundation (now Arnold Ventures). In the midst of our 16-year run, we once went 19 months without any funding, yet kept the site alive.

Finally, though, in 2022, the age of the site's web code became too expensive to fix, funding grew too scarce, and the publisher (me) was getting too old. So we decided to take the site down. (You can see what it looked like via the Wayback Machine, but give it some time to load.)

I am pleased that AHCJ is devoting fresh attention to the criteria that were at the core of our efforts to help people improve their critical thinking about claims made about health care interventions.

Gary Schwitzer was a health care journalist for nearly 50 years and is the founder and publisher of HealthNewsReview.org, which was active for 16 years.

What others had to say about HealthNewsReview.org


A Guide to Reading Health Care News. JAMA Internal Medicine. Published online May 05, 2014. doi:10.1001/jamainternmed.2014.1359.

News coverage: