Natural experiments can illuminate problems in medicine

Economist and physician Anupam "Bapu" Jena, MD, PhD, speaks on Nov. 9 at Learn Serve Lead 2024: The AAMC Annual Meeting in Atlanta.

Credit: Kaveh Sardari

As a resident physician at Massachusetts General Hospital in Boston, Anupam "Bapu" Jena, MD, PhD, recalled noticing that a substantial cohort of cardiologists were out of the hospital during the week of the annual cardiologists' convention.

He wondered if this fact might have an impact on the health outcomes of patients who had cardiac events during that week each year, and as a researcher who looks at how random events can impact health, he gathered and examined the data.

To his surprise, mortality rates dropped by 10 percentage points when more cardiologists were out of town.

"In medicine, there's this idea of less is more," said Jena, the Joseph P. Newhouse Professor of Health Care Policy at Harvard Medical School, a physician in the Department of Medicine at Massachusetts General Hospital, and a faculty research associate at the National Bureau of Economic Research. His hypothesis is that fewer interventions may improve outcomes for some patients.

This was one of several examples Jena presented during his session at Learn Serve Lead: The AAMC Annual Meeting on Nov. 9 in Atlanta, Georgia, of how random acts - from the month a child is born to a city hosting a marathon - can have an impact on health outcomes.

"I spend most of my time studying questions at the intersection of medicine and economics," Jena explained. "It's kind of like Freakonomics meets medicine."

Jena presented the audience with various questions and walked them through the brainstorming process that he and his team go through when deciding what phenomena to study.

"The exercises we're going to go through, I spend several hours doing with my colleagues every week," Jena said.

While many of the observational studies he has published and highlighted in the book he co-authored, Random Acts of Medicine: The Hidden Forces That Sway Doctors, Impact Patients, and Shape Our Health, don't have obvious policy solutions, Jena shared the impact of one of his studies.

After he used data analysis to show that children who are born in the summer are less likely than children born in the fall to receive a flu shot, a pediatrician's office wrote to tell him they would begin hosting flu shot clinics for kids with summer birthdays in October.

"Most of the ideas that we have are not good, most don't result in papers," Jena said. "It just takes one idea that's good. That's enough."

Jena encouraged the clinicians in the audience to be curious and use observation and creative thinking to take new looks at old problems.

And in closing, Jena promised not to study data on patient outcomes the week of the AAMC annual meeting.

Before the meeting, AAMCNews spoke with Jena about the inspiration behind his experiments and what he's learned about the randomness of health and medicine through his research.

What inspired you to pursue research at the intersection of economics and medicine?

I fell into it by chance. When I was an undergraduate at Massachusetts Institute of Technology, I was working in a lab and I planned to do an MD and a PhD in the basic sciences. And when I was going around the country and interviewing for MD-PhD programs, the director of the program at the University of Chicago said to me, "I noticed you majored in economics as well as biology. Would you want to do your PhD in economics instead?" I hadn't thought about it at all up until that point. But I went to the economics department that afternoon, met with some professors, applied a week later, and started just a few months later in the economics program and medical school.

In your book, you explore a wide variety of "random acts." Can you walk me through the process you and your colleagues go through when deciding what phenomena to study?

So, a couple of things: one is that natural experiments are very common as a tool to study phenomena in the field of economics. They're much less common in medicine and epidemiology. And so, part of my interest in those tools came from my disciplinary background in economics, but also the particular types of questions that I was interested in looking at, which are very much Freakonomics-meets-medicine-style questions. I was influenced by my PhD adviser, Steve Levitt, a coauthor of Freakonomics. He was doing some interesting work in the fields of crime and other social behaviors. I thought, "Wouldn't it be interesting to apply that same kind of thinking - using large data, natural experiments, creative questions - in medicine and in health?" And that's where it took off.

You and your coauthor explore a wide variety of natural experiments, from how a child's birth month could influence their likelihood of getting a flu shot to how a doctor's politics might influence the kind of care a patient gets. How do you decide what factors and phenomena you're going to study?

A lot of the studies in the book come from stories where I had an experience that most people would probably just overlook. You mentioned the birth month and flu. That happened because I was at the pediatrician's office and the flu shot was not [yet] available for our son because he has an August birthday [and his annual checkup was scheduled close to his birthday, as is customary]. The flu shot isn't typically available in pediatricians' offices until early September, so had my son been born a few weeks later, he would have gotten it at that annual checkup. [Instead, as Jena explains in his book, he had to make an additional visit to the pediatrician for his son's flu shot. This extra visit creates a barrier to care for many families whose children have August birthdays, reducing flu-vaccine coverage, his experiment found.]

I think we see many of these kinds of phenomena in the real world. In terms of what it is that makes us settle in on a question, I think [it's] a few things: One is that we need to have the data to be able to answer it. Two is, there has to be some sort of experiment. Something that creates different paths which people take by chance, whether it be a surgery or no surgery, or medication or no medication, whatever it may be. And the third thing is, we look for things where the outcomes are measurable and matter. So, receipt of a flu vaccine probably matters; it can be life or death. The last thing is maybe the most important criterion, which is, I try to work on things I could explain to anybody. You don't have to be an expert in medicine or in economics.

In your conversations with your colleagues, or with friends and family who maybe don't have experience in natural experiments or economics, what kind of reactions do you get?

I get mixed reactions. I have a close friend from residency at Massachusetts General Hospital who used to always poke fun of me anytime I had a study come out. He'd say, "There you go again. Bapu is just publishing a study about how water is wet."

One view of the world is, like, OK, of course, when roads are blocked, people can't get to the hospital, and delays in care might lead to problems with health - in this case, increased mortality - because of a marathon. It's so obvious when you think about it. It's, of course, not obvious when you're thinking about the design of these sorts of big events that have huge infrastructure disruptions. People are not thinking about that. Everybody understands the questions that I ask. Sometimes they actually say, "This is obvious. Why would you study it?" But I think most of the time, people are, like, "That's cool. How did you come up with that idea? It makes total sense when you say it, but I would never have thought about it." That's the dominant reaction, which is good, because if it weren't, then I'd have a problem.

I'm also curious about how cardiologists respond to your study that found that mortality decreases from heart attacks during the week that the annual cardiologists' convention is held?

When that study came out, there was actually a Freakonomics podcast episode about it, and the then president of the American Heart Association made an interesting statement about the study, saying that patients weren't being harmed with cardiologists being away. He was right about that!

I think the tools of natural experiments are very familiar to economists, so they understood what the study was saying, and they could believe the results. In medicine it's harder, because you look at a study like this and you say, "Well, what are all the factors that could be going on?" And if you're a cardiologist and you're looking at this, the last thing you want to think is that outside of the dates of these meetings, you might be doing something that is suboptimal for patients. But clearly that has to be the case. Not everything we do in medicine is informed by a randomized controlled trial, or even when it is, it's not always perfectly applied. And to think that we couldn't do better in medicine, or really in any occupation, has got to be incorrect.

You have a chapter on what makes a good doctor. What did you learn in your research?

The studies don't tell us so much about what makes a good doctor, but maybe what they tell us is, if you had preconceived notions about certain characteristics of doctors that would make them good or bad, or better or worse, they may not be completely true. For example, do doctors who trained at very prestigious medical schools have better [patient] outcomes? That's a belief that someone might have. Or they may believe older doctors have better outcomes than younger doctors. We show that, in fact, outside of surgery, older doctors tend to have worse outcomes than younger doctors.

But the point isn't to say you should pick a young doctor or pick a female doctor or pick a doctor who trained outside of this country - because we show that women and foreign-born medical doctors also have better outcomes. Instead, the solution is to say, if you are afraid of getting a younger doctor because you thought they might be a little less experienced, that's probably not a big concern.

What has your research brought to the field of medicine, and has it impacted the way you go about your practice and your life?

Most of my contribution is bringing to the forefront this use of natural experiments, but also interjecting some creativity and fun into the type of research that people can do. As far as me as a clinician, that's hard to say, since I work with so many different topics. I think there are some examples where I would think differently. We have this paper about left-digit bias, so when someone is 79 years old, you think of them as in their 70s, and if someone is 80 years old, you think of them as in their 80s.

Now, it would be nice to say that because of that study, when I'm in the hospital, I see somebody and think very carefully and thoughtfully about what their actual age is, and what their ability is to tolerate a treatment and not be sort of swayed by these heuristics that sometimes cause cognitive biases in clinical decision-making. So, maybe I pay a little bit more attention to those kinds of things.

The one thing that I try to do in medicine is to spend more time with people. I have worked in a lot of clinical settings where things are busy. Oftentimes the medical teams I work with - the residents, the interns - they've got so much on their plates, it's hard to spend time with patients. I think as I've gotten older, I see that as being more important than I did a decade ago.

What we highlight in the book are areas where people can make mistakes in the way they think about problems. They can come to the wrong conclusion not because they are not working hard, but because when there's so much information to grapple with, and the time they have to make decisions is short, biases can creep into the decisions that doctors make. And we don't really know, if you give clinicians more time to make these decisions, if you give them more time to talk to patients, how much better could actual clinical outcomes be? Yet, the amount of time that clinicians spend with patients is getting shorter - not longer - over time, and so these issues that we highlight, I would expect would get more prominent over time, not less.