Deputy Administrator Isobel Coleman on The Power of Cost-Effectiveness Evidence to Improve Lives

DEPUTY ADMINISTRATOR ISOBEL COLEMAN: Thank you so much, Eeshani [Kandpal], for that introduction, and to all of you for joining us today as we explore the power of cost-effectiveness evidence to improve our work, and ultimately, to improve lives around the world. I do want to extend a special thanks to the Center for Global Development, which as we just heard has been advocating for greater cost effectiveness across the development ecosystem for years now, 20 years, and certainly has helped influence our own approach here at USAID. So, thank you.

In development and humanitarian assistance, we are dealing with a fundamental problem: a world in which global needs are skyrocketing, yet budgets are not keeping pace. Faced with a widening gap between needs and the resources available to meet those needs, we simply must make our dollars go further.

So this is why I'm here today: to lay out why and how USAID is deepening our focus on "cost-effectiveness" or impact per dollar, and to make the case for all of our partners around the world to join us in this journey.

We know from the global body of impact evaluation evidence that there are some approaches that deliver extraordinary returns. For example, USAID's Bureau for Humanitarian Assistance has invested in an evidence based approach to increasing income for poor and marginalized households, known as the "graduation approach." This is a sequenced set of supports, including monetary transfers, business trainings, and access to savings accounts, that help poor households move from requiring humanitarian assistance to earning sustainable livelihoods for themselves.

We know from randomized evaluations spanning 20 countries that this model provides vulnerable households with the tools they need to increase their economic well-being in both the short-run and longer term. We even saw in a study of one of our own programs in Uganda that, for every single dollar that we spent, households saw over four times the return in economic benefits. Results like this, and the broader body of evidence, are why we are scaling this approach in our own programming.

This is just one example of the power of cost-effectiveness evidence to improve lives around the world.

Even just 10 years ago, my predecessors at USAID wouldn't have had the opportunity that I do today to commit the agency to infusing rigorous evidence more broadly and deeply across all our programming. The evidence simply didn't exist at the scale that it does today. Over the past decade, the number of impact evaluations in low- and middle-income countries has quadrupled, expanding from 2,000 studies in 2014 to more than 8,000 today.

That body of evidence is an incredible resource, but of course, it's not perfect. It can answer some questions more conclusively than others. But it often lacks good cost data to help us understand not just impact but impact per dollar. And data collection often doesn't extend long enough to know how the impact and cost-effectiveness of a program might change over time.

Nevertheless, with all its flaws, we have a moral imperative, a duty to the communities we seek to serve, to more consistently and systematically draw on this vast, and ever-growing, body of evidence as we design and implement our programs.

It's only in this way that we can increasingly deliver as much impact as possible for every dollar we spend. And where that evidence is imperfect, we equally have a duty to strengthen that evidence by conducting more impact evaluations of our own programs to both inform our own decisions and contribute to the knowledge that others around the world can tap.

I want to acknowledge that this focus on cost-effectiveness can sometimes be uncomfortable. Following the evidence sometimes means we're going to stop investing in activities that are demonstrably "good", because the evidence shows we could make greater progress toward the same objectives through other approaches. It's hard to stop a program that is doing some good, but that's exactly what we need to do when we know we could achieve even more by working in a different way. So we need advocates both inside and outside the Agency, such as those here in CGD, to keep pushing us towards better.

Another barrier is that the very size of this evidence base can be a challenge. It's a great problem to have, but when there are literally hundreds of evaluations on a particular topic, sometimes with confusing or even contradictory results, the conclusions to take away from that body of literature may not be clear or easily accessible to our staff.

So since day one, building the institutional structures to synthesize existing cost-effectiveness evidence, encourage its use, and fill gaps where evidence is lacking has been a top priority for Administrator [Samantha] Power and me.

These two Position Papers that we're launching today are part of that institutionalization, articulating for our staff and our partners how they should approach cost-effectiveness, and underscoring the particularly cost-effective approach of direct monetary transfers.

Last year, we created a dedicated, independent Office of the Chief Economist at USAID. This Office embeds cost-effectiveness evidence into the institutional fabric of the Agency, with the backing of dedicated resources and staff. We brought in one of the world's leading development economists, Dean Karlan, as our Chief Economist to build and lead the office. Their north star is supporting the rest of the Agency in improving the impact per dollar of our programs.

Since the office opened just last July, it has grown to 30 staff, and they have already worked with dozens of teams across USAID to advise on how cost-effectiveness evidence can be applied. We've seen an enthusiastic response from the Agency workforce, who really understand the value that cost-effectiveness evidence can provide as they try to do the best they can with the resources they've got.

Impact evaluations help us answer important questions about the causal impact-per dollar-of development and humanitarian programs. And with enough evidence, we can also start to unpack why - to understand what program design tweaks, or contextual features, are necessary for a "good buy" to work its best.

Combining the evidence with deep contextual knowledge and field experience of our staff in Missions allows us to assess which "good" and "promising buys" are most likely to deliver results in that time and place. Thanks to the extraordinary partnership of teams across the Agency, we have already improved the cost-effectiveness of more than $1.5 billion in USAID programming.

So, what does all of this look like in practice?

The work of championing cost-effectiveness requires behavior change, doing things differently, ultimately being willing to upend the status quo and discontinuing programs that, even if they are doing some good, aren't delivering as much impact as evidence-based alternatives.

Let me share an example of where a Mission not only produced cutting-edge evidence on how much impact they could achieve through different approaches - they also applied the global body of evidence to fundamentally change how they, and their government partner, did business.

Around the world, many countries have students making their way through primary school without learning to read a single sentence. In Kenya, data showed that nearly 40 percent of students in Grade Two couldn't read a single word of grade-level text. USAID and its government partner launched the Primary Mathematics and Reading Initiative, which aimed to dramatically improve literacy for Kenyan children.

Our Kenya team was committed to investing in what works. So, they drew from evidence around the world suggesting that four investments together are important for achieving real gains in primary education: teacher training, providing books and materials, teacher mentoring, and the provision of structured lesson plans or guides. Combining those four elements is an example of a structured pedagogy program, and the evidence shows that combined, they can drive meaningful change in a child's learning.

Our Kenya Mission followed those results by implementing structured pedagogy into their educational approach through a program that operated at a national scale in nearly 25,000 schools. The team conducted an impact evaluation and cost-effectiveness study to investigate the program's impact per dollar, which helped contribute to the growing body of evidence on structured pedagogy.

Indeed, our own evidence demonstrated that structured pedagogy, this combination of educational elements, yields more impact per dollar spent than simply providing books or teacher training in isolation. Over the life of the program, the percentage of zero readers in English decreased by nearly 20 percent in Grade One, and by nearly 25 percent in Grade Two.

In a recent review of cost-effectiveness evidence, USAID found that structured pedagogy programs typically cost between $8 and $60 per child. And, among all of the basic education programs that have been tested in hundreds of studies in global evidence, structured pedagogy programs generate better-than-average improvements in children's literacy. This isn't just driven by one or two studies, it's a consistent finding across a number of contexts.

Compared to that, alternative approaches, such as providing teaching and learning materials in isolation, consistently show lower impact on literacy scores. There are some programs which are consistently found to have zero impact on student achievement, such as providing laptops or desktop computers without accompanying software or integration into the curriculum. And the costs of such approaches can be enormous, as much as $200 per child in some cases.

So, while the exact cost or the exact impact of structured pedagogy may vary from place to place, we're still very confident in saying that it tends to have a higher ratio of impacts to costs than most alternatives. This is what we're talking about, when we say it's a good buy.

You can even think of this kind of strong, consistently cost-effective approach as a benchmark intervention against which others can be compared.

USAID, of course, has a history with benchmarking, with which many of you in the room will no doubt be familiar. Over the last decade, USAID funded a landmark series of evaluations comparing direct monetary transfers to alternative approaches in a number of sectors and contexts. And full disclosure, I was involved in those benchmarking exercises as the COO of GiveDirectly, that you heard earlier, during several of those years.

Beyond the utility of a benchmark in a purely evaluative sense, the concept of a benchmark, or a hurdle to clear, is a compelling one in the development space. A benchmark can guide USAID to evidence-based, reliable interventions, or inspire us to innovate to beat the benchmark.

We want USAID staff looking at the full set of cost-effectiveness evidence when they do activity design and thinking about whether the alternatives under consideration will plausibly deliver more impact per dollar than a known quantity such as structured pedagogy.

USAID has made recent strides toward further integrating this kind of evidence into our decision-making. Notably, we've overhauled our program design and implementation policy to more strongly focus on using cost-effectiveness evidence, not just our own evidence, but also the much larger body of high-quality evidence generated by others around the world.

Let me turn now to direct monetary transfers, which we know to be a highly cost-effective and evidence based intervention that can help our dollars do more for more people. Moreover, they are a uniquely local form of assistance, putting resources directly into the hands of recipients, giving them agency and dignity to make their own decisions, to prioritize their own needs.

More than one hundred impact evaluations across 34 countries show that direct monetary transfers consistently drive strong impacts on food security, nutrition, and resilience; they can help workers earn higher income, they can help children attend school and set them up for a brighter future. USAID has contributed to this evidence base.

Impact evaluations of USAID's Direct Monetary Transfer programs in Malawi and Liberia showed families were more food secure even after transfers stopped. Many of these impacts continue to benefit families in the face of crises such as floods, drought, and disease, long after the transfers have ended.

A study in Uganda, conducted during the Covid-19 pandemic, showed that individuals who received transfers in 2008, twelve years earlier, still had a 17 percent higher income than peers who did not.

In Kenya, recipients of transfers prior to the pandemic were less hungry, less depressed, and less likely to report that a household member was sick, compared to those who had not received transfers.

In light of these benefits, USAID Missions such as Ghana, Kenya, and Malawi have been following the evidence and using Direct Monetary Transfers in their programming.

As one example: in Kenya, our Mission provided between $100 to $500 to local microenterprises. When one of the recipients, Doreen, used the transfers to expand her business and buy more from local producers, she more than doubled her sales from $20 to $50 a day. She continues to put some money back into growing her business and uses the rest to pay for her children's school fees. Doreen embodies what the evidence shows: when people's incomes and businesses grow, so do local economies.

The multiplier effect of direct monetary transfers is very real. One study, conducted outside of USAID, investigated the impact of transfers on markets in rural Kenya. In this case, large one-time transfers were given to poor households. The study tracked not only the income and investments of those who received transfers, which grew significantly, but also of the neighboring households and businesses in those communities.

The findings were incredible. For every $1 transferred, roughly $2.50 of additional economic activity was created. Local businesses generated more sales and had higher income. Even local households which had not received transfers had measurably higher income themselves, possibly because of greater labor demand in those communities.

The evidence on the impact of direct monetary transfers is sufficiently strong that USAID missions are increasingly embracing this tool as a development approach. Last year, for example, USAID Ghana was looking for opportunities to improve resilience among rural households during the lean season. We know from dozens of studies on this topic that larger transfers not only tide recipients through lean times, but also enable them to invest productively and accumulate assets. These effects on household effects and consumption are documented for years after transfers themselves have ended.

Based on this clear alignment of evidence for their objectives, the Ghana team partnered with the World Food Program to deliver $315 in direct monetary transfers to 35,000 farmers, including women and youth. Households receive these transfers through mobile money or e-transfers and are able to convert this into physical cash at a network of agents. Notably, the transfer size was larger than that typically used in humanitarian settings, which is more geared to providing immediate lifesaving assistance. $315 was chosen to give us the best chance at achieving our development objectives, including spurring economic growth, and improving resilience, nutrition, health and beyond, while still reaching tens of thousands of households.

For our implementing partners, many of whom are represented in this room, we encourage you to likewise expand the way you apply direct monetary transfers, using your operational experience from humanitarian settings while following the evidence to achieve impacts on development outcomes.

These are just a few examples of how USAID is using the proven, highly cost-effective approach of direct monetary transfers to improve the impact we have around the world.

So, where do we go from here?

Let me come back to a theme I started with, that we have a moral imperative to use each dollar entrusted to us as wisely and impactfully as possible.

Ruth Levine, a champion for cost-effectiveness who has served at both CGD and USAID, said it well when she stated that, "Empirical analysis is not a substitute for the value judgments that inform a theory of justice in any society. But empirical analysis is an essential complement to those value judgments, helping to turn the 'what we believe' into the 'what we do.'" Let me say that one more time, "empirical analysis is an essential complement to our value judgments."

At USAID, we are working toward a vision of knowing that we are delivering as much impact per dollar across our roughly $30 billion annual budget as we possibly can. Although USAID's budget is less than half a percent of the federal budget, it represents enormous potential for impact, and it is our duty to steward it wisely.

In the years ahead, this will mean that any USAID staff member designing or managing a program should be able to access, at their fingertips, insights from the global evidence base about which approaches are most cost-effective for the outcomes they are trying to advance. It will also mean USAID is a larger contributor to that global body of evidence, conducting more impact evaluations to help inform our own decisions and those of our partners.

In the near term, we're taking concrete steps to achieve this bold vision.

We're building a network of nearly 100 "Cost-Effectiveness Evidence Champions" embedded in teams around the Agency and supporting this group of pioneering staff to serve as cost-effectiveness ambassadors to their peers.

And, we're partnering with external experts on this journey. We've recently launched a new global platform with the Center for Effective Global Action at the University of California, Berkeley to provide world-class external technical support to USAID and its partners on cost-effectiveness. This is an important step towards harnessing the expertise of the global community as we work to embrace the evidence in all that we do at USAID.

Luckily, USAID is not on this journey alone. The development and humanitarian community as a whole is working to more meaningfully tap into high-quality impact evaluation and cost- effectiveness evidence to deliver more per dollar for the communities that we serve.

CGD's recent working group, that you heard about, on this highlights that many governments, implementers, and aid agencies are grappling with this same challenge of how to leverage insights from the global evidence base, and how to contribute to expanding and improving the evidence base.

This kind of collaboration and partnership will continue to be at the heart of our work to more fully integrate cost effectiveness evidence to improve lives around the world. The two position papers we are launching today represent USAID's work to identify "good buys" in which we can invest for maximum impact. And we know we are working within a larger evidence ecosystem of continuously evolving insights with the power to improve the way that we work.

Investing in cost effective, evidence-based approaches is essential to our work to close the global gap between development and humanitarian needs and solutions.

So, I thank all of you for your partnership on this journey that we are undertaking.

Thank you.

Public link

Please use the above public link if you want to share this noodl on another website.