Impact Evaluation

Impact evaluation is the branch of development economics that tries to answer a deceptively simple question: did this program actually work?

8 frameworks, from 1958 to today · Development Economics · Economics

still a working view ended · each leads to its entry below

Overview

Impact evaluation is the branch of development economics that tries to answer a deceptively simple question: did this program actually work? Not "was it well funded," not "did the beneficiaries say they liked it," but whether the people it touched ended up better off than they would have been without it. The counterfactual—what would have happened anyway—is the whole problem1. You can never observe a person both receiving and not receiving a program at the same time, so the field's history is largely a history of ingenious ways to approximate that missing comparison1. What is at stake is enormous: governments and donors spend billions on schools, clinics, cash transfers, and roads, and the difference between a program that helps and one that merely spends money is invisible without credible evaluation.

Appraisal Before Evaluation: Cost-Benefit Analysis and the Project Cycle

The earliest systematic attempt to judge development projects came not from economists studying outcomes but from engineers and planners deciding whether to build. In the 1950s and 1960s, as multilateral lenders like the World Bank scaled up infrastructure lending, they needed a discipline for choosing among projects. The answer was cost-benefit analysis (CBA): enumerate a project's costs and benefits, convert them to a common monetary unit using shadow prices (adjusted prices meant to reflect true social value where markets are distorted—for instance, the social cost of employing an unemployed worker is near zero, not the wage), and compute a rate of return. The 1968 Manual of Industrial Project Analysis by Ian Little and James Mirrlees, and the 1972 UNIDO Guidelines for Project Evaluation by Partha Dasgupta, Amartya Sen, and Stephen Marglin, codified competing approaches. The 1975 World Bank study by Lyn Squire and Herman van der Tak became the operational standard.

CBA's limitation was structural: it evaluated projects ex ante, on paper, using assumptions about how benefits would flow. It could not tell you, after the fact, whether the clinic built had actually improved health. The gap between the projected and realized rate of return was often large, and nobody had a rigorous way to measure it.

The Social Experiment Tradition

In parallel, a different lineage grew out of American social policy. In the late 1960s and 1970s, the United States ran large social experiments—randomized trials of negative income tax schemes, housing allowances, and health insurance—precisely because policymakers wanted to know what would happen before national rollout. The New Jersey Income Maintenance Experiment (1968–1972) is the landmark: families were randomly assigned to different guaranteed-income levels, and researchers measured labor supply responses. These experiments established the machinery of randomized evaluation—random assignment, treatment and control groups, intention-to-treat analysis—outside medicine.

The evaluation profession also developed theory-based evaluation in this era. Rather than asking only "did it work," evaluators like Carol Weiss (who wrote influentially on evaluation research from the 1970s) and later Michael Quinn Patton argued for articulating the program theory: the causal chain from inputs to activities to outcomes, and testing whether each link held. This mattered because a null result could mean the program failed or that it was never implemented as designed. Theory-based approaches—logical frameworks, realist evaluation—became standard in the evaluation departments of aid agencies, even as they remained softer than the experimental ideal.

The Credibility Revolution Arrives in Development

Through the 1980s and 1990s, development economics evaluated programs mostly through quasi-experimental designs: comparing participants to non-participants using statistical controls, or exploiting natural variation. The workhorse was the difference-in-differences estimator—compare the change over time in a treated group to the change in an untreated group, assuming that absent the program both would have moved in parallel12. Other tools included instrumental variables (finding an external nudge that affects participation but nothing else) and regression discontinuity (comparing people just above and just below a cutoff that determines eligibility)13. These methods were powerful but fragile: their credibility rested on assumptions—parallel trends, exclusion restrictions—that were often untestable and sometimes implausible. A famous critique by James Heckman and colleagues argued that many evaluations of job-training programs were biased because participants differed from non-participants in unmeasured ways.

The turning point came in the mid-1990s, when randomized controlled trials (RCTs) migrated from American social experiments into development. Random assignment solves the counterfactual problem directly: if assignment is truly random, treatment and control groups are statistically identical on average, so any difference in outcomes is attributable to the program1. The pioneering case was PROGRESA (later Oportunidades), Mexico's conditional cash transfer program launched in 1997. Because the program could not reach all eligible villages at once, the government randomized the order of phase-in, and evaluators—working with researchers including Paul Gertler—compared early and late villages. The results were striking: the transfers raised school enrollment and improved health, and the evaluation helped secure the program's political survival and global replication.

The RCT Movement and Its Institutions

The PROGRESA evaluation demonstrated that randomization was feasible in real government programs, not just in rich-country laboratories. In 2003, Abhijit Banerjee, Esther Duflo, and Sendhil Mullainathan founded the Abdul Latif Jameel Poverty Action Lab (J-PAL) at MIT, which became the movement's institutional hub—funding trials, training researchers, and advocating for evidence-based policy. Michael Kremer at Harvard had already been running randomized trials in Kenyan schools (textbooks, deworming), and the deworming work by Miguel and Kremer (2004) became a celebrated—and later fiercely contested—example of an RCT finding with large claimed spillovers.

The movement's logic was cumulative: rather than evaluate whole programs, break them into components and test which specific design features work. Does giving free uniforms raise attendance more than cash? Do teacher incentives tied to test scores improve learning or just teaching to the test? Hundreds of trials followed across education, health, microfinance, agriculture, and governance. The approach won the 2019 Nobel Prize for Banerjee, Duflo, and Kremer4.

The External Validity Problem

As the stock of RCT results grew, a new problem sharpened: external validity, or whether a finding travels15. A program that worked in one Kenyan district might fail elsewhere because of different prices, institutions, or baseline conditions. The field's internal critics—notably Angus Deaton, who argued in a 2010 paper that RCTs answer narrow questions and cannot by themselves guide policy at scale—pressed this point. The response was a turn toward scale-up evaluation: testing whether effects persist when programs expand, and studying general equilibrium effects (what happens to wages or prices when a program goes from a pilot to a region). A cash transfer that raises one village's income may do little harm to local prices; the same transfer at national scale could move them. Researchers also began running multi-site trials and meta-analyses to map where effects replicate and where they don't.

Structural Modeling for Extrapolation

The other answer to external validity came from an older tradition: structural modeling6. Instead of estimating a single reduced-form effect, build a model of household or firm behavior—utility functions, production functions, constraints—estimate its parameters from data, and then simulate what would happen under new conditions. This lets a researcher ask counterfactuals never observed: what if the program were scaled up, prices changed, or it were applied to a different population? The approach descends from the structural econometrics tradition associated with Heckman and others, and in development it has been used to extrapolate from pilot results, to evaluate policies that cannot be randomized, and to study general equilibrium. Its weakness is that results depend on the model being correctly specified—a strong assumption—so structural and experimental approaches are often used as complements rather than rivals.

The Present

Impact evaluation today is a plural field. RCTs remain central but are no longer treated as the only credible design; the field has absorbed the lesson that quasi-experimental methods—difference-in-differences with modern refinements for staggered adoption, synthetic control (constructing a comparison unit from a weighted combination of untreated units), regression discontinuity—are indispensable where randomization is impossible or unethical. Theory-based evaluation persists in the practitioner world of aid agencies, often paired with experimental measurement of specific links. Cost-benefit analysis has been reinvigorated by the demand to know not just whether a program works but whether it is worth its cost, and cost-effectiveness comparisons across interventions are now a standard output. The open questions are the ones the history kept generating: how to move credibly from pilot to scale, how to evaluate system-level reforms that cannot be randomized, how to handle spillovers and equilibrium effects, and how to make evidence actually inform the decisions of governments with limited capacity. The counterfactual problem has not been solved; it has been surrounded by an increasingly sophisticated toolkit.

References

  1. Impact evaluation (introduction, Counterfactual evaluation designs, Quasi-experimental design, Estimation methods, Experimental approaches, Randomised control trials (RCTs)), Wikipedia
  2. Difference in differences, Wikipedia
  3. Regression discontinuity design, Wikipedia
  4. Abdul Latif Jameel Poverty Action Lab, Wikipedia
  5. External validity, Wikipedia
  6. Structural estimation, Wikipedia

The frameworks of Impact Evaluation

Each framework is a way of working the field has used: when it held, and the question it answers.

  1. 1958-1985
  2. 1968-1990
  3. 1972-Present

    Theory-driven evaluation

    Why did this program succeed or fail?

    Fly to its tower in Nousfarer

  4. 1985-Present

    Quasi-experiment

    Did the program cause the change?

    Fly to its tower in Nousfarer

  5. 1990-Present

    Structural estimation

    What would happen under new conditions?

    Fly to its tower in Nousfarer

  6. 1997-Present
  7. 2000-Present

    Cost-Effectiveness Comparison

    Which program gives most value per dollar?

    Fly to its tower in Nousfarer

  8. 2010-Present

How the ideas connect

Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.

Cost-Benefit AnalysisSocial Experiment TraditionTheory-driven evaluationQuasi-experimentStructural estimationRandomized controlled trialCost-Effectiveness ComparisonExternal Validity and Scale-Up Evalua…

Beyond the field

Where Impact Evaluation's ideas came from in other fields, and what they led to there: the strongest 12 of 59 links.

A city in Nousfarer: a red-carpeted stair climbing terraces to a crowned tower on an acropolis, more towers round it, forest and snowy mountains beyond.
A city in Nousfarer, the game played in the browser.

Impact Evaluation is a planet in Nousfarer

Its city has a tower for each of these 8 frameworks. Land, walk in, and climb one concept at a time.

Land on Impact Evaluation

Related fields

Fields its ideas are linked to

More in Development Economics

See where Impact Evaluation sits on the Map of Human Knowledge