Impact Evaluation
Impact evaluation is the branch of development economics that tries to answer a deceptively simple question: did this program actually work?
8 frameworks, from 1958 to today · Development Economics · Economics
- Cost-Benefit Analysis 1958-1985
- Social Experiment Tradition 1968-1990
- Theory-driven evaluation 1972-Present
- Quasi-experiment 1985-Present
- Structural estimation 1990-Present
- Randomized controlled trial 1997-Present
- Cost-Effectiveness Comparison 2000-Present
- External Validity and Scale-Up Evaluation 2010-Present
Overview
Impact evaluation is the branch of development economics that tries to answer a deceptively simple question: did this program actually work? Not "was it well funded," not "did the beneficiaries say they liked it," but whether the people it touched ended up better off than they would have been without it. The counterfactual—what would have happened anyway—is the whole problem1. You can never observe a person both receiving and not receiving a program at the same time, so the field's history is largely a history of ingenious ways to approximate that missing comparison1. What is at stake is enormous: governments and donors spend billions on schools, clinics, cash transfers, and roads, and the difference between a program that helps and one that merely spends money is invisible without credible evaluation.
Appraisal Before Evaluation: Cost-Benefit Analysis and the Project Cycle
The earliest systematic attempt to judge development projects came not from economists studying outcomes but from engineers and planners deciding whether to build. In the 1950s and 1960s, as multilateral lenders like the World Bank scaled up infrastructure lending, they needed a discipline for choosing among projects. The answer was cost-benefit analysis (CBA): enumerate a project's costs and benefits, convert them to a common monetary unit using shadow prices (adjusted prices meant to reflect true social value where markets are distorted—for instance, the social cost of employing an unemployed worker is near zero, not the wage), and compute a rate of return. The 1968 Manual of Industrial Project Analysis by Ian Little and James Mirrlees, and the 1972 UNIDO Guidelines for Project Evaluation by Partha Dasgupta, Amartya Sen, and Stephen Marglin, codified competing approaches. The 1975 World Bank study by Lyn Squire and Herman van der Tak became the operational standard.
CBA's limitation was structural: it evaluated projects ex ante, on paper, using assumptions about how benefits would flow. It could not tell you, after the fact, whether the clinic built had actually improved health. The gap between the projected and realized rate of return was often large, and nobody had a rigorous way to measure it.
The Social Experiment Tradition
In parallel, a different lineage grew out of American social policy. In the late 1960s and 1970s, the United States ran large social experiments—randomized trials of negative income tax schemes, housing allowances, and health insurance—precisely because policymakers wanted to know what would happen before national rollout. The New Jersey Income Maintenance Experiment (1968–1972) is the landmark: families were randomly assigned to different guaranteed-income levels, and researchers measured labor supply responses. These experiments established the machinery of randomized evaluation—random assignment, treatment and control groups, intention-to-treat analysis—outside medicine.
The evaluation profession also developed theory-based evaluation in this era. Rather than asking only "did it work," evaluators like Carol Weiss (who wrote influentially on evaluation research from the 1970s) and later Michael Quinn Patton argued for articulating the program theory: the causal chain from inputs to activities to outcomes, and testing whether each link held. This mattered because a null result could mean the program failed or that it was never implemented as designed. Theory-based approaches—logical frameworks, realist evaluation—became standard in the evaluation departments of aid agencies, even as they remained softer than the experimental ideal.
The Credibility Revolution Arrives in Development
Through the 1980s and 1990s, development economics evaluated programs mostly through quasi-experimental designs: comparing participants to non-participants using statistical controls, or exploiting natural variation. The workhorse was the difference-in-differences estimator—compare the change over time in a treated group to the change in an untreated group, assuming that absent the program both would have moved in parallel12. Other tools included instrumental variables (finding an external nudge that affects participation but nothing else) and regression discontinuity (comparing people just above and just below a cutoff that determines eligibility)13. These methods were powerful but fragile: their credibility rested on assumptions—parallel trends, exclusion restrictions—that were often untestable and sometimes implausible. A famous critique by James Heckman and colleagues argued that many evaluations of job-training programs were biased because participants differed from non-participants in unmeasured ways.
The turning point came in the mid-1990s, when randomized controlled trials (RCTs) migrated from American social experiments into development. Random assignment solves the counterfactual problem directly: if assignment is truly random, treatment and control groups are statistically identical on average, so any difference in outcomes is attributable to the program1. The pioneering case was PROGRESA (later Oportunidades), Mexico's conditional cash transfer program launched in 1997. Because the program could not reach all eligible villages at once, the government randomized the order of phase-in, and evaluators—working with researchers including Paul Gertler—compared early and late villages. The results were striking: the transfers raised school enrollment and improved health, and the evaluation helped secure the program's political survival and global replication.
The RCT Movement and Its Institutions
The PROGRESA evaluation demonstrated that randomization was feasible in real government programs, not just in rich-country laboratories. In 2003, Abhijit Banerjee, Esther Duflo, and Sendhil Mullainathan founded the Abdul Latif Jameel Poverty Action Lab (J-PAL) at MIT, which became the movement's institutional hub—funding trials, training researchers, and advocating for evidence-based policy. Michael Kremer at Harvard had already been running randomized trials in Kenyan schools (textbooks, deworming), and the deworming work by Miguel and Kremer (2004) became a celebrated—and later fiercely contested—example of an RCT finding with large claimed spillovers.
The movement's logic was cumulative: rather than evaluate whole programs, break them into components and test which specific design features work. Does giving free uniforms raise attendance more than cash? Do teacher incentives tied to test scores improve learning or just teaching to the test? Hundreds of trials followed across education, health, microfinance, agriculture, and governance. The approach won the 2019 Nobel Prize for Banerjee, Duflo, and Kremer4.
The External Validity Problem
As the stock of RCT results grew, a new problem sharpened: external validity, or whether a finding travels15. A program that worked in one Kenyan district might fail elsewhere because of different prices, institutions, or baseline conditions. The field's internal critics—notably Angus Deaton, who argued in a 2010 paper that RCTs answer narrow questions and cannot by themselves guide policy at scale—pressed this point. The response was a turn toward scale-up evaluation: testing whether effects persist when programs expand, and studying general equilibrium effects (what happens to wages or prices when a program goes from a pilot to a region). A cash transfer that raises one village's income may do little harm to local prices; the same transfer at national scale could move them. Researchers also began running multi-site trials and meta-analyses to map where effects replicate and where they don't.
Structural Modeling for Extrapolation
The other answer to external validity came from an older tradition: structural modeling6. Instead of estimating a single reduced-form effect, build a model of household or firm behavior—utility functions, production functions, constraints—estimate its parameters from data, and then simulate what would happen under new conditions. This lets a researcher ask counterfactuals never observed: what if the program were scaled up, prices changed, or it were applied to a different population? The approach descends from the structural econometrics tradition associated with Heckman and others, and in development it has been used to extrapolate from pilot results, to evaluate policies that cannot be randomized, and to study general equilibrium. Its weakness is that results depend on the model being correctly specified—a strong assumption—so structural and experimental approaches are often used as complements rather than rivals.
The Present
Impact evaluation today is a plural field. RCTs remain central but are no longer treated as the only credible design; the field has absorbed the lesson that quasi-experimental methods—difference-in-differences with modern refinements for staggered adoption, synthetic control (constructing a comparison unit from a weighted combination of untreated units), regression discontinuity—are indispensable where randomization is impossible or unethical. Theory-based evaluation persists in the practitioner world of aid agencies, often paired with experimental measurement of specific links. Cost-benefit analysis has been reinvigorated by the demand to know not just whether a program works but whether it is worth its cost, and cost-effectiveness comparisons across interventions are now a standard output. The open questions are the ones the history kept generating: how to move credibly from pilot to scale, how to evaluate system-level reforms that cannot be randomized, how to handle spillovers and equilibrium effects, and how to make evidence actually inform the decisions of governments with limited capacity. The counterfactual problem has not been solved; it has been surrounded by an increasingly sophisticated toolkit.
References
- Impact evaluation (introduction, Counterfactual evaluation designs, Quasi-experimental design, Estimation methods, Experimental approaches, Randomised control trials (RCTs)), Wikipedia
- Difference in differences, Wikipedia
- Regression discontinuity design, Wikipedia
- Abdul Latif Jameel Poverty Action Lab, Wikipedia
- External validity, Wikipedia
- Structural estimation, Wikipedia
The frameworks of Impact Evaluation
Each framework is a way of working the field has used: when it held, and the question it answers.
- 1958-1985
- 1972-Present
- 1985-Present
- 1990-Present
- 1997-Present
- 2000-Present
Cost-Effectiveness Comparison
Which program gives most value per dollar?
- 2010-Present
External Validity and Scale-Up Evaluation
Will this work elsewhere or at scale?
How the ideas connect
Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.
Randomized controlled trial grew out of Social Experiment Tradition
The machinery of randomized social experiments—random assignment, treatment and control groups, intention-to-treat—was carried into development by the PROGRESA evaluation in 1997, which randomized village phase-in.
Carried by New Jersey Income Maintenance Experiment; PROGRESA.
Randomized controlled trial reacted against Quasi-experiment
Randomized controlled trials in development arose as a response to the fragility of quasi-experimental assumptions such as parallel trends and exclusion restrictions, offering direct randomization as a more credible counterfactual.
Carried by PROGRESA; Michael Kremer.
External Validity and Scale-Up Evaluation reacted against Randomized controlled trial
The external validity and scale-up agenda arose from criticism that RCT results are narrow and may not travel; Deaton's 2010 critique and the response of multi-site trials and general equilibrium studies defined the new framework.
Carried by Angus Deaton.
External Validity and Scale-Up Evaluation influenced by Structural estimation
Structural modeling provided a tool for extrapolating experimental results to new settings by estimating behavioral parameters and simulating counterfactuals, which became one answer to the external validity problem.
Carried by James Heckman.
Cost-Effectiveness Comparison grew out of Randomized controlled trial
Once RCTs produced credible effect estimates, cost-effectiveness comparison grew out of them as a way to rank interventions by impact per dollar, using experimental effect sizes as inputs.
External Validity and Scale-Up Evaluation influenced by Quasi-experiment
Quasi-experimental methods such as difference-in-differences with staggered adoption were revived and refined within the scale-up agenda to study programs at regional or national scale where randomization is infeasible.
Beyond the field
Where Impact Evaluation's ideas came from in other fields, and what they led to there: the strongest 12 of 59 links.
Cost-Benefit Analysis influenced Cost-Effectiveness Analysis in Health Technology Assessment
Cost-benefit analysis supplied the basic framework of comparing costs and consequences in commensurate terms that health cost-effectiveness analysis adapted by using natural health-effect units instead of dollars.
Social Experiment Tradition influenced Program Design Turn in Social Insurance And Redistribution
The social experiment tradition of the late 1960s and 1970s, notably the negative income tax experiments, established randomised field trials of social policy that the program design turn built on and refined.
Theory-driven evaluation led to Impact Evaluation in Applied Theater Theory
Impact evaluation in the 1990s built directly on Chen and Rossi's theory-driven evaluation, adopting its program-theory and mechanism-based logic to explain how interventions produce effects. Theory-driven evaluation supplied the causal-modeling core that impact evaluation formalised with counterfactual methods.
Carried by Huey-Tsyh Chen; Peter Rossi.
Quasi-experiment led to Impact Evaluation in Applied Theater Theory
Impact evaluation's core designs are quasi-experiments: difference-in-differences, regression discontinuity and matching, developed in econometrics and imported into development evaluation in the 1990s and 2000s. The quasi-experiment is the methodological backbone of the field.
Randomized controlled trial led to Randomized controlled trial in Rural Development Economics
Development RCTs grew directly out of the earlier impact-evaluation tradition in economics, which used randomization and comparison groups to measure program effects. The development variant extended that approach to poverty interventions in low-income countries.
Randomized controlled trial influenced by Randomized Controlled Trial in Biostatistics
The randomised controlled trial as developed in biostatistics, with random allocation and intention-to-treat analysis, is the direct methodological ancestor of the RCT in development impact evaluation.
Carried by Austin Bradford Hill.
Cost-Effectiveness Comparison influenced by Cost-benefit analysis in Health Policy Evaluation
Cost-benefit analysis, developed for public investment appraisal from the 1950s, supplied the discounting, valuation and incremental-comparison techniques that health cost-effectiveness analysis adapted in the 1970s.
External Validity and Scale-Up Evaluation influenced by Randomized Controlled Trial Evaluation in Public Health Program Evaluation
The randomized controlled trial tradition in public health, codified from the 1948 streptomycin trial onward, supplied the randomization and comparison design that randomized impact evaluation in development adopted in the 2000s.
Carried by Austin Bradford Hill.
Cost-Benefit Analysis influenced Cost-effectiveness analysis in Health Economics
Cost-benefit analysis in impact evaluation supplied the practical evaluation techniques that cost-effectiveness analysis adapted to health programmes. The later framework took the comparison-of-alternatives method from it.
Cost-Benefit Analysis influenced Environmental Impact Assessment in Biodiversity And Conservation Law
Cost-benefit analysis, developed in economics and applied to water projects, supplied the quantitative comparison of gains and losses that environmental impact assessment adopted. The National Environmental Policy Act's impact statements borrowed cost-benefit framing to weigh environmental costs against development benefits.
Cost-Benefit Analysis influenced Environmental Impact Assessment in Environmental Risk Assessment
Cost-benefit analysis provided the economic valuation techniques that EIA statements used to weigh environmental costs against project benefits. The 1960s economics of project appraisal was carried into environmental impact statements as a standard component.
Cost-Benefit Analysis influenced Environmental impact assessment in Infrastructure Planning
Cost-benefit analysis supplied the monetised comparison of gains and losses that environmental impact assessment adopted for weighing project alternatives.

Impact Evaluation is a planet in Nousfarer
Its city has a tower for each of these 8 frameworks. Land, walk in, and climb one concept at a time.
Land on Impact EvaluationRelated fields
Fields its ideas are linked to
- Public Health Program Evaluation 10 links
- Applied Theater Theory 3 links
- Health Technology Assessment 3 links
- Global Health Metrics And Evaluation 3 links
- Infrastructure Planning 2 links
- Poverty Economics 2 links
- Structural Estimation 2 links
- Health Policy Evaluation 2 links
- Comparative Effectiveness Research 2 links
- Health Services Research 2 links
- Epidemiological Methods 2 links
- Decision And Risk Analysis 2 links
More in Development Economics
- Development Economics
- Development Finance
- Human Capital Development
- Institutional Development
- Political Economy Of Development
- Poverty Economics
- Structural Transformation
See where Impact Evaluation sits on the Map of Human Knowledge