Experimental Design

Experimental design is the branch of statistics concerned with planning how data will be collected, so that the answers we want can actually be extracted from the data once they arrive.

11 frameworks, from 1919 to today · Statistics · Mathematics

still a working view ended · each leads to its entry below

Overview

Experimental design is the branch of statistics concerned with planning how data will be collected, so that the answers we want can actually be extracted from the data once they arrive1. Its central problem is that an experiment is a question posed to nature, and a badly posed question yields an answer that cannot be interpreted: a treatment may look effective only because it was tested on the best plots, or in the coolest month, or on patients who were already recovering. Experimental design is the discipline that arranges who or what receives which treatment, when, and how often, so that the effect of interest is separated from everything else that might mimic it. It is one of the most consequential branches of statistics, because the designs it produces underlie medicine, agriculture, industry, and the online experiments that shape what billions of people see each day.

Randomization, replication, and blocking: the founding triad

Before the twentieth century, experiments were usually arranged by judgment. A chemist varied one ingredient at a time; an agronomist compared a new fertilizer on a field he considered representative. The weakness is that one-at-a-time variation cannot reveal interactions—whether a fertilizer works only at a certain moisture—and judgmental allocation lets the experimenter's expectations leak into the assignment.

Ronald A. Fisher, working at Rothamsted Experimental Station in England from 1919, transformed this. His 1925 book Statistical Methods for Research Workers and his 1935 The Design of Experiments set out three ideas that remain the field's foundation. Replication means running each treatment more than once, so that natural variability can be measured rather than assumed1. Randomization means assigning treatments to experimental units by a chance mechanism, so that no systematic difference between groups can be attributed to the treatment; it also gives a probabilistic basis for inference. Blocking means grouping similar units together—plots in one field corner, litters of the same animals—and comparing treatments within blocks, removing the block-to-block variation from the error1.

Fisher's classic illustration is the lady tasting tea: a colleague claimed she could tell whether milk or tea had been poured first. Fisher showed that only a randomized arrangement of the cups, with all possible orders enumerated, could give a meaningful test of her claim. The same logic, scaled up, governs the agricultural field trials at Rothamsted, where Fisher analyzed decades of crop data and developed the analysis of variance alongside the designs.

Factorial and fractional factorial designs

A factorial design varies several factors at once, in all combinations, rather than one at a time12. With two factors at two levels each, that is four treatment combinations; with many factors the count explodes2. The payoff is that factorial designs estimate main effects (the average effect of one factor) and interactions (how the effect of one factor depends on another) with the same runs, and with greater precision than separate one-factor experiments.

The combinatorial burden was addressed by fractional factorial designs, which run only a carefully chosen subset of the full grid23. The idea, developed in the 1940s by statisticians including Cuthbert Daniel, Oscar Kempthorne, and later Box and Hunter, is that higher-order interactions are usually negligible, so a fraction can be chosen that keeps main effects and low-order interactions estimable while confounding the rest. The price is aliasing: some effects cannot be distinguished from others, and the design must be chosen so that the aliases are harmless4. These designs became the workhorse of industrial experimentation, where many candidate factors must be screened cheaply4.

Response surface methodology

Sometimes the goal is not to compare treatments but to find the settings that optimize a response—the temperature and concentration that maximize yield. Response surface methodology, developed by George Box and K. B. Wilson in 1951, fits a smooth model (often a quadratic) to the response over the factor space and uses it to locate an optimum5. The method proceeds in stages: a screening stage, then a steepest-ascent path toward the region of interest, then a local second-order design (such as a central composite design) to map the curvature near the peak. It turned experimentation from a comparison exercise into an optimization tool, and it remains standard in chemical engineering and process improvement.

Split-plot and multi-level designs

Not all experimental units are interchangeable, and some factors are hard to change. In a split-plot design, one factor is applied to large units (whole plots) and another to sub-units within them—irrigation to a whole field, fertilizer to sub-plots. The structure originated in agriculture, where a farmer cannot randomize irrigation at the plant level, but the same logic appears in industrial and clinical settings whenever one factor is applied at a coarser level than another. Such designs have two error terms, one for each level of randomization, and analyzing them as if they were fully randomized gives misleading precision. The broader family of multi-level or hierarchical designs generalizes this: units nested within units, with randomization at each level, and the analysis must respect the nesting.

Randomized controlled trials

The most socially consequential descendant of Fisher's triad is the randomized controlled trial (RCT). In medicine, the landmark is the 1948 streptomycin trial for pulmonary tuberculosis, run by the British Medical Research Council under Austin Bradford Hill, which randomly allocated patients to streptomycin or bed rest and compared outcomes. The trial demonstrated the drug's benefit and, just as importantly, established a template: a control group, random allocation, and blinded assessment. Hill's 1962 book Statistical Methods in Clinical and Preventive Medicine codified the approach. The RCT became the gold standard for causal evidence in medicine, and its logic—randomization breaks the link between treatment and confounders—is the same as Fisher's, transplanted to patients.

Potential outcomes and the causal turn

The RCT's authority rests on a framework formalized later. The potential outcomes framework, associated with Jerzy Neyman's 1923 paper on agricultural experiments and developed by Donald Rubin from the 1970s, defines the causal effect of a treatment on a unit as the difference between the outcome under treatment and the outcome under control—only one of which is ever observed. Randomization makes the treatment assignment independent of these potential outcomes, so the observed difference in means estimates the average causal effect. This framework clarified what experiments can and cannot establish, and it connected experimental design to observational causal inference, where the same logic is applied without randomization.

Optimal design theory

Fisher's designs are elegant and balanced, but they are not always possible or optimal. Design optimality theory, developed from the 1950s onward by Jack Kiefer and others, asks: given a model and a set of candidate experimental conditions, which allocation of observations minimizes the variance of the estimated parameters? Different criteria—D-optimality (maximizing the determinant of the information matrix), A-optimality (minimizing average variance), and others—give different answers, and the theory provides algorithms for constructing designs under constraints. It is the natural language for computer-generated designs, where the experimenter has an irregular region of feasible settings and cannot rely on classical balanced layouts.

Bayesian experimental design

A parallel tradition treats design as a decision problem under uncertainty. Bayesian experimental design, developed from the 1950s and 1960s (with early contributions by Dennis Lindley and later work by Kathryn Chaloner and Isabella Verdinelli), chooses the experiment that maximizes the expected information gained about the parameters, or the expected utility of the eventual decision. The Bayesian approach naturally handles prior knowledge and sequential experimentation, and it provides a unified way to compare designs that classical criteria treat separately. Its practical use grew with computing, since the required expectations are often evaluated by simulation.

Taguchi methods and robust design

In the 1950s and 1960s, Genichi Taguchi, working in Japanese industry, developed a philosophy of robust design: rather than merely optimizing a response, design products and processes so that performance is insensitive to uncontrollable variation (noise)67. Taguchi's methods use orthogonal arrays—fractional factorial designs under another name—and a signal-to-noise ratio to measure robustness1. His ideas were influential in Japanese manufacturing and spread to the West in the 1980s, though statisticians debated his analysis methods; the lasting contribution is the emphasis on designing for insensitivity to noise, which reshaped quality engineering.

A/B testing and multi-armed bandits

The internet made experiments cheap and continuous. A/B testing—randomly showing two versions of a web page, ad, or feature to different users and comparing outcomes—is the direct descendant of the RCT, applied at massive scale8. It became standard practice at technology companies in the 2000s and 2010s, and it raised new design questions: how long to run, how to handle multiple comparisons, and how to avoid peeking at results and stopping early.

A related but distinct approach is multi-armed bandit allocation, named after the problem of choosing among slot machines with unknown payoffs9. Instead of fixing the allocation in advance, a bandit algorithm adapts: it balances exploration (trying arms to learn their value) against exploitation (favoring the arm that looks best)9. The problem was formalized by Herbert Robbins in 1952, and modern algorithms (such as Thompson sampling and upper confidence bounds) are used in online advertising and recommendation systems where the cost of a bad allocation is real and ongoing. Bandits trade the clean inference of a fixed experiment for efficiency in decision-making, and the tension between the two is an active area of design research.

The present

Experimental design today is a family of approaches rather than a single method. Classical randomization, blocking, and factorial designs remain the backbone of agricultural, industrial, and clinical research. Optimal design theory supplies computer-generated designs for irregular settings10. Bayesian design handles prior information and sequential decisions11. The potential outcomes framework provides the causal language in which all of these are now discussed, and it links experimental design to the analysis of observational data12. Online experimentation has pushed the field toward adaptive and sequential methods, and the multi-armed bandit problem has become a bridge between statistics and machine learning. Open questions include how to design experiments when units are networked and interfere with one another, how to combine many small experiments into reliable knowledge, and how to make adaptive allocation methods yield valid inference as well as good decisions.

References

  1. Design of experiments (introduction, Fisher's principles, Experimental designs after Fisher), Wikipedia
  2. Factorial experiment, Wikipedia
  3. Fractional factorial design, Wikipedia
  4. Aliasing (factorial experiments), Wikipedia
  5. Response surface methodology, Wikipedia
  6. Taguchi methods, Wikipedia
  7. Genichi Taguchi, Wikipedia
  8. A/B testing, Wikipedia
  9. Multi-armed bandit, Wikipedia
  10. Optimal experimental design, Wikipedia
  11. Bayesian experimental design, Wikipedia
  12. Rubin causal model, Wikipedia

The frameworks of Experimental Design

Each framework is a way of working the field has used: when it held, and the question it answers.

  1. 1919-Present
  2. 1923-Present
  3. 1925-Present
  4. 1940-Present
  5. 1948-Present
  6. 1950-Present
  7. 1950-Present
  8. 1951-Present
  9. 1952-Present
  10. 1956-Present
  11. 2000-Present

How the ideas connect

Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.

Randomized Block and Factorial DesignRubin causal modelSplit-Plot and Multilevel DesignFractional Factorial DesignRandomized controlled trialOptimal experimental designTaguchi methodsResponse Surface MethodologyMulti-armed banditBayesian Experimental DesignA/B testing

Beyond the field

Where Experimental Design's ideas came from in other fields, and what they led to there: the strongest 12 of 48 links.

A city in Nousfarer: a red-carpeted stair climbing terraces to a crowned tower on an acropolis, more towers round it, forest and snowy mountains beyond.
A city in Nousfarer, the game played in the browser.

Experimental Design is a planet in Nousfarer

Its city has a tower for each of these 11 frameworks. Land, walk in, and climb one concept at a time.

Land on Experimental Design

Related fields

Fields its ideas are linked to

More in Statistics

See where Experimental Design sits on the Map of Human Knowledge