- Atlas
- Mathematics
- Experimental Design
Experimental Design
Experimental design is the branch of statistics concerned with planning how data will be collected, so that the answers we want can actually be extracted from the data once they arrive.
11 frameworks, from 1919 to today · Statistics · Mathematics
- Randomized Block and Factorial Design 1919-Present
- Rubin causal model 1923-Present
- Split-Plot and Multilevel Design 1925-Present
- Fractional Factorial Design 1940-Present
- Randomized controlled trial 1948-Present
- Optimal experimental design 1950-Present
- Taguchi methods 1950-Present
- Response Surface Methodology 1951-Present
- Multi-armed bandit 1952-Present
- Bayesian Experimental Design 1956-Present
- A/B testing 2000-Present
Overview
Experimental design is the branch of statistics concerned with planning how data will be collected, so that the answers we want can actually be extracted from the data once they arrive1. Its central problem is that an experiment is a question posed to nature, and a badly posed question yields an answer that cannot be interpreted: a treatment may look effective only because it was tested on the best plots, or in the coolest month, or on patients who were already recovering. Experimental design is the discipline that arranges who or what receives which treatment, when, and how often, so that the effect of interest is separated from everything else that might mimic it. It is one of the most consequential branches of statistics, because the designs it produces underlie medicine, agriculture, industry, and the online experiments that shape what billions of people see each day.
Randomization, replication, and blocking: the founding triad
Before the twentieth century, experiments were usually arranged by judgment. A chemist varied one ingredient at a time; an agronomist compared a new fertilizer on a field he considered representative. The weakness is that one-at-a-time variation cannot reveal interactions—whether a fertilizer works only at a certain moisture—and judgmental allocation lets the experimenter's expectations leak into the assignment.
Ronald A. Fisher, working at Rothamsted Experimental Station in England from 1919, transformed this. His 1925 book Statistical Methods for Research Workers and his 1935 The Design of Experiments set out three ideas that remain the field's foundation. Replication means running each treatment more than once, so that natural variability can be measured rather than assumed1. Randomization means assigning treatments to experimental units by a chance mechanism, so that no systematic difference between groups can be attributed to the treatment; it also gives a probabilistic basis for inference. Blocking means grouping similar units together—plots in one field corner, litters of the same animals—and comparing treatments within blocks, removing the block-to-block variation from the error1.
Fisher's classic illustration is the lady tasting tea: a colleague claimed she could tell whether milk or tea had been poured first. Fisher showed that only a randomized arrangement of the cups, with all possible orders enumerated, could give a meaningful test of her claim. The same logic, scaled up, governs the agricultural field trials at Rothamsted, where Fisher analyzed decades of crop data and developed the analysis of variance alongside the designs.
Factorial and fractional factorial designs
A factorial design varies several factors at once, in all combinations, rather than one at a time12. With two factors at two levels each, that is four treatment combinations; with many factors the count explodes2. The payoff is that factorial designs estimate main effects (the average effect of one factor) and interactions (how the effect of one factor depends on another) with the same runs, and with greater precision than separate one-factor experiments.
The combinatorial burden was addressed by fractional factorial designs, which run only a carefully chosen subset of the full grid23. The idea, developed in the 1940s by statisticians including Cuthbert Daniel, Oscar Kempthorne, and later Box and Hunter, is that higher-order interactions are usually negligible, so a fraction can be chosen that keeps main effects and low-order interactions estimable while confounding the rest. The price is aliasing: some effects cannot be distinguished from others, and the design must be chosen so that the aliases are harmless4. These designs became the workhorse of industrial experimentation, where many candidate factors must be screened cheaply4.
Response surface methodology
Sometimes the goal is not to compare treatments but to find the settings that optimize a response—the temperature and concentration that maximize yield. Response surface methodology, developed by George Box and K. B. Wilson in 1951, fits a smooth model (often a quadratic) to the response over the factor space and uses it to locate an optimum5. The method proceeds in stages: a screening stage, then a steepest-ascent path toward the region of interest, then a local second-order design (such as a central composite design) to map the curvature near the peak. It turned experimentation from a comparison exercise into an optimization tool, and it remains standard in chemical engineering and process improvement.
Split-plot and multi-level designs
Not all experimental units are interchangeable, and some factors are hard to change. In a split-plot design, one factor is applied to large units (whole plots) and another to sub-units within them—irrigation to a whole field, fertilizer to sub-plots. The structure originated in agriculture, where a farmer cannot randomize irrigation at the plant level, but the same logic appears in industrial and clinical settings whenever one factor is applied at a coarser level than another. Such designs have two error terms, one for each level of randomization, and analyzing them as if they were fully randomized gives misleading precision. The broader family of multi-level or hierarchical designs generalizes this: units nested within units, with randomization at each level, and the analysis must respect the nesting.
Randomized controlled trials
The most socially consequential descendant of Fisher's triad is the randomized controlled trial (RCT). In medicine, the landmark is the 1948 streptomycin trial for pulmonary tuberculosis, run by the British Medical Research Council under Austin Bradford Hill, which randomly allocated patients to streptomycin or bed rest and compared outcomes. The trial demonstrated the drug's benefit and, just as importantly, established a template: a control group, random allocation, and blinded assessment. Hill's 1962 book Statistical Methods in Clinical and Preventive Medicine codified the approach. The RCT became the gold standard for causal evidence in medicine, and its logic—randomization breaks the link between treatment and confounders—is the same as Fisher's, transplanted to patients.
Potential outcomes and the causal turn
The RCT's authority rests on a framework formalized later. The potential outcomes framework, associated with Jerzy Neyman's 1923 paper on agricultural experiments and developed by Donald Rubin from the 1970s, defines the causal effect of a treatment on a unit as the difference between the outcome under treatment and the outcome under control—only one of which is ever observed. Randomization makes the treatment assignment independent of these potential outcomes, so the observed difference in means estimates the average causal effect. This framework clarified what experiments can and cannot establish, and it connected experimental design to observational causal inference, where the same logic is applied without randomization.
Optimal design theory
Fisher's designs are elegant and balanced, but they are not always possible or optimal. Design optimality theory, developed from the 1950s onward by Jack Kiefer and others, asks: given a model and a set of candidate experimental conditions, which allocation of observations minimizes the variance of the estimated parameters? Different criteria—D-optimality (maximizing the determinant of the information matrix), A-optimality (minimizing average variance), and others—give different answers, and the theory provides algorithms for constructing designs under constraints. It is the natural language for computer-generated designs, where the experimenter has an irregular region of feasible settings and cannot rely on classical balanced layouts.
Bayesian experimental design
A parallel tradition treats design as a decision problem under uncertainty. Bayesian experimental design, developed from the 1950s and 1960s (with early contributions by Dennis Lindley and later work by Kathryn Chaloner and Isabella Verdinelli), chooses the experiment that maximizes the expected information gained about the parameters, or the expected utility of the eventual decision. The Bayesian approach naturally handles prior knowledge and sequential experimentation, and it provides a unified way to compare designs that classical criteria treat separately. Its practical use grew with computing, since the required expectations are often evaluated by simulation.
Taguchi methods and robust design
In the 1950s and 1960s, Genichi Taguchi, working in Japanese industry, developed a philosophy of robust design: rather than merely optimizing a response, design products and processes so that performance is insensitive to uncontrollable variation (noise)67. Taguchi's methods use orthogonal arrays—fractional factorial designs under another name—and a signal-to-noise ratio to measure robustness1. His ideas were influential in Japanese manufacturing and spread to the West in the 1980s, though statisticians debated his analysis methods; the lasting contribution is the emphasis on designing for insensitivity to noise, which reshaped quality engineering.
A/B testing and multi-armed bandits
The internet made experiments cheap and continuous. A/B testing—randomly showing two versions of a web page, ad, or feature to different users and comparing outcomes—is the direct descendant of the RCT, applied at massive scale8. It became standard practice at technology companies in the 2000s and 2010s, and it raised new design questions: how long to run, how to handle multiple comparisons, and how to avoid peeking at results and stopping early.
A related but distinct approach is multi-armed bandit allocation, named after the problem of choosing among slot machines with unknown payoffs9. Instead of fixing the allocation in advance, a bandit algorithm adapts: it balances exploration (trying arms to learn their value) against exploitation (favoring the arm that looks best)9. The problem was formalized by Herbert Robbins in 1952, and modern algorithms (such as Thompson sampling and upper confidence bounds) are used in online advertising and recommendation systems where the cost of a bad allocation is real and ongoing. Bandits trade the clean inference of a fixed experiment for efficiency in decision-making, and the tension between the two is an active area of design research.
The present
Experimental design today is a family of approaches rather than a single method. Classical randomization, blocking, and factorial designs remain the backbone of agricultural, industrial, and clinical research. Optimal design theory supplies computer-generated designs for irregular settings10. Bayesian design handles prior information and sequential decisions11. The potential outcomes framework provides the causal language in which all of these are now discussed, and it links experimental design to the analysis of observational data12. Online experimentation has pushed the field toward adaptive and sequential methods, and the multi-armed bandit problem has become a bridge between statistics and machine learning. Open questions include how to design experiments when units are networked and interfere with one another, how to combine many small experiments into reliable knowledge, and how to make adaptive allocation methods yield valid inference as well as good decisions.
References
- Design of experiments (introduction, Fisher's principles, Experimental designs after Fisher), Wikipedia
- Factorial experiment, Wikipedia
- Fractional factorial design, Wikipedia
- Aliasing (factorial experiments), Wikipedia
- Response surface methodology, Wikipedia
- Taguchi methods, Wikipedia
- Genichi Taguchi, Wikipedia
- A/B testing, Wikipedia
- Multi-armed bandit, Wikipedia
- Optimal experimental design, Wikipedia
- Bayesian experimental design, Wikipedia
- Rubin causal model, Wikipedia
The frameworks of Experimental Design
Each framework is a way of working the field has used: when it held, and the question it answers.
- 1919-Present
- 1923-Present
- 1925-Present
- 1940-Present
- 1948-Present
- 1950-Present
- 1950-Present
- 1951-Present
- 1952-Present
- 1956-Present
- 2000-Present
How the ideas connect
Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.
Randomized controlled trial influenced by Randomized Block and Factorial Design
The rigorous application of randomization and control groups, pioneered by Fisher, was directly translated into the methodology of randomized controlled trials in medicine.
Carried by Ronald A. Fisher; Austin Bradford Hill.
Split-Plot and Multilevel Design influenced by Randomized Block and Factorial Design
The need to account for non-interchangeable experimental units, a concept arising from practical agricultural limitations, led to the development of designs accommodating nested structures.
Carried by Ronald A. Fisher.
Fractional Factorial Design grew out of Randomized Block and Factorial Design
The combinatorial explosion inherent in full factorial designs led to the development of fractional factorial designs to screen factors efficiently using subsets of runs.
Response Surface Methodology grew out of Randomized Block and Factorial Design
Response Surface Methodology builds upon the systematic variation of factors and the use of statistical modeling to map responses, extending the comparison exercise into optimization.
Carried by George Box; K. B. Wilson.
A/B testing influenced by Rubin causal model
The causal language and focus on treatment effects established by the potential outcomes framework underpin the interpretation of results in modern online experiments.
Taguchi methods influenced by Fractional Factorial Design
Taguchi methods employed orthogonal arrays, which are a specific, highly structured form of fractional factorial design.
A/B testing grew out of Multi-armed bandit
The sequential decision-making nature of online experimentation directly maps onto the problem structure solved by multi-armed bandit algorithms.
A/B testing influenced by Bayesian Experimental Design
The Bayesian approach provides a natural framework for sequential decision-making and updating beliefs, which is the core mechanism of modern online experimentation.
Beyond the field
Where Experimental Design's ideas came from in other fields, and what they led to there: the strongest 12 of 48 links.
Randomized Block and Factorial Design influenced Randomized Controlled Trial in Biostatistics
Fisher's randomized block and factorial designs, developed at Rothamsted from the 1920s, supplied the randomization and replication logic that the first MRC streptomycin trial adopted for clinical use.
Carried by Ronald A. Fisher; The Design of Experiments (1935).
Randomized Block and Factorial Design influenced Randomized controlled trial in Clinical Medicine
Randomised block and factorial designs developed in agricultural experimentation by Fisher supplied the randomisation and blocking techniques that the clinical randomised controlled trial adopted. The streptomycin trial's random allocation drew directly on this statistical design tradition.
Carried by Ronald A. Fisher; Austin Bradford Hill.
Split-Plot and Multilevel Design grew out of Analysis of Variance and Experimental Design in Regression Analysis
Split-plot and multilevel designs grew directly out of Fisher's analysis of variance and experimental design, extending the error-stratification of ANOVA to experiments with more than one randomisation unit. The split-plot analysis is a branch of the ANOVA programme.
Carried by Ronald A. Fisher; Frank Yates.
Fractional Factorial Design grew out of Analysis of Variance and Experimental Design in Regression Analysis
Fractional factorial design grew directly out of Fisher's factorial design and analysis of variance, extending it by deliberately confounding higher-order interactions so that only a fraction of treatment combinations need be run. The analysis remains an ANOVA of the selected fraction.
Carried by Ronald A. Fisher; Finney.
Randomized controlled trial grew out of Randomized Experimental Design in Causal Inference
The randomized controlled trial grew directly out of Fisher's randomized experimental design, applying random allocation and significance testing to clinical comparisons. The 1948 streptomycin trial used these design principles.
Carried by Austin Bradford Hill; Medical Research Council.
Randomized controlled trial influenced Experimental and Quasi-Experimental Design in Public Health Program Evaluation
The randomized controlled trial as an experimental design supplied the assignment and comparison procedures that experimental and quasi-experimental evaluation formalised for programs.
Carried by Donald T. Campbell; Julian C. Stanley.
Randomized controlled trial led to Stepped-wedge trial in Implementation And Dissemination Research
The stepped-wedge trial is a variant of the randomized controlled trial in which clusters cross over to intervention in sequence. It grew directly out of randomized experimental design, retaining randomisation while changing the allocation scheme.
Taguchi methods influenced by Design of experiments in Design Theory
Taguchi's orthogonal arrays and fractional factorial plans descend directly from the statistical design of experiments developed by Fisher and his school; Taguchi took the combinatorial design techniques and applied them to industrial quality problems.
Carried by Genichi Taguchi; Ronald A. Fisher.
Taguchi methods influenced by Design of Experiments in Quality Engineering
Taguchi built his robust design methods on classical design of experiments, adopting orthogonal arrays and factorial designs from Fisher's tradition and adapting them to quality engineering in Japan from the late 1940s.
Carried by Genichi Taguchi; Ronald A. Fisher.
Taguchi methods influenced Design for Reliability and Robust Design in Reliability Engineering
Taguchi methods supplied robust design its core tools: orthogonal arrays, parameter design and the signal-to-noise ratio, adopted into design for reliability to make products insensitive to noise factors. Genichi Taguchi's techniques entered reliability practice largely through this route.
Carried by Genichi Taguchi.
Response Surface Methodology grew out of Design of experiments in Design Theory
Response surface methodology is a direct branch of statistical design of experiments: Box and Wilson's 1951 paper built on factorial and fractional factorial designs to develop sequential experimentation for optimizing response surfaces.
Carried by George E. P. Box; K. B. Wilson.
Response Surface Methodology grew out of Design of Experiments in Quality Engineering
Response surface methodology is a specialized development of classical design of experiments, extending factorial designs into sequential optimization procedures for industrial processes.
Carried by George E. P. Box; K. B. Wilson.

Experimental Design is a planet in Nousfarer
Its city has a tower for each of these 11 frameworks. Land, walk in, and climb one concept at a time.
Land on Experimental DesignRelated fields
Fields its ideas are linked to
- Design Theory 4 links
- Causal Inference 4 links
- Regression Analysis 4 links
- Quality Engineering 3 links
- Combinatorics 3 links
- Biostatistics 3 links
- Statistical Inference 2 links
- Agronomy 1 link
- Crop Physiology 1 link
- Cropping Systems 1 link
- Reinforcement Learning 1 link
- Development Economics 1 link
More in Statistics
- Bayesian Statistics
- Causal Inference
- Multivariate Statistics
- Nonparametric Statistics
- Regression Analysis
- Spatial Statistics
- Statistical Computing
- Statistical Inference
- Statistical Learning
- Statistics
- Survey Sampling
- Survival Analysis
- Time Series Analysis
See where Experimental Design sits on the Map of Human Knowledge