- Atlas
- Mathematics
- Regression Analysis
Regression Analysis
Regression analysis is the part of statistics that asks how one quantity depends on others.
9 frameworks, from 1805 to today · Statistics · Mathematics
- Ordinary least squares 1805-Present
- Analysis of Variance and Experimental Design 1919-Present
- Non-linear least squares 1960-Present
- Robust Regression 1964-Present
- Regularization and Penalized Regression 1970-Present
- Generalized linear model 1972-Present
- Quantile Regression 1978-Present
- Resampling and Model Assessment 1979-Present
- Causal inference 1980-Present
Overview
Regression analysis is the part of statistics that asks how one quantity depends on others1. Given measurements of a response—a crop yield, a price, a patient's blood pressure—and of one or more inputs that may explain it—rainfall, supply, dosage—regression fits a relationship and uses it to predict, to estimate effects, and to judge how much of the variation the inputs account for1. It is the workhorse of applied statistics: most quantitative claims in medicine, economics, engineering, and the social sciences pass through a regression at some point. Its history is a sequence of attempts to make that fitting honest—to find estimates that are optimal under stated conditions, to check those conditions, and to keep the answers trustworthy when they fail.
Least Squares and the Birth of Fitting
The founding problem was astronomical. In the late eighteenth century, astronomers had more observations of a planet or comet than unknowns in its orbit, and the observations disagreed. Which orbit fits best? Adrien-Marie Legendre published in 1805 the rule now called least squares: choose the parameters that make the sum of squared differences between observed and predicted values as small as possible1. Carl Friedrich Gauss had used the method earlier, from 1795, and gave it a probabilistic justification in 1809: if errors are independent and follow a normal (bell-shaped) distribution, least squares gives the most probable parameter values. Squaring errors is not arbitrary—it is what the normal distribution implies.
The method spread because it was computable and defensible. By hand, least squares reduces to solving a system of linear equations, and the fitted line or plane has a closed form2. Gauss and others applied it to geodesy and astronomy; in the nineteenth century it became standard in surveying and in the new social sciences. Francis Galton, studying heredity in the 1880s, coined "regression" for the tendency of tall parents' children to be closer to average height than their parents—regression toward the mean—and introduced the correlation coefficient with Karl Pearson in the 1890s. The linear regression model, with its fitted coefficients, standard errors, and tests, was largely in place by the early twentieth century.
Analysis of Variance and Designed Experiments
A parallel line of work asked not how to fit a given set of data but how to collect data so that effects could be separated. Ronald Fisher, at Rothamsted Experimental Station from 1919, developed the analysis of variance (ANOVA): a way of partitioning total variation into parts attributable to treatments and to error, then testing whether treatment differences exceed what chance would produce. His 1925 Statistical Methods for Research Workers and 1935 The Design of Experiments made randomization, replication, and blocking standard practice. Randomization—assigning treatments by chance—was the key idea: it turns unmeasured confounders into random noise rather than systematic bias.
ANOVA and regression are two views of the same linear model. A designed experiment with categorical treatments is a regression on indicator variables; Fisher's variance decomposition is the regression sum of squares. This unity, made explicit in the mid-twentieth century, meant that the tools of experimental design and the tools of observational fitting could be taught together. George Box's work on response surface methodology in the 1950s extended designed experimentation to finding optimal operating conditions, and Box and colleagues' 1951 paper on non-normality began the study of how sensitive least squares is to its assumptions.
The Linear Model Codified
Between the 1920s and 1960s the linear regression model was formalized as a package: coefficients estimated by least squares, standard errors from the residual variance, hypothesis tests and confidence intervals, and diagnostic checks. Gauss–Markov theory established that under certain conditions—errors uncorrelated, with constant variance and zero mean—least squares estimates are the best linear unbiased estimators. The matrix formulation, treating the model as a vector equation with a design matrix, became standard teaching in the postwar decades and made the generalization to many predictors routine.
The codification also exposed the model's fragility. If errors have unequal variance, or are correlated, or if a few observations are wildly atypical, least squares can be badly misled. The 1960s and 1970s brought a wave of responses.
Robustness and the Limits of Least Squares
Robust regression asks what happens when the normal-error assumption fails, especially when data contain outliers3. The earliest systematic treatment is usually credited to Box's 1953 study of non-normality, but the field took off in the 1960s and 1970s. Peter Huber's 1964 paper introduced M-estimation: instead of minimizing squared errors, minimize a function that grows more slowly for large errors, so a single wild point cannot drag the fit. Frank Hampel, in the late 1960s and 1970s, developed influence functions and breakdown points—the fraction of data that can be corrupted before an estimator fails completely. Least squares has a breakdown point of zero: one bad point can ruin it4. The median has a breakdown point of one-half.
The practical payoff is concrete. In a simple regression of income on years of education, a single data-entry error in the income column can swing the slope substantially under least squares; a robust fit downweights that point and stays close to the bulk of the data. Robust methods—Huber's M-estimators, least median of squares, least trimmed squares—entered statistical software through the 1980s and 1990s and are now standard options.
Generalized Linear Models
Least squares assumes a continuous response with constant variance. Much data is not like that: counts, proportions, survival times, binary outcomes. The generalized linear model (GLM), set out by John Nelder and Robert Wedderburn in 1972, unified these cases. The idea is to keep the linear predictor—a weighted sum of inputs—but connect it to the mean of the response through a link function, and to let the variance depend on the mean. Logistic regression for binary outcomes (link: log-odds), Poisson regression for counts (link: logarithm), and the classical normal linear model all become special cases. Estimation proceeds by iteratively reweighted least squares, a computational scheme that repeatedly solves weighted least-squares problems5.
The GLM made regression applicable across the biomedical and social sciences and set the template for later extensions: generalized additive models (Hastie and Tibshirani, 1986) allow smooth nonlinear terms; generalized estimating equations (Liang and Zeger, 1986) handle correlated data such as repeated measurements on the same patient. The framework's economy—one fitting engine, many response types—is why it remains the default in applied work.
Nonlinear Least Squares
When the relationship between parameters and response is not linear—exponential decay, Michaelis–Menten kinetics, growth curves—least squares can still be applied, but the solution generally requires iteration1. The Gauss–Newton algorithm, descending from Gauss's work, linearizes the model at the current parameter guess, solves a linear least-squares problem, and repeats. The Levenberg–Marquardt algorithm (Levenberg 1944, Marquardt 1963) interpolates between Gauss–Newton and gradient descent to stabilize convergence. Nonlinear least squares became practical with computing in the 1960s and is now routine in chemistry, pharmacology, and engineering, where mechanistic models with interpretable parameters are preferred to flexible statistical fits.
Regularization and the High-Dimensional Turn
Classical least squares breaks down when predictors are many and correlated, or outnumber observations: estimates become unstable or undefined. Regularization adds a penalty on the size of coefficients. Hoerl and Kennard's ridge regression (1970) adds a penalty proportional to the sum of squared coefficients, shrinking them toward zero and stabilizing the fit at the cost of some bias. The LASSO (Tibshirani, 1996) penalizes the sum of absolute values, which drives some coefficients exactly to zero—automatic variable selection6. Elastic net (Zou and Hastie, 2005) combines the two. These methods were built for prediction and for screening many candidate inputs, and they became central as datasets grew wider than they were tall—genomics, text, imaging. Their lineage runs from numerical analysis and from the bias–variance tradeoff formalized in the 1990s, and they now sit at the boundary between statistics and machine learning.
Quantile Regression
Least squares estimates the conditional mean. Sometimes the question is about other parts of the distribution: how does a policy affect the lowest earners rather than the average? Quantile regression, introduced by Roger Koenker and Gilbert Bassett in 1978, estimates the conditional median or any other quantile by minimizing an asymmetric absolute-loss function. It is robust to outliers in the response and describes the whole conditional distribution rather than a single summary. It has become standard in economics—wage distributions, for instance—and in ecology and medicine where extremes matter more than averages.
Causal Inference and the Regression Framework
Regression coefficients are often read causally, and a long tradition—from Sewall Wright's path analysis in the 1910s and 1920s through structural equation models—tried to justify that reading. The modern causal framework, developed from the 1980s by Donald Rubin, Judea Pearl, and James Robins, made the conditions explicit: a regression coefficient estimates a causal effect only if the right variables are adjusted for, and the framework of potential outcomes and directed acyclic graphs tells you which those are. The lesson for practice is that regression is a tool for adjustment, and its causal interpretation depends on assumptions—no unmeasured confounding, correct model form—that the data cannot verify. This reframing changed how regression is taught in epidemiology and economics and produced methods (propensity scores, inverse-probability weighting, doubly robust estimators) that combine regression with design-based reasoning.
Resampling and Model Assessment
How much should a fitted model be trusted? Classical answers rely on distributional formulas. From the late 1970s, resampling offered an alternative: the bootstrap (Efron, 1979) estimates the variability of an estimate by repeatedly resampling the data; cross-validation estimates out-of-sample prediction error by fitting on part of the data and testing on the rest78. These methods made model assessment possible where formulas were unavailable or unreliable, and they became the standard way to compare competing models and tune regularization penalties. Their rise tracked cheap computing and is now inseparable from applied regression practice.
The Present
Regression today is a family of methods rather than a single technique1. The linear model and its GLM extensions remain the default for interpretable effect estimation; robust and quantile methods handle distributional departures; regularization and resampling carry the field into high-dimensional prediction; and the causal framework supplies the assumptions needed to read coefficients as effects. Software has absorbed all of these into routine commands, which raises the standing risk that a fit is run without its assumptions checked. The open questions are largely about honesty under complexity: how to do inference after variable selection, how to certify causal claims from observational data, and how to keep interpretable regression competitive with flexible machine-learning predictors when both are applied to the same problem.
References
- Regression analysis (introduction, History, Regression model, Nonlinear regression), Wikipedia
- Least squares, Wikipedia
- Robust regression, Wikipedia
- Robust Regression and Outlier Detection (Background), Wikipedia
- Generalized linear model, Wikipedia
- Lasso (statistics) (introduction, History), Wikipedia
- Cross-validation (statistics), Wikipedia
- Bootstrapping (statistics) (introduction, History), Wikipedia
The frameworks of Regression Analysis
Each framework is a way of working the field has used: when it held, and the question it answers.
- 1805-Present
- 1919-Present
Analysis of Variance and Experimental Design
Do treatments differ beyond chance?
- 1960-Present
- 1964-Present
- 1970-Present
Regularization and Penalized Regression
How do we fit many predictors stably?
- 1972-Present
- 1978-Present
- 1979-Present
- 1980-Present
How the ideas connect
Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.
Robust Regression reacted against Ordinary least squares
Robust regression arose because least squares has a breakdown point of zero: one outlier can ruin the fit. Huber's M-estimation replaced the squared-error loss with a slower-growing loss to resist contamination.
Carried by Peter Huber; Frank Hampel.
Generalized linear model grew out of Ordinary least squares
The generalized linear model keeps the linear predictor and least-squares fitting engine but generalizes the error distribution and links the mean to the predictor. Nelder and Wedderburn presented the normal linear model as one special case.
Carried by John Nelder; Robert Wedderburn.
Analysis of Variance and Experimental Design grew out of Ordinary least squares
Fisher's analysis of variance partitioned the same total sum of squares that least squares minimizes, and ANOVA for designed experiments was later recognized as regression on indicator variables. Fisher built the variance decomposition on the least-squares fitting framework.
Carried by Ronald Fisher; Statistical Methods for Research Workers (1925).
Non-linear least squares grew out of Ordinary least squares
Nonlinear least squares extends the least-squares criterion to models nonlinear in parameters, solved iteratively by Gauss-Newton and Levenberg-Marquardt. The criterion and normal-equation logic come directly from Legendre and Gauss.
Carried by Donald Marquardt; Kenneth Levenberg.
Regularization and Penalized Regression grew out of Ordinary least squares
Ridge regression modifies the least-squares normal equations by adding a penalty on coefficient size, stabilizing estimates when predictors are collinear or numerous. Hoerl and Kennard framed it explicitly as a fix to ordinary least squares.
Carried by Arthur Hoerl; Robert Kennard.
Quantile Regression reacted against Ordinary least squares
Quantile regression was introduced as an alternative to least squares when the question concerns parts of the conditional distribution other than the mean. Koenker and Bassett estimated conditional quantiles by minimizing absolute rather than squared residuals.
Carried by Roger Koenker; Gilbert Bassett.
Causal inference influenced by Analysis of Variance and Experimental Design
Causal inference in observational studies drew on Fisher's randomization and experimental-design logic, especially the potential-outcomes framework. Rubin and others formalized causal effects by analogy to randomized experiments.
Carried by Donald Rubin; Ronald Fisher.
Beyond the field
Where Regression Analysis's ideas came from in other fields, and what they led to there: the strongest 12 of 51 links.
Analysis of Variance and Experimental Design led to Multivariate analysis of variance in Multivariate Statistics
MANOVA is a direct generalization of Fisher's analysis of variance to several dependent variables, replacing sums of squares with matrices of sums of squares and cross products.
Carried by Ronald A. Fisher; Samuel S. Wilks.
Analysis of Variance and Experimental Design led to Response Surface Methodology in Experimental Design
Response surface methodology grew directly out of Fisherian analysis of variance and experimental design, using factorial designs and regression fitting of polynomial models to experimental data.
Carried by George E. P. Box; Ronald A. Fisher.
Analysis of Variance and Experimental Design led to Split-Plot and Multilevel Design in Experimental Design
Split-plot and multilevel designs grew directly out of Fisher's analysis of variance and experimental design, extending the error-stratification of ANOVA to experiments with more than one randomisation unit. The split-plot analysis is a branch of the ANOVA programme.
Carried by Ronald A. Fisher; Frank Yates.
Analysis of Variance and Experimental Design led to Fractional Factorial Design in Experimental Design
Fractional factorial design grew directly out of Fisher's factorial design and analysis of variance, extending it by deliberately confounding higher-order interactions so that only a fraction of treatment combinations need be run. The analysis remains an ANOVA of the selected fraction.
Carried by Ronald A. Fisher; Finney.
Non-linear least squares influenced Nonlinear mixed-effects modeling in Pharmacometrics
Nonlinear least squares supplied the numerical optimization algorithms that nonlinear mixed-effects modeling extended to include random effects.
Carried by Non-linear least squares.
Non-linear least squares grew out of Least squares in Statistical Inference
Non-linear least squares is the direct generalization of linear least squares: the same sum-of-squares criterion minimized iteratively (Gauss-Newton, Levenberg-Marquardt) when the model is nonlinear in parameters.
Carried by Kenneth Levenberg; Donald Marquardt.
Non-linear least squares grew out of Least Squares and Classical Regression in Statistical Learning
Non-linear least squares grew directly out of classical least-squares regression by relaxing linearity in the parameters while keeping the least-squares criterion.
Robust Regression reacted against Ordinary Least Squares Regression in Microeconometrics
Robust regression arose as a response to the fragility of ordinary least squares under outliers and heavy-tailed errors, defining itself by rejecting OLS's sensitivity to contamination.
Carried by Peter J. Huber; Frank Hampel.
Regularization and Penalized Regression grew out of Least Squares and Classical Regression in Statistical Learning
Penalized regression grew directly out of classical least-squares regression by adding a penalty to the residual sum of squares, as in ridge regression.
Carried by Arthur Hoerl; Robert Kennard.
Regularization and Penalized Regression influenced Regularization and High-Dimensional Methods in ML Econometrics
Tibshirani's lasso and the ridge regression tradition of penalised regression supplied the estimators that high-dimensional econometrics adopted for variable selection and shrinkage.
Carried by Robert Tibshirani; Hoerl and Kennard (1970).
Generalized linear model grew out of Least Squares and Classical Regression in Statistical Learning
The generalized linear model grew directly out of classical least-squares regression by allowing non-normal error distributions and a link function while retaining the linear predictor.
Carried by John Nelder; Robert Wedderburn.
Quantile Regression grew out of Ordinary Least Squares Regression in Microeconometrics
Quantile regression was formulated as a direct generalisation of ordinary least squares: minimising a sum of asymmetric absolute residuals yields conditional quantiles where least squares yields the conditional mean. Koenker and Bassett defined their estimator by analogy with and in contrast to OLS.
Carried by Roger Koenker; Gilbert Bassett; Regression Quantiles (1978).

Regression Analysis is a planet in Nousfarer
Its city has a tower for each of these 9 frameworks. Land, walk in, and climb one concept at a time.
Land on Regression AnalysisRelated fields
Fields its ideas are linked to
- Statistical Inference 6 links
- Microeconometrics 4 links
- Experimental Design 4 links
- Statistical Learning 4 links
- Multivariate Statistics 3 links
- Nonparametric Statistics 3 links
- Causal Inference 2 links
- Agricultural Economics 1 link
- Farm Economics 1 link
- Machine Learning 1 link
- Inverse Problems 1 link
- Numerical Analysis 1 link
More in Statistics
- Bayesian Statistics
- Causal Inference
- Experimental Design
- Multivariate Statistics
- Nonparametric Statistics
- Spatial Statistics
- Statistical Computing
- Statistical Inference
- Statistical Learning
- Statistics
- Survey Sampling
- Survival Analysis
- Time Series Analysis
See where Regression Analysis sits on the Map of Human Knowledge