Regression Analysis

Regression analysis is the part of statistics that asks how one quantity depends on others.

9 frameworks, from 1805 to today · Statistics · Mathematics

still a working view ended · each leads to its entry below

Overview

Regression analysis is the part of statistics that asks how one quantity depends on others1. Given measurements of a response—a crop yield, a price, a patient's blood pressure—and of one or more inputs that may explain it—rainfall, supply, dosage—regression fits a relationship and uses it to predict, to estimate effects, and to judge how much of the variation the inputs account for1. It is the workhorse of applied statistics: most quantitative claims in medicine, economics, engineering, and the social sciences pass through a regression at some point. Its history is a sequence of attempts to make that fitting honest—to find estimates that are optimal under stated conditions, to check those conditions, and to keep the answers trustworthy when they fail.

Least Squares and the Birth of Fitting

The founding problem was astronomical. In the late eighteenth century, astronomers had more observations of a planet or comet than unknowns in its orbit, and the observations disagreed. Which orbit fits best? Adrien-Marie Legendre published in 1805 the rule now called least squares: choose the parameters that make the sum of squared differences between observed and predicted values as small as possible1. Carl Friedrich Gauss had used the method earlier, from 1795, and gave it a probabilistic justification in 1809: if errors are independent and follow a normal (bell-shaped) distribution, least squares gives the most probable parameter values. Squaring errors is not arbitrary—it is what the normal distribution implies.

The method spread because it was computable and defensible. By hand, least squares reduces to solving a system of linear equations, and the fitted line or plane has a closed form2. Gauss and others applied it to geodesy and astronomy; in the nineteenth century it became standard in surveying and in the new social sciences. Francis Galton, studying heredity in the 1880s, coined "regression" for the tendency of tall parents' children to be closer to average height than their parents—regression toward the mean—and introduced the correlation coefficient with Karl Pearson in the 1890s. The linear regression model, with its fitted coefficients, standard errors, and tests, was largely in place by the early twentieth century.

Analysis of Variance and Designed Experiments

A parallel line of work asked not how to fit a given set of data but how to collect data so that effects could be separated. Ronald Fisher, at Rothamsted Experimental Station from 1919, developed the analysis of variance (ANOVA): a way of partitioning total variation into parts attributable to treatments and to error, then testing whether treatment differences exceed what chance would produce. His 1925 Statistical Methods for Research Workers and 1935 The Design of Experiments made randomization, replication, and blocking standard practice. Randomization—assigning treatments by chance—was the key idea: it turns unmeasured confounders into random noise rather than systematic bias.

ANOVA and regression are two views of the same linear model. A designed experiment with categorical treatments is a regression on indicator variables; Fisher's variance decomposition is the regression sum of squares. This unity, made explicit in the mid-twentieth century, meant that the tools of experimental design and the tools of observational fitting could be taught together. George Box's work on response surface methodology in the 1950s extended designed experimentation to finding optimal operating conditions, and Box and colleagues' 1951 paper on non-normality began the study of how sensitive least squares is to its assumptions.

The Linear Model Codified

Between the 1920s and 1960s the linear regression model was formalized as a package: coefficients estimated by least squares, standard errors from the residual variance, hypothesis tests and confidence intervals, and diagnostic checks. Gauss–Markov theory established that under certain conditions—errors uncorrelated, with constant variance and zero mean—least squares estimates are the best linear unbiased estimators. The matrix formulation, treating the model as a vector equation with a design matrix, became standard teaching in the postwar decades and made the generalization to many predictors routine.

The codification also exposed the model's fragility. If errors have unequal variance, or are correlated, or if a few observations are wildly atypical, least squares can be badly misled. The 1960s and 1970s brought a wave of responses.

Robustness and the Limits of Least Squares

Robust regression asks what happens when the normal-error assumption fails, especially when data contain outliers3. The earliest systematic treatment is usually credited to Box's 1953 study of non-normality, but the field took off in the 1960s and 1970s. Peter Huber's 1964 paper introduced M-estimation: instead of minimizing squared errors, minimize a function that grows more slowly for large errors, so a single wild point cannot drag the fit. Frank Hampel, in the late 1960s and 1970s, developed influence functions and breakdown points—the fraction of data that can be corrupted before an estimator fails completely. Least squares has a breakdown point of zero: one bad point can ruin it4. The median has a breakdown point of one-half.

The practical payoff is concrete. In a simple regression of income on years of education, a single data-entry error in the income column can swing the slope substantially under least squares; a robust fit downweights that point and stays close to the bulk of the data. Robust methods—Huber's M-estimators, least median of squares, least trimmed squares—entered statistical software through the 1980s and 1990s and are now standard options.

Generalized Linear Models

Least squares assumes a continuous response with constant variance. Much data is not like that: counts, proportions, survival times, binary outcomes. The generalized linear model (GLM), set out by John Nelder and Robert Wedderburn in 1972, unified these cases. The idea is to keep the linear predictor—a weighted sum of inputs—but connect it to the mean of the response through a link function, and to let the variance depend on the mean. Logistic regression for binary outcomes (link: log-odds), Poisson regression for counts (link: logarithm), and the classical normal linear model all become special cases. Estimation proceeds by iteratively reweighted least squares, a computational scheme that repeatedly solves weighted least-squares problems5.

The GLM made regression applicable across the biomedical and social sciences and set the template for later extensions: generalized additive models (Hastie and Tibshirani, 1986) allow smooth nonlinear terms; generalized estimating equations (Liang and Zeger, 1986) handle correlated data such as repeated measurements on the same patient. The framework's economy—one fitting engine, many response types—is why it remains the default in applied work.

Nonlinear Least Squares

When the relationship between parameters and response is not linear—exponential decay, Michaelis–Menten kinetics, growth curves—least squares can still be applied, but the solution generally requires iteration1. The Gauss–Newton algorithm, descending from Gauss's work, linearizes the model at the current parameter guess, solves a linear least-squares problem, and repeats. The Levenberg–Marquardt algorithm (Levenberg 1944, Marquardt 1963) interpolates between Gauss–Newton and gradient descent to stabilize convergence. Nonlinear least squares became practical with computing in the 1960s and is now routine in chemistry, pharmacology, and engineering, where mechanistic models with interpretable parameters are preferred to flexible statistical fits.

Regularization and the High-Dimensional Turn

Classical least squares breaks down when predictors are many and correlated, or outnumber observations: estimates become unstable or undefined. Regularization adds a penalty on the size of coefficients. Hoerl and Kennard's ridge regression (1970) adds a penalty proportional to the sum of squared coefficients, shrinking them toward zero and stabilizing the fit at the cost of some bias. The LASSO (Tibshirani, 1996) penalizes the sum of absolute values, which drives some coefficients exactly to zero—automatic variable selection6. Elastic net (Zou and Hastie, 2005) combines the two. These methods were built for prediction and for screening many candidate inputs, and they became central as datasets grew wider than they were tall—genomics, text, imaging. Their lineage runs from numerical analysis and from the bias–variance tradeoff formalized in the 1990s, and they now sit at the boundary between statistics and machine learning.

Quantile Regression

Least squares estimates the conditional mean. Sometimes the question is about other parts of the distribution: how does a policy affect the lowest earners rather than the average? Quantile regression, introduced by Roger Koenker and Gilbert Bassett in 1978, estimates the conditional median or any other quantile by minimizing an asymmetric absolute-loss function. It is robust to outliers in the response and describes the whole conditional distribution rather than a single summary. It has become standard in economics—wage distributions, for instance—and in ecology and medicine where extremes matter more than averages.

Causal Inference and the Regression Framework

Regression coefficients are often read causally, and a long tradition—from Sewall Wright's path analysis in the 1910s and 1920s through structural equation models—tried to justify that reading. The modern causal framework, developed from the 1980s by Donald Rubin, Judea Pearl, and James Robins, made the conditions explicit: a regression coefficient estimates a causal effect only if the right variables are adjusted for, and the framework of potential outcomes and directed acyclic graphs tells you which those are. The lesson for practice is that regression is a tool for adjustment, and its causal interpretation depends on assumptions—no unmeasured confounding, correct model form—that the data cannot verify. This reframing changed how regression is taught in epidemiology and economics and produced methods (propensity scores, inverse-probability weighting, doubly robust estimators) that combine regression with design-based reasoning.

Resampling and Model Assessment

How much should a fitted model be trusted? Classical answers rely on distributional formulas. From the late 1970s, resampling offered an alternative: the bootstrap (Efron, 1979) estimates the variability of an estimate by repeatedly resampling the data; cross-validation estimates out-of-sample prediction error by fitting on part of the data and testing on the rest78. These methods made model assessment possible where formulas were unavailable or unreliable, and they became the standard way to compare competing models and tune regularization penalties. Their rise tracked cheap computing and is now inseparable from applied regression practice.

The Present

Regression today is a family of methods rather than a single technique1. The linear model and its GLM extensions remain the default for interpretable effect estimation; robust and quantile methods handle distributional departures; regularization and resampling carry the field into high-dimensional prediction; and the causal framework supplies the assumptions needed to read coefficients as effects. Software has absorbed all of these into routine commands, which raises the standing risk that a fit is run without its assumptions checked. The open questions are largely about honesty under complexity: how to do inference after variable selection, how to certify causal claims from observational data, and how to keep interpretable regression competitive with flexible machine-learning predictors when both are applied to the same problem.

References

  1. Regression analysis (introduction, History, Regression model, Nonlinear regression), Wikipedia
  2. Least squares, Wikipedia
  3. Robust regression, Wikipedia
  4. Robust Regression and Outlier Detection (Background), Wikipedia
  5. Generalized linear model, Wikipedia
  6. Lasso (statistics) (introduction, History), Wikipedia
  7. Cross-validation (statistics), Wikipedia
  8. Bootstrapping (statistics) (introduction, History), Wikipedia

The frameworks of Regression Analysis

Each framework is a way of working the field has used: when it held, and the question it answers.

  1. 1805-Present
  2. 1919-Present
  3. 1960-Present
  4. 1964-Present

    Robust Regression

    How do we fit data with outliers?

    Fly to its tower in Nousfarer

  5. 1970-Present
  6. 1972-Present

    Generalized linear model

    How do we fit counts and proportions?

    Fly to its tower in Nousfarer

  7. 1978-Present

    Quantile Regression

    How does the effect vary across the distribution?

    Fly to its tower in Nousfarer

  8. 1979-Present
  9. 1980-Present

    Causal inference

    Does an input actually cause the outcome?

    Fly to its tower in Nousfarer

How the ideas connect

Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.

Ordinary least squaresAnalysis of Variance and Experimental…Non-linear least squaresRobust RegressionRegularization and Penalized Regressi…Generalized linear modelQuantile RegressionResampling and Model AssessmentCausal inference

Beyond the field

Where Regression Analysis's ideas came from in other fields, and what they led to there: the strongest 12 of 51 links.

A city in Nousfarer: a red-carpeted stair climbing terraces to a crowned tower on an acropolis, more towers round it, forest and snowy mountains beyond.
A city in Nousfarer, the game played in the browser.

Regression Analysis is a planet in Nousfarer

Its city has a tower for each of these 9 frameworks. Land, walk in, and climb one concept at a time.

Land on Regression Analysis

Related fields

Fields its ideas are linked to

More in Statistics

See where Regression Analysis sits on the Map of Human Knowledge