Loading field map
Regression analysis is a family of statistical methods for modeling the relationship between a dependent variable (the outcome or response) and one or more independent variables (predictors, covariates, or features). Its central purpose is to estimate how the typical value of the dependent variable changes as the independent variables change, while accounting for the fact that individual observations deviate from any such systematic pattern. The field encompasses not only the mathematical machinery for fitting such models but also the principles for interpreting them, assessing their adequacy, and understanding the conditions under which their conclusions are trustworthy.
At its heart, regression addresses a deceptively simple question: given data on several variables, what is the form of the relationship between them, and how strong is that relationship? The stakes are high because regression is the workhorse of empirical science and policy. Economists use it to estimate the effect of education on earnings; epidemiologists use it to quantify the association between smoking and disease risk; engineers use it to calibrate sensors; and social scientists use it to test theories about human behavior. In all these settings, the goal is not merely to describe the data at hand but to make inferences about a broader population or process, and often to predict future or unseen outcomes.
The fundamental difficulty is that observational data rarely come from controlled experiments. The independent variables are typically correlated with each other and with unmeasured factors that also influence the outcome. This means that a naive regression can produce estimates that are statistically precise but substantively wrong. A large portion of the field's intellectual history is a response to this problem: how to separate the signal of a genuine relationship from the noise of confounding, and how to know when one has failed to do so.
The origins of regression lie in the nineteenth-century work of Francis Galton, who studied how characteristics such as height are transmitted across generations. Galton observed that the heights of children of very tall parents tended to be closer to the population average than the parents' heights—a phenomenon he called "regression toward the mean." He developed a statistical procedure to quantify this tendency, fitting a line that described how a child's height depended on the parents' heights. The term "regression" stuck, even though the modern method is used for far more than mean reversion.
The mathematical foundation was consolidated in the early twentieth century by Karl Pearson and others, who formalized the method of least squares: choosing the line (or more generally, the curve) that minimizes the sum of squared vertical distances between the observed data points and the fitted values. This criterion is computationally tractable and has elegant statistical properties when certain assumptions hold. Ronald Fisher later placed regression within the framework of statistical inference, providing methods for testing hypotheses about the coefficients and constructing confidence intervals.
For much of the twentieth century, regression analysis was primarily a tool of the applied statistician, taught as a set of procedures for fitting lines and planes to data. The computational revolution of the late twentieth century transformed the field. The ability to fit complex models to large datasets led to the development of many extensions—generalized linear models, mixed-effects models, nonparametric and semiparametric methods, and regularization techniques—that have made regression a far more flexible and powerful tool than the simple line-fitting of its origins.
The starting point for most of regression analysis is the linear model:
\[ yi = \beta0 + \beta1 x{i1} + \beta2 x{i2} + \cdots + \betap x{ip} + \varepsilon_i \]
Here, \(yi\) is the outcome for observation \(i\), the \(x{ij}\) are the values of the \(p\) predictors, the \(\beta\)'s are unknown coefficients to be estimated, and \(\varepsiloni\) is an error term capturing everything not explained by the predictors. The model is "linear" in the parameters, not necessarily in the variables: one can include \(x^2\), \(\log(x)\), or interactions like \(x1 x_2\) as predictors, and the model remains linear in the \(\beta\)'s.
The coefficients have a straightforward interpretation: \(\betaj\) is the expected change in \(y\) associated with a one-unit increase in \(xj\), holding all other predictors constant. This "holding constant" language is the source of both the method's power and its peril. It is powerful because it allows one to isolate the association of a single predictor from the others in the model. It is perilous because the model can only hold constant what is actually measured and included; unmeasured confounders remain a threat.
The classical assumptions under which the least-squares estimates have their desirable properties are: the relationship is truly linear, the errors have mean zero and constant variance, the errors are independent of each other, and the predictors are not perfectly collinear. Under these conditions, the Gauss-Markov theorem guarantees that least squares is the best linear unbiased estimator. If, in addition, the errors are normally distributed, then the standard tools of inference—\(t\)-tests, \(F\)-tests, and confidence intervals—are exact in finite samples.
These assumptions are rarely fully met in practice, and a substantial part of applied regression analysis is diagnostic checking: examining residuals (the differences between observed and fitted values) for patterns that suggest nonlinearity, heteroscedasticity (non-constant variance), or dependence. The field has developed a rich toolkit of graphical and numerical diagnostics, as well as remedies such as transforming variables, using weighted least squares, or fitting more flexible models.
The classical linear model assumes a continuous outcome with normally distributed errors. Many real-world outcomes are not like this: they are binary (yes/no), counts, proportions, or times-to-events. The generalized linear model (GLM), developed in the 1970s, extends regression to these settings within a unified framework.
A GLM has three components: a linear predictor (the same \(\beta0 + \beta1 x_1 + \cdots\) as before), a link function that connects the linear predictor to the mean of the outcome, and a probability distribution for the outcome. For binary outcomes, the logistic regression model uses the logit link and the Bernoulli distribution; the coefficients are interpreted in terms of changes in the log-odds of the outcome. For count data, Poisson regression uses the log link and the Poisson distribution. For proportions, one might use a binomial model with a logit or probit link.
The parameters of a GLM are estimated by maximum likelihood: choosing the coefficients that make the observed data most probable under the model. This approach has a unified theory of inference based on the likelihood function, and it subsumes the classical linear model as a special case (with the identity link and the normal distribution). GLMs are now standard in fields as diverse as medicine, ecology, and marketing, and they represent one of the most important extensions of regression beyond its classical origins.
A recurring theme in regression is the tension between model complexity and estimation stability. A model with many predictors can fit the training data very well, but it may also fit noise, leading to poor predictions on new data. This is the problem of overfitting, and it becomes acute when the number of predictors is large relative to the number of observations.
Regularization methods address this by adding a penalty to the least-squares or likelihood criterion that shrinks the coefficients toward zero. Ridge regression adds a penalty proportional to the sum of squared coefficients, which shrinks them but does not set any exactly to zero. The lasso adds a penalty proportional to the sum of absolute values of the coefficients, which has the effect of setting some coefficients exactly to zero, thereby performing variable selection. The elastic net combines both penalties.
These methods embody a fundamental tradeoff: by introducing some bias (shrinking coefficients away from their unpenalized values), they can substantially reduce variance, leading to better overall prediction. The choice of the penalty strength is typically made by cross-validation, a procedure in which the data are repeatedly split into training and validation subsets. Regularization has become essential in the era of high-dimensional data, where the number of predictors can exceed the number of observations, and it forms a bridge between classical regression and modern machine learning.
The linear model imposes a strong assumption: that the relationship between predictors and outcome has a known parametric form. When this assumption is wrong, the model can be badly misleading. Nonparametric regression relaxes this by allowing the relationship to be any smooth function, estimated from the data.
The simplest nonparametric method is the moving average or kernel smoother, which estimates the value of the function at a point by averaging nearby observations, with weights that decline with distance. More sophisticated methods include splines, which fit piecewise polynomials joined at knots, and local polynomial regression, which fits a low-degree polynomial in a moving window. These methods are flexible but require the choice of a smoothing parameter that controls the bias-variance tradeoff, and they become difficult to use when there are many predictors—a phenomenon known as the curse of dimensionality.
Semiparametric models occupy a middle ground. For example, a partially linear model has the form \(y = \beta x + g(z) + \varepsilon\), where the effect of \(x\) is linear but the effect of \(z\) is an arbitrary smooth function. Such models allow one to focus on the coefficient of interest while avoiding parametric assumptions about nuisance variables. They are particularly useful in econometrics and biostatistics, where the primary interest is in a treatment effect or exposure effect, and the confounding variables are treated nonparametrically.
Many datasets have a natural hierarchical structure: students within classrooms, patients within hospitals, repeated measurements within individuals. In such settings, observations are not independent, and the classical assumption of independent errors is violated. Mixed-effects models, also called hierarchical or multilevel models, address this by including both fixed effects (the usual regression coefficients) and random effects (deviations associated with groups or subjects).
A random intercept model, for example, allows each group to have its own baseline level, with the group-level intercepts drawn from a normal distribution. This induces correlation among observations within the same group, which is accounted for in the inference. Random slopes allow the effect of a predictor to vary across groups. These models are widely used in education, psychology, and clinical trials, where the hierarchical structure is intrinsic to the data collection.
Mixed-effects models also provide a principled framework for handling missing data and for borrowing strength across groups: estimates for groups with few observations are pulled toward the overall average, a phenomenon known as shrinkage. This is closely related to the regularization methods discussed above, and it reflects a deep connection between Bayesian and frequentist approaches to hierarchical modeling.
A persistent theme in regression analysis is the distinction between association and causation. A regression coefficient estimates the average difference in the outcome associated with a one-unit difference in the predictor, holding the other predictors constant. This is a statement about association, not necessarily about what would happen if one intervened to change the predictor.
The conditions under which a regression coefficient can be interpreted causally are stringent. In an experiment where the predictor is randomly assigned, the simple regression of outcome on treatment estimates the average causal effect. In observational data, causal interpretation requires that all confounders—variables that affect both the predictor and the outcome—are measured and included in the model. This assumption, sometimes called ignorability or no unmeasured confounding, is rarely verifiable from the data alone.
The modern causal inference literature, associated with the potential outcomes framework, has clarified these issues and developed methods that go beyond standard regression. These include propensity score methods, instrumental variables, and regression discontinuity designs. These methods often use regression as a building block but embed it in a more explicit model of the data-generating process. The relationship between regression and causal inference remains an active area of methodological research, and the practical lesson is that regression coefficients should be interpreted causally only with great care and explicit justification.
The present landscape of regression analysis is characterized by a rich interplay between classical methods and newer developments. The classical linear model and its generalized extensions remain the foundation of statistical practice in most applied fields. They are taught in introductory courses, implemented in every statistical software package, and used daily in thousands of research studies.
At the same time, the boundaries between regression and machine learning have become increasingly porous. Many machine learning methods—such as decision trees, random forests, and neural networks—can be viewed as flexible nonlinear regression models. The field of statistical learning has systematized the bias-variance tradeoff, cross-validation, and regularization, and these ideas have flowed back into regression practice. The lasso, originally developed in statistics, is now a standard tool in machine learning pipelines.
The modern practitioner has a wide menu of options: classical linear models for interpretability and inference, GLMs for non-normal outcomes, mixed-effects models for hierarchical data, nonparametric methods for flexible curve fitting, and regularized methods for high-dimensional problems. The choice among these is guided by the research question, the nature of the data, and the relative importance of interpretability versus predictive accuracy.
A central theme that unites all these approaches is the acknowledgment of uncertainty. Regression analysis is not a mechanical procedure that produces the "true" relationship; it is a disciplined way of using data to learn about relationships while quantifying what remains unknown. The field's enduring contribution is not any single method but a framework for thinking about how to learn from noisy, imperfect data—a framework that balances mathematical rigor with practical judgment, and that remains as relevant today as it was in Galton's time.