Loading field map
Multivariate statistics is the branch of statistics concerned with data in which each observation consists of multiple measurements, and with the relationships among those measurements. Its defining feature is that the variables are analyzed jointly, not one at a time. A study of height alone is univariate; a study of height, weight, and age together, where the correlations among them are part of the question, is multivariate. The field develops mathematical models and procedures for describing, testing, and predicting with such data, and for reducing their complexity without losing the essential structure.
The core problems of multivariate statistics arise from the fact that variables carry information both individually and jointly. A first set of questions concerns dependence: How are variables related to one another? Are the relationships linear or nonlinear, strong or weak? Can the dependence structure be summarized by a few underlying factors? A second set concerns group differences: Do two or more populations differ in their multivariate profiles, and if so, which variables drive the difference? A third concerns dimension reduction: When many variables are measured, can the data be represented in a lower-dimensional space while preserving the patterns that matter? A fourth concerns classification and prediction: Given a new observation, to which group does it belong, or what value of one variable is implied by the others?
These questions are not independent. A method for group differences may rely on a particular model of dependence; a dimension-reduction technique may be justified by its success at classification. The field is unified by a common mathematical language—vectors and matrices—and by a common concern: that the joint behavior of variables, not their separate behaviors, is the object of study.
The roots of multivariate statistics lie in the late nineteenth and early twentieth centuries, when biometricians began to measure multiple traits of individuals and sought to understand their covariation. Francis Galton’s work on heredity introduced the idea of correlation, and Karl Pearson developed the product-moment correlation coefficient and methods for fitting lines and planes to multivariate data. These were not yet a distinct discipline, but they established the central notion that variables could be studied through their joint distribution.
The field proper emerged in the 1920s and 1930s, largely through the work of statisticians who were also mathematicians. Ronald Fisher developed analysis of variance and discriminant analysis for classifying individuals into groups. Harold Hotelling introduced principal component analysis as a way to find the linear combinations of variables that carry the most variance, and canonical correlation analysis for studying the relationship between two sets of variables. John Wishart derived the sampling distribution of the sample covariance matrix, which became the foundation for multivariate inference. These developments were consolidated in the 1940s and 1950s, notably in the work of S. N. Roy, who developed a general approach to multivariate hypothesis testing, and in the influential textbooks that codified the field.
A second major wave came with the rise of computers. Methods that were computationally prohibitive for more than a few variables became feasible for large datasets. This enabled the practical use of techniques that had been known theoretically, and it also stimulated new methods. In the 1960s and 1970s, factor analysis, which had been developed in psychology for measuring latent traits, was refined and distinguished from principal component analysis. Cluster analysis and multidimensional scaling grew from psychometrics and ecology into general-purpose tools. In the 1980s and 1990s, structural equation modeling integrated factor analysis with regression to test complex causal hypotheses. More recently, the explosion of high-dimensional data—where the number of variables can exceed the number of observations—has reshaped the field, leading to new methods for regularization, sparse models, and graphical models that estimate the structure of conditional dependence.
The field is not organized into a small number of rival schools, but rather into families of methods that address different problems. These families overlap, and many procedures combine elements from several.
The oldest and most widely used multivariate methods aim to reduce the number of variables while retaining the information they carry. Principal component analysis (PCA) finds new variables, called principal components, that are linear combinations of the original ones, are uncorrelated with each other, and successively capture the maximum possible variance. The first component is the direction of greatest spread in the data; the second is the direction of greatest remaining spread orthogonal to the first, and so on. PCA is a descriptive technique: it makes no assumptions about an underlying generative process. It is used for visualization, for compressing data, and as a preprocessing step for other methods.
Factor analysis is often confused with PCA but rests on a different model. It assumes that the observed variables are linear functions of a smaller number of unobserved common factors plus unique error terms. The factors are latent constructs—such as intelligence or socioeconomic status—that are not directly measured but are inferred from the pattern of correlations. Factor analysis is therefore a model-based technique with a hypothesis about the structure of the data, whereas PCA is a purely algebraic transformation. The two often give similar results in practice, but they answer different questions: PCA asks which linear combinations carry the most variance; factor analysis asks whether a hypothesized set of latent variables can explain the observed correlations.
Multidimensional scaling (MDS) addresses a different kind of reduction. Given a matrix of dissimilarities or distances between objects, MDS finds a configuration of points in a low-dimensional space such that the distances between points approximate the given dissimilarities. It is used when the data are not variables measured on a common scale but pairwise judgments of similarity, as in psychology or market research. Correspondence analysis is a related technique for categorical data, representing the rows and columns of a contingency table in a joint low-dimensional space.
A second family of methods models the dependence structure among variables explicitly. Canonical correlation analysis (CCA) studies the relationship between two sets of variables. Given, say, a set of cognitive test scores and a set of personality measures, CCA finds linear combinations of each set that are maximally correlated with each other. It generalizes multiple regression, which predicts one variable from several, to the case where both sides are multivariate.
Structural equation modeling (SEM) goes further by allowing the researcher to specify a network of causal or associative relationships among observed and latent variables. A model might state that socioeconomic status affects both educational attainment and health, and that educational attainment affects health directly as well as through income. SEM estimates the strengths of these paths and tests whether the implied covariance structure matches the observed data. It is widely used in the social sciences, where randomized experiments are often impossible and researchers must rely on observational data. Its strength—the ability to test complex hypotheses—is also its weakness: the conclusions depend heavily on the correctness of the specified model, and a good fit does not prove that the model is true, since many different models may fit the same data equally well.
Graphical models are a more recent approach to dependence. They represent variables as nodes in a graph, with edges indicating conditional dependence: two variables are connected if they are dependent even after conditioning on all other variables. The absence of an edge means that the two variables are conditionally independent. Gaussian graphical models assume the data are multivariate normal, in which case the graph is determined by the inverse of the covariance matrix. These models are particularly useful in high-dimensional settings, such as genomics, where the goal is to infer which genes interact directly rather than merely through chains of correlation.
A third family addresses the question of differences between groups. Multivariate analysis of variance (MANOVA) generalizes analysis of variance to several dependent variables. Instead of testing whether group means differ on one variable, it tests whether the vectors of means differ across groups, taking into account the correlations among the variables. The test statistics—such as Wilks’ lambda or Roy’s largest root—are functions of the eigenvalues of a matrix that compares between-group and within-group variation.
Discriminant analysis and logistic regression address classification. In discriminant analysis, one assumes that the observations in each group follow a multivariate normal distribution with a common covariance matrix, and then derives the linear combination of variables that best separates the groups. New observations are assigned to the group whose mean is closest in the discriminant space. Logistic regression, by contrast, models the probability of group membership directly as a function of the variables, without assuming normality. It is more flexible and has largely supplanted discriminant analysis in many applications, though discriminant analysis remains useful when the normality assumption is reasonable and when the goal is dimension reduction for separation.
Cluster analysis is the unsupervised counterpart of classification. Here the groups are not known in advance; the goal is to discover them from the data. Methods include hierarchical clustering, which builds a tree of nested groupings, and k-means, which partitions the data into a fixed number of clusters by minimizing within-cluster variance. Cluster analysis is inherently less well-defined than classification because there is no single correct answer: different algorithms and distance measures can yield very different groupings, and the choice depends on the purpose of the analysis.
The traditional methods above were developed for settings where the number of variables is modest relative to the number of observations. Modern data—gene expression arrays, images, text, financial time series—often have thousands or millions of variables and relatively few observations. In such settings, the sample covariance matrix is singular or unstable, and classical methods break down.
A major response has been regularization. In ridge regression, a penalty term is added to the estimation criterion to shrink the coefficients toward zero, trading bias for variance. The lasso uses a different penalty that sets some coefficients exactly to zero, performing variable selection as part of estimation. These ideas have been extended to multivariate settings: sparse principal component analysis finds components that are linear combinations of only a few variables; the graphical lasso estimates a sparse inverse covariance matrix; sparse discriminant analysis selects a subset of variables for classification. These methods make it possible to analyze high-dimensional data, but they require careful tuning of the penalty strength, and the results are sensitive to that choice.
A second response has been to develop procedures that are valid even when the dimension is large. Random matrix theory, which studies the behavior of matrices with random entries, has provided corrections for the eigenvalues of sample covariance matrices in high dimensions. These corrections are essential for interpreting principal components or MANOVA statistics when the number of variables is not small relative to the sample size.
The families described above are not mutually exclusive. Principal component analysis is often used before cluster analysis to reduce noise and dimension. Discriminant analysis can be seen as a supervised version of dimension reduction, where the components are chosen to separate groups rather than to maximize variance. Canonical correlation analysis and structural equation modeling both model dependence, but the former is exploratory and the latter confirmatory. Graphical models provide a unifying framework for many classical methods: under multivariate normality, the absence of an edge in a graphical model corresponds to a zero in the inverse covariance matrix, which is related to partial correlations and to the coefficients in a regression of one variable on all others.
The field also has a persistent tension between exploratory and confirmatory approaches. PCA, cluster analysis, and MDS are primarily exploratory: they summarize data and suggest hypotheses. MANOVA, discriminant analysis, and SEM are primarily confirmatory: they test hypotheses that were specified in advance. The distinction is not absolute—exploratory methods can be used to test, and confirmatory methods can be used to explore—but it shapes how results are interpreted. A cluster solution found by exploring one dataset is likely to overfit that data; a structural equation model that fits well in one sample may not fit in another.
Contemporary multivariate statistics is characterized by several developments. The first is the continued growth of high-dimensional methods, driven by applications in genomics, neuroscience, finance, and image analysis. These methods are not merely extensions of classical techniques; they involve new theory, new computational algorithms, and new ways of thinking about statistical error. In high dimensions, the classical distinction between estimation and testing blurs, and the focus shifts to prediction accuracy and to the recovery of sparse structures.
The second is the integration of multivariate statistics with machine learning. Many machine learning methods—support vector machines, random forests, neural networks—are inherently multivariate, but they are not always framed in the language of statistical inference. The two communities have converged on common problems: dimension reduction, classification, clustering, and graphical modeling. The differences are largely cultural: statisticians emphasize uncertainty quantification and model assumptions, while machine learning researchers emphasize prediction performance and algorithmic scalability. The boundary is porous, and many researchers work in both traditions.
The third is the development of methods for non-normal and non-linear data. Classical multivariate statistics relies heavily on the multivariate normal distribution, which is mathematically convenient but often unrealistic. Modern methods include copulas, which separate the marginal distributions from the dependence structure; nonparametric approaches that do not assume a parametric form; and methods for discrete, ordinal, and mixed-type data. These extensions are important because much real-world data—counts, categories, rankings, and mixtures of types—does not fit the normal model.
A fourth development is the emphasis on causal inference from observational multivariate data. Classical multivariate statistics is largely associational: it describes how variables are related, but it does not by itself establish that one variable causes another. The potential-outcomes framework and the directed acyclic graph approach have provided tools for drawing causal conclusions from observational data under explicit assumptions. These methods are not a replacement for multivariate statistics but an extension of it, and they have become central in epidemiology, economics, and the social sciences.
The field remains unified by its core commitment: to understand data in which multiple measurements are taken on each unit, and to do so with methods that respect the joint nature of the data. The specific techniques have multiplied and evolved, but the fundamental questions—how variables depend on each other, how groups differ, how complexity can be reduced, and how new observations can be predicted or classified—remain the same.