- Atlas
- Mathematics
- Multivariate Statistics
Multivariate Statistics
Multivariate statistics is the part of statistics that deals with several measurements taken on the same unit at once—many test scores for one student, many symptoms for one patient, many prices for one day, many species counts for one plot.
9 frameworks, from 1904 to today · Statistics · Mathematics
- Factor analysis 1904-Present
- Multivariate normal distribution 1928-Present
- Multivariate analysis of variance 1932-Present
- Principal component analysis 1933-Present
- Discriminant analysis 1936-Present
- Cluster analysis 1950-Present
- Graphical model 1970-Present
- Regularization 1970-Present
- Resampling 1979-Present
Overview
Multivariate statistics is the part of statistics that deals with several measurements taken on the same unit at once—many test scores for one student, many symptoms for one patient, many prices for one day, many species counts for one plot1. What makes it a field in its own right, rather than just a pile of separate analyses, is that the measurements are usually correlated, and the pattern of correlation is often the most interesting thing about them. A student's verbal and mathematical scores move together; a patient's blood pressure and cholesterol move together; a stock's price today is tied to its price yesterday. Multivariate statistics builds the tools for describing that joint structure, testing hypotheses about it, and using it to classify, predict, and simplify.
The field's history is a sequence of responses to that single problem: how to represent a cloud of points in many dimensions, how to test claims about it, and how to do both when the number of variables rivals or exceeds the number of observations.
Correlation and the geometry of the multivariate normal
The subject begins with correlation. Francis Galton, studying heredity in the 1880s, noticed that tall parents tend to have tall children but less tall than themselves—"regression toward mediocrity"—and that two traits could be associated without one causing the other. Karl Pearson gave correlation its modern coefficient in the 1890s, and with it the idea that a set of variables could be summarized by a matrix of pairwise associations rather than examined one at a time.
The theoretical backbone arrived with the multivariate normal distribution: the joint distribution of several variables in which every linear combination is normally distributed, fully described by a mean vector and a covariance matrix (the table of variances and covariances). Its mathematical development is associated with Pearson and later with Harold Hotelling and S. S. Wilks in the 1920s and 1930s. The multivariate normal made the field tractable because it turned questions about a cloud of points into questions about a matrix, and matrix algebra—eigenvalues, determinants, inverses—became the working language. The covariance matrix is the object that most of the classical methods below either decompose, test, or invert.
Principal components: finding the axes of a cloud
The first great data-reduction idea was principal component analysis. Its statistical formulation is credited to Hotelling in 1933 (the underlying mathematics of fitting lines to point clouds goes back further, to Pearson's 1901 work on lines of closest fit). The problem it solved: when you have, say, a dozen correlated measurements, how do you describe the main ways the data vary without drawing a dozen scatterplots? The answer is to find new variables—linear combinations of the originals—that are uncorrelated with one another and ordered so that the first captures as much variance as possible, the second as much of the remainder, and so on1. Geometrically, you rotate the coordinate axes to line up with the long directions of the point cloud1. In practice a dozen measurements might collapse to two or three components that still carry most of the variation, which is why PCA became a standard first look at any multivariate dataset.
Factor analysis: latent causes behind correlations
Factor analysis grew from the same soil but asked a different question1. Charles Spearman, working on intelligence tests around 1904, observed that scores on seemingly different mental tests were all positively correlated, and proposed that a single unobserved "general intelligence" factor could explain the pattern2. The idea generalizes: if many observed variables are correlated, perhaps a few hidden (latent) variables—factors—generate those correlations, with each observed variable loading on the factors plus its own unique noise2. This is a model of the data's causal structure, not merely a rotation of it, and the two approaches have been distinguished ever since. Factor analysis was developed through the 1930s and 1940s by Spearman, Cyril Burt, L. L. Thurstone (whose 1947 Multiple-Factor Analysis was a landmark), and others, and it became the workhorse of psychometrics and later of the social sciences. The distinction between PCA and factor analysis—one a variance-maximizing transformation, the other a latent-variable model—remains a source of confusion and debate.
Testing hypotheses in many dimensions: MANOVA and its relatives
If data are multivariate, so are hypotheses. The 1930s brought a family of tests generalizing the analysis of variance to several response variables at once. Wilks introduced the likelihood-ratio criterion that bears his name in 1932; Hotelling contributed his trace and $T^2$ statistics; and the resulting framework, multivariate analysis of variance (MANOVA), asks whether group means differ across a vector of outcomes jointly rather than one outcome at a time. The motivation is real: running separate univariate tests ignores the correlations among outcomes and inflates the chance of false positives. The price is that "difference in means" is no longer a single number but a direction in a multidimensional space, and different test statistics (Wilks' lambda, Hotelling's trace, Pillai's trace) weight that difference differently. This body of work—multivariate hypothesis testing, confidence regions, and the distribution theory behind them—was largely complete by the 1950s and is the classical core of the field.
Discrimination and classification
A parallel line asked how to sort units into groups. R. A. Fisher's 1936 paper on the iris data—still the most famous dataset in the field—introduced linear discriminant analysis: find the linear combination of measurements that best separates two known groups, then classify new units by which side of a boundary they fall on. The method assumes groups share a covariance structure and differ in means; when that assumption fails, quadratic boundaries follow. Discriminant analysis sits between description and prediction, and it opened the question that dominates modern practice: given labeled examples, how well can a rule classify new cases? The answer depends on estimating error rates honestly, a theme that returns below.
Finding groups without labels: cluster analysis
Where discriminant analysis sorts into known groups, cluster analysis looks for groups that are not yet known1. Its history is more scattered, with roots in numerical taxonomy (biologists grouping organisms by similarity in the 1950s and 1960s), psychology, and engineering. The core methods—hierarchical clustering, which builds a tree of nested groupings; and partitioning methods such as k-means, whose statistical formulation is associated with Stuart Lloyd's work at Bell Labs in the 1950s and 1960s—trade off different notions of what a cluster is34. There is no single correct answer, because "group" must be defined by a distance measure and a criterion, and different choices give different clusters5. This ambiguity is intrinsic, not a defect, and it shaped how the field thinks about unsupervised learning.
Graphical models: correlation as conditional independence
A later strand turned the covariance matrix into a picture. A graphical model represents variables as nodes and conditional independencies as missing edges: if two variables are independent given the rest, no edge connects them6. The Gaussian graphical model, in which the pattern of zeros in the inverse covariance matrix (the precision matrix) encodes these independencies, connects multivariate normal theory to network structure. Early work on Markov random fields and undirected graphs in the 1970s and 1980s (associated with John Besag, Julian Besag, and others in spatial statistics, and with Judea Pearl's later work on directed graphs and causality) gave the field a language for structure. The practical payoff—recovering a network of dependencies from data—became central in genomics, where genes' expression levels are modeled as a graph, and in finance and neuroscience.
The high-dimensional turn: regularization and resampling
The classical methods all assumed more observations than variables, so that the covariance matrix could be estimated and inverted. Modern data broke that assumption: gene expression studies with thousands of genes and dozens of samples, text with tens of thousands of word counts, imaging with millions of voxels7. When variables outnumber observations, the sample covariance matrix is singular and classical tests and inversions fail outright.
The response came from two directions. Regularization, developed from the 1970s onward and crystallized in Robert Tibshirani's 1996 lasso, adds a penalty that shrinks estimates toward simplicity—forcing some coefficients to zero, stabilizing an otherwise unusable covariance estimate, or selecting a sparse set of variables. Ridge regression (Hoerl and Kennard, 1970) and the lasso are the anchors; later methods extended the idea to covariance estimation, graphical models (the graphical lasso), and discriminant analysis. The second response was resampling: the bootstrap, introduced by Bradley Efron in 1979, and permutation methods estimate the variability of a statistic by repeatedly resampling the data rather than relying on formulas that may not hold89. In multivariate settings, where exact distributions are often intractable, resampling made honest inference possible—for example, assessing whether a clustering is real or whether a classifier's apparent accuracy is inflated.
Where the field stands
Today the classical core—multivariate normal theory, PCA, factor analysis, MANOVA, discriminant analysis—is taught as a standard toolkit and remains in daily use. Around it, the high-dimensional and computational strands have become dominant in research: sparse and regularized estimation, resampling-based inference, and graphical models for structure learning. The open questions are recognizably descendants of the old ones. How should a covariance matrix be estimated when variables vastly outnumber observations, and how much structure can be trusted? When a method finds clusters or a network, how do we know the pattern is not an artifact of the algorithm or the sample? And how should the classical tests be adapted when their distributional assumptions—normality, equal covariance, enough data—plainly fail? The field's history is, in large part, a record of taking each of those questions seriously in turn.
References
- Multivariate statistics (introduction, Multivariate analysis, Types of analysis), Wikipedia
- Factor analysis (History, introduction), Wikipedia
- Hierarchical clustering, Wikipedia
- K-means clustering (History), Wikipedia
- Cluster analysis, Wikipedia
- Graphical model, Wikipedia
- Clustering high-dimensional data, Wikipedia
- Bootstrapping (statistics) (History), Wikipedia
- Permutation test, Wikipedia
The frameworks of Multivariate Statistics
Each framework is a way of working the field has used: when it held, and the question it answers.
- 1904-Present
Factor analysis
What hidden causes explain correlations among measurements?
- 1928-Present
Multivariate normal distribution
How do several correlated measurements vary together?
- 1932-Present
Multivariate analysis of variance
Do groups differ across many outcomes at once?
- 1933-Present
Principal component analysis
How can many measurements be reduced to a few?
- 1936-Present
- 1950-Present
- 1970-Present
- 1970-Present
- 1979-Present
How the ideas connect
Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.
Multivariate analysis of variance grew out of Multivariate normal distribution
MANOVA generalized the analysis of variance to vector-valued responses by using the multivariate normal distribution and its Wishart covariance theory. Wilks' likelihood-ratio criterion and Hotelling's T² were built directly on that distributional foundation.
Carried by S. S. Wilks; Harold Hotelling.
Discriminant analysis grew out of Multivariate normal distribution
Fisher's linear discriminant analysis assumed two multivariate normal populations with a common covariance matrix and derived the linear boundary from that model. The classification rule is a direct application of multivariate normal likelihood ratios.
Carried by R. A. Fisher.
Principal component analysis rival of Factor analysis
Factor analysis and principal component analysis competed as answers to how to reduce many correlated variables to a few. Spearman's latent-factor model and Hotelling's variance-maximizing components were developed as distinct, often-confused approaches to the same data-reduction problem.
Carried by Charles Spearman; Harold Hotelling.
Principal component analysis influenced by Multivariate normal distribution
Hotelling's principal component analysis was formulated as the eigen-decomposition of the covariance matrix of a multivariate normal vector. The method's statistical justification rests on that distribution's second-moment structure.
Carried by Harold Hotelling.
Graphical model grew out of Multivariate normal distribution
Gaussian graphical models read conditional independencies off the inverse covariance (precision) matrix of a multivariate normal vector. The graphical representation is a direct extension of multivariate normal theory to network structure.
Carried by John Besag.
Regularization influenced by Multivariate normal distribution
Regularization addressed the failure of the sample covariance matrix when variables outnumber observations, a problem defined within multivariate normal estimation. Ridge regression and shrinkage estimators were developed to stabilize covariance and regression estimates in that setting.
Carried by Arthur Hoerl; Robert Kennard.
Beyond the field
Where Multivariate Statistics's ideas came from in other fields, and what they led to there: the strongest 12 of 28 links.
Factor analysis influenced Factor-Analytic Trait Models in Personality Psychology
The statistical technique of factor analysis, developed in multivariate statistics, is the method on which factor-analytic trait models are built. Cattell and Eysenck applied the mathematical factoring procedures to personality questionnaire data.
Carried by Raymond Cattell; Hans Eysenck.
Factor analysis influenced by Mental Testing and the IQ Tradition in Psychometrics
Spearman's work on mental testing and general intelligence directly produced his 1904 factor analysis, which he devised to isolate a general factor from correlations among test scores.
Carried by Charles Spearman.
Factor analysis led to Structural Equation Modeling in Psychometrics
Structural equation modeling grew directly out of factor analysis, generalising it to systems of observed and latent variables with structural relations among them. Jöreskog's LISREL combined factor-analytic measurement models with path models.
Carried by Karl Jöreskog; LISREL.
Factor analysis influenced Factor Investing in Investment Management
Factor investing's statistical machinery descends from Spearman's factor analysis and its later multivariate refinements: portfolio managers use factor models estimated by factor-analytic methods to explain covariance among securities. The technique passed from psychometrics and multivariate statistics into finance through the work of Ross and others building factor models of asset returns.
Carried by Charles Spearman; Stephen Ross.
Multivariate analysis of variance grew out of Analysis of Variance and Experimental Design in Regression Analysis
MANOVA is a direct generalization of Fisher's analysis of variance to several dependent variables, replacing sums of squares with matrices of sums of squares and cross products.
Carried by Ronald A. Fisher; Samuel S. Wilks.
Principal component analysis influenced by Factor Analysis in Psychometrics
Principal component analysis was introduced by Hotelling in 1933 as a method closely related to, and developed in contrast with, Spearman's and Thurstone's factor analysis, seeking uncorrelated linear combinations rather than latent factors.
Carried by Harold Hotelling.
Graphical model influenced Graphical model in Probabilistic Ai
Graphical models in AI and machine learning draw on the multivariate-statistics tradition of graphical models, including Markov random fields, covariance selection and conditional-independence graphs developed by Darroch, Speed and Lauritzen. That statistical literature supplied the independence semantics and estimation theory the AI framework adopted.
Carried by Steffen Lauritzen; David Spiegelhalter; Joe Whittaker.
Graphical model influenced Probabilistic Graphical Models for Vision in Computer Vision
Graphical models for vision were built directly on the multivariate-statistics formalism of conditional independence graphs and Markov random fields. Vision researchers imported that machinery for image and scene modelling.
Regularization influenced by Tikhonov Regularization in Inverse Problems
Tikhonov regularization in inverse problems supplied the penalty-based stabilization of ill-posed problems that statistical regularization adopted. Ridge regression and later penalized methods are the statistical analogue of Tikhonov's stabilizing functional.
Carried by Andrey Tikhonov.
Factor analysis influenced by Mental measurement in Educational Assessment
The tradition of mental measurement supplied the correlation matrices of test scores that factor analysis was built to decompose, beginning with Spearman's analysis of sensory discrimination and school marks.
Carried by Charles Spearman.
Factor analysis influenced by Intelligence and Standardized Testing in Education
Standardized testing programmes produced the large correlation tables that Thurstone and others factor-analyzed, and the multiple-factor model was developed to interpret batteries of ability tests.
Carried by Louis Leon Thurstone.
Factor analysis influenced by Pearsonian Biometry in Statistical Inference
Pearson's biometrical correlation and the tetrachoric methods gave Spearman the statistical machinery for factor analysis; Spearman worked within the Pearsonian biometric tradition before turning to psychology.
Carried by Karl Pearson; Charles Spearman.

Multivariate Statistics is a planet in Nousfarer
Its city has a tower for each of these 9 frameworks. Land, walk in, and climb one concept at a time.
Land on Multivariate StatisticsRelated fields
Fields its ideas are linked to
- Linear Algebra 3 links
- Data Mining 3 links
- Psychometrics 3 links
- Regression Analysis 3 links
- Personality Psychology 2 links
- Statistical Inference 2 links
- Computer Vision 1 link
- Machine Learning 1 link
- Probabilistic Ai 1 link
- Inverse Problems 1 link
- Education 1 link
- Educational Assessment 1 link
More in Statistics
- Bayesian Statistics
- Causal Inference
- Experimental Design
- Nonparametric Statistics
- Regression Analysis
- Spatial Statistics
- Statistical Computing
- Statistical Inference
- Statistical Learning
- Statistics
- Survey Sampling
- Survival Analysis
- Time Series Analysis
See where Multivariate Statistics sits on the Map of Human Knowledge