Multivariate Statistics

Multivariate statistics is the part of statistics that deals with several measurements taken on the same unit at once—many test scores for one student, many symptoms for one patient, many prices for one day, many species counts for one plot.

9 frameworks, from 1904 to today · Statistics · Mathematics

still a working view ended · each leads to its entry below

Overview

Multivariate statistics is the part of statistics that deals with several measurements taken on the same unit at once—many test scores for one student, many symptoms for one patient, many prices for one day, many species counts for one plot1. What makes it a field in its own right, rather than just a pile of separate analyses, is that the measurements are usually correlated, and the pattern of correlation is often the most interesting thing about them. A student's verbal and mathematical scores move together; a patient's blood pressure and cholesterol move together; a stock's price today is tied to its price yesterday. Multivariate statistics builds the tools for describing that joint structure, testing hypotheses about it, and using it to classify, predict, and simplify.

The field's history is a sequence of responses to that single problem: how to represent a cloud of points in many dimensions, how to test claims about it, and how to do both when the number of variables rivals or exceeds the number of observations.

Correlation and the geometry of the multivariate normal

The subject begins with correlation. Francis Galton, studying heredity in the 1880s, noticed that tall parents tend to have tall children but less tall than themselves—"regression toward mediocrity"—and that two traits could be associated without one causing the other. Karl Pearson gave correlation its modern coefficient in the 1890s, and with it the idea that a set of variables could be summarized by a matrix of pairwise associations rather than examined one at a time.

The theoretical backbone arrived with the multivariate normal distribution: the joint distribution of several variables in which every linear combination is normally distributed, fully described by a mean vector and a covariance matrix (the table of variances and covariances). Its mathematical development is associated with Pearson and later with Harold Hotelling and S. S. Wilks in the 1920s and 1930s. The multivariate normal made the field tractable because it turned questions about a cloud of points into questions about a matrix, and matrix algebra—eigenvalues, determinants, inverses—became the working language. The covariance matrix is the object that most of the classical methods below either decompose, test, or invert.

Principal components: finding the axes of a cloud

The first great data-reduction idea was principal component analysis. Its statistical formulation is credited to Hotelling in 1933 (the underlying mathematics of fitting lines to point clouds goes back further, to Pearson's 1901 work on lines of closest fit). The problem it solved: when you have, say, a dozen correlated measurements, how do you describe the main ways the data vary without drawing a dozen scatterplots? The answer is to find new variables—linear combinations of the originals—that are uncorrelated with one another and ordered so that the first captures as much variance as possible, the second as much of the remainder, and so on1. Geometrically, you rotate the coordinate axes to line up with the long directions of the point cloud1. In practice a dozen measurements might collapse to two or three components that still carry most of the variation, which is why PCA became a standard first look at any multivariate dataset.

Factor analysis: latent causes behind correlations

Factor analysis grew from the same soil but asked a different question1. Charles Spearman, working on intelligence tests around 1904, observed that scores on seemingly different mental tests were all positively correlated, and proposed that a single unobserved "general intelligence" factor could explain the pattern2. The idea generalizes: if many observed variables are correlated, perhaps a few hidden (latent) variables—factors—generate those correlations, with each observed variable loading on the factors plus its own unique noise2. This is a model of the data's causal structure, not merely a rotation of it, and the two approaches have been distinguished ever since. Factor analysis was developed through the 1930s and 1940s by Spearman, Cyril Burt, L. L. Thurstone (whose 1947 Multiple-Factor Analysis was a landmark), and others, and it became the workhorse of psychometrics and later of the social sciences. The distinction between PCA and factor analysis—one a variance-maximizing transformation, the other a latent-variable model—remains a source of confusion and debate.

Testing hypotheses in many dimensions: MANOVA and its relatives

If data are multivariate, so are hypotheses. The 1930s brought a family of tests generalizing the analysis of variance to several response variables at once. Wilks introduced the likelihood-ratio criterion that bears his name in 1932; Hotelling contributed his trace and $T^2$ statistics; and the resulting framework, multivariate analysis of variance (MANOVA), asks whether group means differ across a vector of outcomes jointly rather than one outcome at a time. The motivation is real: running separate univariate tests ignores the correlations among outcomes and inflates the chance of false positives. The price is that "difference in means" is no longer a single number but a direction in a multidimensional space, and different test statistics (Wilks' lambda, Hotelling's trace, Pillai's trace) weight that difference differently. This body of work—multivariate hypothesis testing, confidence regions, and the distribution theory behind them—was largely complete by the 1950s and is the classical core of the field.

Discrimination and classification

A parallel line asked how to sort units into groups. R. A. Fisher's 1936 paper on the iris data—still the most famous dataset in the field—introduced linear discriminant analysis: find the linear combination of measurements that best separates two known groups, then classify new units by which side of a boundary they fall on. The method assumes groups share a covariance structure and differ in means; when that assumption fails, quadratic boundaries follow. Discriminant analysis sits between description and prediction, and it opened the question that dominates modern practice: given labeled examples, how well can a rule classify new cases? The answer depends on estimating error rates honestly, a theme that returns below.

Finding groups without labels: cluster analysis

Where discriminant analysis sorts into known groups, cluster analysis looks for groups that are not yet known1. Its history is more scattered, with roots in numerical taxonomy (biologists grouping organisms by similarity in the 1950s and 1960s), psychology, and engineering. The core methods—hierarchical clustering, which builds a tree of nested groupings; and partitioning methods such as k-means, whose statistical formulation is associated with Stuart Lloyd's work at Bell Labs in the 1950s and 1960s—trade off different notions of what a cluster is34. There is no single correct answer, because "group" must be defined by a distance measure and a criterion, and different choices give different clusters5. This ambiguity is intrinsic, not a defect, and it shaped how the field thinks about unsupervised learning.

Graphical models: correlation as conditional independence

A later strand turned the covariance matrix into a picture. A graphical model represents variables as nodes and conditional independencies as missing edges: if two variables are independent given the rest, no edge connects them6. The Gaussian graphical model, in which the pattern of zeros in the inverse covariance matrix (the precision matrix) encodes these independencies, connects multivariate normal theory to network structure. Early work on Markov random fields and undirected graphs in the 1970s and 1980s (associated with John Besag, Julian Besag, and others in spatial statistics, and with Judea Pearl's later work on directed graphs and causality) gave the field a language for structure. The practical payoff—recovering a network of dependencies from data—became central in genomics, where genes' expression levels are modeled as a graph, and in finance and neuroscience.

The high-dimensional turn: regularization and resampling

The classical methods all assumed more observations than variables, so that the covariance matrix could be estimated and inverted. Modern data broke that assumption: gene expression studies with thousands of genes and dozens of samples, text with tens of thousands of word counts, imaging with millions of voxels7. When variables outnumber observations, the sample covariance matrix is singular and classical tests and inversions fail outright.

The response came from two directions. Regularization, developed from the 1970s onward and crystallized in Robert Tibshirani's 1996 lasso, adds a penalty that shrinks estimates toward simplicity—forcing some coefficients to zero, stabilizing an otherwise unusable covariance estimate, or selecting a sparse set of variables. Ridge regression (Hoerl and Kennard, 1970) and the lasso are the anchors; later methods extended the idea to covariance estimation, graphical models (the graphical lasso), and discriminant analysis. The second response was resampling: the bootstrap, introduced by Bradley Efron in 1979, and permutation methods estimate the variability of a statistic by repeatedly resampling the data rather than relying on formulas that may not hold89. In multivariate settings, where exact distributions are often intractable, resampling made honest inference possible—for example, assessing whether a clustering is real or whether a classifier's apparent accuracy is inflated.

Where the field stands

Today the classical core—multivariate normal theory, PCA, factor analysis, MANOVA, discriminant analysis—is taught as a standard toolkit and remains in daily use. Around it, the high-dimensional and computational strands have become dominant in research: sparse and regularized estimation, resampling-based inference, and graphical models for structure learning. The open questions are recognizably descendants of the old ones. How should a covariance matrix be estimated when variables vastly outnumber observations, and how much structure can be trusted? When a method finds clusters or a network, how do we know the pattern is not an artifact of the algorithm or the sample? And how should the classical tests be adapted when their distributional assumptions—normality, equal covariance, enough data—plainly fail? The field's history is, in large part, a record of taking each of those questions seriously in turn.

References

  1. Multivariate statistics (introduction, Multivariate analysis, Types of analysis), Wikipedia
  2. Factor analysis (History, introduction), Wikipedia
  3. Hierarchical clustering, Wikipedia
  4. K-means clustering (History), Wikipedia
  5. Cluster analysis, Wikipedia
  6. Graphical model, Wikipedia
  7. Clustering high-dimensional data, Wikipedia
  8. Bootstrapping (statistics) (History), Wikipedia
  9. Permutation test, Wikipedia

The frameworks of Multivariate Statistics

Each framework is a way of working the field has used: when it held, and the question it answers.

  1. 1904-Present

    Factor analysis

    What hidden causes explain correlations among measurements?

    Fly to its tower in Nousfarer

  2. 1928-Present

    Multivariate normal distribution

    How do several correlated measurements vary together?

    Fly to its tower in Nousfarer

  3. 1932-Present

    Multivariate analysis of variance

    Do groups differ across many outcomes at once?

    Fly to its tower in Nousfarer

  4. 1933-Present

    Principal component analysis

    How can many measurements be reduced to a few?

    Fly to its tower in Nousfarer

  5. 1936-Present

    Discriminant analysis

    How can units be sorted into known groups?

    Fly to its tower in Nousfarer

  6. 1950-Present

    Cluster analysis

    How can groups be found without labels?

    Fly to its tower in Nousfarer

  7. 1970-Present

    Graphical model

    Which variables are independent given the others?

    Fly to its tower in Nousfarer

  8. 1970-Present

    Regularization

    How can estimates be stabilized when data are scarce?

    Fly to its tower in Nousfarer

  9. 1979-Present

    Resampling

    How reliable is an estimate from limited data?

    Fly to its tower in Nousfarer

How the ideas connect

Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.

Factor analysisMultivariate normal distributionMultivariate analysis of variancePrincipal component analysisDiscriminant analysisCluster analysisGraphical modelRegularizationResampling

Beyond the field

Where Multivariate Statistics's ideas came from in other fields, and what they led to there: the strongest 12 of 28 links.

A city in Nousfarer: a red-carpeted stair climbing terraces to a crowned tower on an acropolis, more towers round it, forest and snowy mountains beyond.
A city in Nousfarer, the game played in the browser.

Multivariate Statistics is a planet in Nousfarer

Its city has a tower for each of these 9 frameworks. Land, walk in, and climb one concept at a time.

Land on Multivariate Statistics

Related fields

Fields its ideas are linked to

More in Statistics

See where Multivariate Statistics sits on the Map of Human Knowledge