Statistical Computing

Statistical computing is the part of statistics that turns methods into working calculations.

11 frameworks, from 1800 to today · Statistics · Mathematics

still a working view ended · each leads to its entry below

Overview

Statistical computing is the part of statistics that turns methods into working calculations1. Its subject is not a body of theory about probability but the machinery—human, mechanical, and electronic—by which estimates, tests, and models are actually produced from data. The stakes are easy to miss until something goes wrong: a regression that silently loses precision, a random-number generator that repeats, a simulation that cannot be reproduced, or an analysis that takes a year instead of an afternoon. The history of the field is a history of removing those obstacles, and of the new statistical methods that became thinkable once the obstacles fell.

Hand computation and the table tradition

For most of the discipline's history, statistical computation was arithmetic done by people, and the central technology was the printed table. Nineteenth-century statistical offices and actuarial firms employed "computers"—staff who carried out long multiplication and summation by hand or with slide rules—and published results in massive tabular volumes. The mathematical tables of the eighteenth and nineteenth centuries, and later the great statistical tables of the early twentieth, were the field's shared infrastructure: a worker looked up a logarithm, a normal-tail probability, or a critical value of t rather than deriving it.

The table tradition shaped statistical practice in ways that outlasted the tables themselves. Because each lookup cost time, methods were designed to need few of them; because tables were finite, interpolation rules mattered; because errors in transcription were common, checking procedures became part of the craft. When Karl Pearson and his collaborators at University College London built the biometric laboratory in the 1890s and after, they combined hand calculation with newly available mechanical calculators and produced the tables of the distributions—chi-square, correlation coefficients—that their own methods required. The same period saw the rise of the mechanical desk calculator, which made routine arithmetic faster but did not change the underlying logic: computation was still a sequence of individually supervised operations.

Punched cards and the mechanization of data

The first real automation of statistical work came from data processing rather than from mathematics. Herman Hollerith's punched-card system, developed for the 1890 U.S. Census, encoded each person's responses as holes in a card and used electromechanical tabulators to count combinations. The approach spread to census offices, insurance companies, and later to scientific laboratories. Punched cards did not compute statistical estimates in a modern sense; they sorted, counted, and cross-tabulated at a scale no clerical staff could match. That was enough to change what surveys and censuses could attempt, and it established a division of labor—data preparation on cards, analysis by other means—that persisted into the early electronic era.

Bespoke programming and the numerical analysis of algorithms

The stored-program electronic computer, available to statisticians from the late 1940s and 1950s, changed the problem from "how do we do this arithmetic" to "how do we write instructions that do it correctly." Early statistical computing was bespoke: a statistician or a programmer wrote a specific program for a specific analysis on a specific machine, often in machine code or assembly. There was no general statistical software to buy.

This era produced a body of work that is easy to overlook because it lives inside other methods: the numerical analysis of statistical algorithms. The question was whether a formula that looked fine on paper would behave when evaluated in finite-precision arithmetic. Sums of squares could lose accuracy catastrophically; matrix inversions could be unstable; iterative procedures might fail to converge or converge to the wrong answer. Work on stable algorithms for regression, on the QR decomposition and other orthogonalization methods, and on the design of random-number generators addressed these problems directly. The quality of a statistical method came to depend on the quality of its implementation, and statisticians began to treat algorithm design as part of their subject rather than an afterthought.

Statistical package culture

The 1960s and 1970s brought the decisive shift: general-purpose statistical packages that a researcher could use without writing the underlying code. The BMDP series, SPSS (from 1968), SAS (from the early 1970s), and later Minitab and others offered menus of standard analyses—regression, analysis of variance, contingency tables—with data management attached. Their appeal was enormous: a scientist with a dataset could obtain standard output without a programmer.

Package culture had costs as well as benefits. Because the package decided which analyses were easy, it shaped which analyses were done; methods not implemented were effectively invisible to many users. Packages also standardized output formats and conventions, which helped communication but encouraged treating a procedure as a black box. Statisticians debated whether this was liberation or a loss of understanding, and the debate itself became part of the field's self-image.

Languages, open source, and the S tradition

A parallel strand built tools that were programmable rather than menu-driven. The S language, developed at Bell Laboratories from the mid-1970s by John Chambers and colleagues, was designed around the idea that statistical analysis is interactive exploration: data objects, functions, and graphics composed by the user. S's descendants—most importantly R, released in the early 1990s by Ross Ihaka and Robert Gentleman and subsequently maintained by a large open-source community—made a full programming language the substrate of statistical work. The open-source ecosystem that grew around R, and later around Python's scientific stack, changed the economics of the field: methods could be shared as code, extended by anyone, and inspected rather than trusted.

Resampling, simulation, and Markov chain Monte Carlo

Cheap computation did more than speed up old methods; it made new inferential strategies practical. Resampling methods—the jackknife, promoted by Bradley Efron in the late 1970s, and the bootstrap, introduced by Efron in 1979—estimated the variability of a statistic by repeatedly recomputing it from resampled data, sidestepping formulas that were hard or unavailable. Simulation-based inference more generally let statisticians study the behavior of procedures by generating data from a known model and watching what the procedure did.

The most consequential computational method of the era was Markov chain Monte Carlo (MCMC): constructing a random walk whose long-run distribution is the posterior or likelihood of interest, then summarizing that distribution from the walk's visited states2. The Metropolis algorithm dates to 1953, and the Metropolis–Hastings generalization to 1970, but MCMC became a routine statistical tool in the late 1980s and 1990s, when it was applied to Bayesian hierarchical models that had no closed-form solution. The Gibbs sampler, popularized in statistics around 1990, made the approach accessible for many models. MCMC turned Bayesian analysis from a set of conjugate special cases into a general methodology, and it made computation, not algebra, the binding constraint on model complexity.

Reproducible and literate analysis

As analyses grew into long chains of code, a new problem appeared: could anyone else—or the original analyst a year later—reproduce the result? The reproducible-research movement addressed this by tying code, data, and narrative together. Literate programming, Donald Knuth's idea from the early 1980s that a program should be written as an explanation with code embedded, was adapted to statistics in tools such as Sweave (early 2000s) and later knitr for R, and in notebook systems for Python and other languages. The goal was a document in which the reported numbers are generated by the code shown, so that the analysis is auditable and rerunnable. This changed expectations about what a statistical publication should contain.

Probabilistic programming and automatic differentiation

The most recent turn makes the model itself the program. Probabilistic programming languages let a user write down a generative model—variables, priors, likelihoods—and leave the inference algorithm to the system3. Automatic differentiation, the technique of computing exact derivatives of a program's output with respect to its inputs by applying the chain rule mechanically, is the engine that makes this practical for gradient-based fitting. Together they have moved Bayesian modeling toward a declarative style in which the analyst specifies the model and the software handles the computation, and they connect statistical computing to the broader machine-learning toolchain.

Scalable and approximate computation

The present frontier is scale. Datasets too large to fit in memory, models with millions of parameters, and inference that would take centuries by naive MCMC have pushed the field toward approximate and distributed methods. Variational inference replaces exact posterior computation with an optimization problem that is faster but only approximate; stochastic gradient methods and subsampling make each step cheap; approximate Bayesian computation handles models whose likelihood cannot be written down; and distributed and streaming algorithms spread work across machines. These approaches trade exactness for feasibility, and the trade-offs—how much accuracy is lost, and when the approximation misleads—are active research questions.

Today the field's main strands all descend from this history: the numerical care of the early algorithm work, the programmability of the S tradition, the inferential reach of MCMC and resampling, the reproducibility norms of literate analysis, and the declarative, differentiable style of probabilistic programming. What remains open is largely the same question in new clothes: how to compute a trustworthy answer when the model, the data, or both are too big for the straightforward calculation—and how to know when an approximation has stopped being good enough.

References

  1. Computational statistics, Wikipedia
  2. Markov chain Monte Carlo, Wikipedia
  3. Probabilistic programming, Wikipedia

The frameworks of Statistical Computing

Each framework is a way of working the field has used: when it held, and the question it answers.

  1. 1800-1950

    Mathematical table

    How can numerical values be looked up reliably?

    Fly to its tower in Nousfarer

  2. 1890-1960

    Unit record equipment

    How can data be tabulated at scale?

    Fly to its tower in Nousfarer

  3. 1945-1965

    Bespoke statistical programming

    How do we write instructions that compute correctly?

    Fly to its tower in Nousfarer

  4. 1950-Present
  5. 1965-2000

    Statistical package culture

    How can standard analyses be made easy to run?

    Fly to its tower in Nousfarer

  6. 1975-Present

    Resampling

    How can variability be estimated by recomputation?

    Fly to its tower in Nousfarer

  7. 1975-Present

    Statistical programming languages

    How can statistical work be programmed interactively?

    Fly to its tower in Nousfarer

  8. 1985-Present

    Markov chain Monte Carlo

    How can complex posteriors be sampled?

    Fly to its tower in Nousfarer

  9. 2000-Present
  10. 2005-Present
  11. 2010-Present

How the ideas connect

Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.

Mathematical tableUnit record equipmentBespoke statistical programmingNumerical analysis of statistical alg…Statistical package cultureResamplingStatistical programming languagesMarkov chain Monte CarloReproducible and literate analysisScalable and approximate computationProbabilistic programming and automat…

Beyond the field

Where Statistical Computing's ideas came from in other fields, and what they led to there: the strongest 12 of 49 links.

A city in Nousfarer: a red-carpeted stair climbing terraces to a crowned tower on an acropolis, more towers round it, forest and snowy mountains beyond.
A city in Nousfarer, the game played in the browser.

Statistical Computing is a planet in Nousfarer

Its city has a tower for each of these 11 frameworks. Land, walk in, and climb one concept at a time.

Land on Statistical Computing

Related fields

Fields its ideas are linked to

More in Statistics

See where Statistical Computing sits on the Map of Human Knowledge