Bayesian Statistics

Bayesian statistics is the branch of statistics in which probability is used to describe uncertainty about unknown quantities themselves—a disease rate, the effect of a drug, the mass of a planet—and in which learning is treated as a rule for updating those probabilities when data arrive.

11 frameworks, from 1763 to today · Statistics · Mathematics

still a working view ended · each leads to its entry below

Overview

Bayesian statistics is the branch of statistics in which probability is used to describe uncertainty about unknown quantities themselves—a disease rate, the effect of a drug, the mass of a planet—and in which learning is treated as a rule for updating those probabilities when data arrive1. Its central object is the posterior distribution: a probability distribution over the possible values of an unknown quantity, given what was believed before the data and what the data show12. The field matters because it supplies a complete, internally consistent calculus for reasoning under uncertainty, and because it lets analysts build models of genuinely complicated things—hierarchies of patients within hospitals within regions, images, genomes, language—by composing simpler probabilistic pieces. Its history is a long argument about whether that calculus is the right foundation for scientific inference, and a series of technical breakthroughs that turned its formulas from pen-and-paper ideals into machinery that runs on ordinary computers.

Inverse probability from the dead and the gaming table

The mathematics came first. In the 1650s and 1660s Christiaan Huygens and Blaise Pascal worked out expected values for games of chance, and in 1763 Richard Price published, posthumously, Thomas Bayes's Essay towards solving a Problem in the Doctrine of Chances134. Bayes asked how to reason backward from observed events to an unknown underlying chance, and gave a solution for a special case using a uniform prior35. Pierre-Simon Laplace, in a series of memoirs from the 1770s onward and in his Théorie analytique des probabilités (1812), generalized the idea into a working method: treat an unknown parameter as having a probability distribution, condition on the data, and read off the answer136. Laplace used it for everything from the sex ratio at birth to the mass of Saturn, and he and others applied it to jurisprudence, estimating the reliability of testimony and of jury verdicts63.

The method was called inverse probability, because it inverted the usual direction of probability statements: instead of asking for the chance of data given a hypothesis, it asked for the chance of a hypothesis given data65. The engine was what we now call Bayes's theorem, a consequence of the definition of conditional probability: the posterior is proportional to the prior times the likelihood1. The prior—the probability distribution expressing belief before seeing data—was the controversial ingredient, and in Laplace's hands it was usually uniform, an expression of "insufficient reason" to prefer one value over another67.

The frequentist revolt

Through the nineteenth century inverse probability was the standard tool of mathematical statistics, used by Siméon Denis Poisson, Adolphe Quetelet and Francis Galton, among others8. The revolt came around 1900. Karl Pearson, in founding the modern discipline of mathematical statistics, objected that a parameter such as a population mean is a fixed number, not a random variable, so it cannot sensibly carry a probability distribution; probability should describe the long-run behavior of repeated sampling1. Ronald Fisher sharpened this into a research program in the 1920s: likelihood, sufficiency, and the analysis of variance, all built on the sampling distribution of estimates rather than on priors8. Jerzy Neyman and Egon Pearson (Karl's son) added hypothesis testing and confidence intervals in the 1930s8. By mid-century the dominant textbooks taught that inverse probability was a historical curiosity, and "Bayesian" was often a term of reproach1.

The frequentist objection had real force: a prior is a choice, and different analysts with different priors can reach different posteriors from the same data. The Bayesian reply, developed over decades, was that this is a feature—it makes assumptions explicit—and that in large samples the data swamp the prior anyway.

Subjective probability and the decision-theoretic foundation

The modern revival began with a rethinking of what probability means. In 1931 the Italian actuary Bruno de Finetti published a memoir arguing that probability is a personal degree of belief, coherently held, and that exchangeable observations—whose joint distribution is unchanged by permuting them—behave as if drawn from a common unknown parameter5. Frank Ramsey had sketched a similar view in Truth and Probability (written 1926, published 1931)5. In 1937 John Maynard Keynes's A Treatise on Probability had already treated probability as a logical relation of partial entailment, a different flavor of the same move.

The decisive synthesis was Leonard Savage's The Foundations of Statistics (1954)5. Savage showed that a rational agent's preferences among gambles, if they satisfy a few axioms, are representable as maximizing expected utility with respect to a personal probability distribution. This gave subjective Bayesianism a decision-theoretic foundation: probability is the betting odds you would accept, and Bayes's theorem is how a coherent agent must change those odds on learning. The same period produced Abraham Wald's statistical decision theory (1950), which reframed estimation and testing as choosing among actions under loss, and gave Bayesian procedures a natural home: the Bayes estimator minimizes expected posterior loss59. Bayesian decision theory, as this body of work is called, made the prior a modeling choice with a clear operational meaning.

Priors, and the argument over objectivity

If priors are personal, what should a scientist who wants to be objective do? One answer was to find priors that let the data speak with minimal distortion. Harold Jeffreys, in Theory of Probability (1939, with later editions), developed priors that are invariant under reparameterization—the same prior whether you measure a scale or its logarithm—derived from the Fisher information510. This objective Bayesianism (also called reference analysis) was extended by José Bernardo, James Berger and others from the 1970s onward into a systematic theory of reference priors. The opposing camp, subjective Bayesianism, held that priors should encode genuine expert knowledge and that pretending otherwise merely hides assumptions; de Finetti and Savage are its patron figures, and Dennis Lindley its most forceful mid-century advocate5.

A third route, empirical Bayes, sidestepped the debate by estimating the prior from the data itself—typically from many parallel cases11. Herbert Robbins's 1956 paper on compound decision problems and Bradley Efron and Carl Morris's 1970s work on baseball batting averages made the idea concrete: with many players each having few at-bats, shrinking each player's average toward the group mean predicts future performance better than the raw average does. The prior is not a belief but a quantity estimated from the ensemble.

Hierarchy and shrinkage

Empirical Bayes pointed toward a deeper structure. In many problems the unknowns are not exchangeable in a flat way but organized in groups: students within classrooms, repeated measurements within patients, trials within centers. Hierarchical modeling treats the group-level parameters as themselves drawn from a population distribution, so that information is shared across groups—each group's estimate is pulled toward the population mean, with the amount of pull determined by how much the groups actually differ112. The idea was articulated in the 1960s and 1970s by Lindley and Smith, by David Blackwell and others, and became a standard tool through the 1980s and 1990s, especially in the social and biological sciences. Its practical payoff is shrinkage: small, noisy groups borrow strength from large ones, and the resulting estimates are more stable than either the raw group means or a single pooled mean11.

Making the posterior computable

For most realistic models the posterior has no closed form; the normalizing constant—the probability of the data—is an integral over many dimensions12. For a century this limited Bayesian analysis to conjugate cases, where the prior and likelihood combine into a known family13. The escape came from simulation.

Markov chain Monte Carlo (MCMC) builds a random walk whose long-run distribution is the posterior, so that averages over the walk approximate posterior expectations14. The Metropolis algorithm dates to 1953, when Nicholas Metropolis and colleagues at Los Alamos used it for statistical mechanics; the Metropolis–Hastings generalization came in 19701415. The statistical world took notice after Stuart Geman and Donald Geman's 1984 paper on image analysis and, decisively, after the 1990 paper of Alan Gelfand and Adrian Smith, which showed how to apply the technique to a wide class of Bayesian models14. The Gibbs sampler, a special case, was popularized in the same period by Gelfand and Smith and by the BUGS software project (from 1989), which let users specify a model graphically and have the computer do the sampling14. By the late 1990s MCMC had made Bayesian methods routine in fields from genetics to epidemiology65.

Checking the model

A posterior is only as good as the model that produced it, and the Bayesian tradition had long been criticized for lacking a way to test fit1. The answer that emerged was posterior predictive checking: simulate data from the fitted model and compare the simulations with the observed data on features of interest. The idea was developed in the 1980s and 1990s, notably by Donald Rubin and by Andrew Gelman, John Carlin, Hal Stern and Donald Rubin in their influential Bayesian Data Analysis (first edition 1995)16. Related work on calibration—whether stated probabilities match observed frequencies—and on model comparison through Bayes factors and information criteria gave the field a toolkit for diagnosing and choosing among models, rather than only fitting them.

Computation goes mainstream

The 2000s and 2010s brought two further shifts. The first was probabilistic programming: languages in which a model is written as ordinary code with random choices, and the system compiles it into an inference algorithm17. BUGS was an early ancestor; Stan, released from 2012, and later PyMC, Edward and others made it possible to specify a hierarchical model in a few lines and fit it with automatic differentiation and Hamiltonian Monte Carlo1819. The second was variational inference, which replaces sampling with optimization: approximate the posterior by a simpler distribution chosen to be as close as possible to the true one, and optimize20. The idea goes back to statistical physics in the 1980s and to Michael Jordan and colleagues' work in the late 1990s; it became central to machine learning in the 2010s because it scales to enormous datasets and to deep generative models, at the cost of a biased approximation21.

Where the field stands

Bayesian statistics today is a mainstream part of the discipline, taught alongside frequentist methods and often used in combination with them65. Hierarchical models are standard in the social, environmental and biomedical sciences; MCMC and its variants are the default engine for complex models; probabilistic programming has lowered the barrier to entry; and variational methods have carried Bayesian ideas into machine learning at scale65. The old arguments persist in new form: objective versus subjective priors, the calibration of Bayesian procedures, the reliability of posterior inference when the model is wrong, and the computational cost of scaling exact inference to the largest problems. The field's history is the record of a single idea—that uncertainty about unknowns can be handled by the ordinary rules of probability—being repeatedly declared impractical and repeatedly made practical by new mathematics and new machines.

References

  1. Bayesian statistics (introduction, Bayes' theorem, Bayesian inference, Statistical modeling, Exploratory analysis of Bayesian models), Wikipedia
  2. Posterior probability, Wikipedia
  3. Bayes' theorem (History), Wikipedia
  4. Decision theory (History), Wikipedia
  5. Bayesian probability (History, Personal probabilities and objective methods for constructing priors, Objective and subjective Bayesian probabilities), Wikipedia
  6. Bayesian inference (History), Wikipedia
  7. Principle of indifference (History), Wikipedia
  8. Inverse probability, Wikipedia
  9. Bayes estimator, Wikipedia
  10. Jeffreys prior, Wikipedia
  11. Empirical Bayes method, Wikipedia
  12. Bayesian hierarchical modeling, Wikipedia
  13. Conjugate prior, Wikipedia
  14. Markov chain Monte Carlo (introduction, History), Wikipedia
  15. Metropolis–Hastings algorithm (History), Wikipedia
  16. Approximate Bayesian computation (History), Wikipedia
  17. Probabilistic programming, Wikipedia
  18. Stan (software), Wikipedia
  19. Hamiltonian Monte Carlo, Wikipedia
  20. Variational Bayesian methods, Wikipedia
  21. Reparameterization trick, Wikipedia

The frameworks of Bayesian Statistics

Each framework is a way of working the field has used: when it held, and the question it answers.

  1. 1763-1900
  2. 1900-Present
  3. 1931-Present
  4. 1939-Present
  5. 1950-Present
  6. 1956-Present
  7. 1965-Present
  8. 1984-Present
  9. 1990-Present
  10. 1998-Present
  11. 2012-Present

How the ideas connect

Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.

Inverse ProbabilityFrequentist StatisticsSubjective BayesianismObjective BayesianismBayesian Decision TheoryEmpirical Bayes methodBayesian hierarchical modelingPosterior Predictive CheckingMarkov Chain Monte CarloVariational Bayesian methodsProbabilistic Programming

Beyond the field

Where Bayesian Statistics's ideas came from in other fields, and what they led to there: the strongest 12 of 73 links.

A city in Nousfarer: a red-carpeted stair climbing terraces to a crowned tower on an acropolis, more towers round it, forest and snowy mountains beyond.
A city in Nousfarer, the game played in the browser.

Bayesian Statistics is a planet in Nousfarer

Its city has a tower for each of these 11 frameworks. Land, walk in, and climb one concept at a time.

Land on Bayesian Statistics

Related fields

Fields its ideas are linked to

More in Statistics

See where Bayesian Statistics sits on the Map of Human Knowledge