- Atlas
- Mathematics
- Bayesian Statistics
Bayesian Statistics
Bayesian statistics is the branch of statistics in which probability is used to describe uncertainty about unknown quantities themselves—a disease rate, the effect of a drug, the mass of a planet—and in which learning is treated as a rule for updating those probabilities when data arrive.
11 frameworks, from 1763 to today · Statistics · Mathematics
- Inverse Probability 1763-1900
- Frequentist Statistics 1900-Present
- Subjective Bayesianism 1931-Present
- Objective Bayesianism 1939-Present
- Bayesian Decision Theory 1950-Present
- Empirical Bayes method 1956-Present
- Bayesian hierarchical modeling 1965-Present
- Posterior Predictive Checking 1984-Present
- Markov Chain Monte Carlo 1990-Present
- Variational Bayesian methods 1998-Present
- Probabilistic Programming 2012-Present
Overview
Bayesian statistics is the branch of statistics in which probability is used to describe uncertainty about unknown quantities themselves—a disease rate, the effect of a drug, the mass of a planet—and in which learning is treated as a rule for updating those probabilities when data arrive1. Its central object is the posterior distribution: a probability distribution over the possible values of an unknown quantity, given what was believed before the data and what the data show12. The field matters because it supplies a complete, internally consistent calculus for reasoning under uncertainty, and because it lets analysts build models of genuinely complicated things—hierarchies of patients within hospitals within regions, images, genomes, language—by composing simpler probabilistic pieces. Its history is a long argument about whether that calculus is the right foundation for scientific inference, and a series of technical breakthroughs that turned its formulas from pen-and-paper ideals into machinery that runs on ordinary computers.
Inverse probability from the dead and the gaming table
The mathematics came first. In the 1650s and 1660s Christiaan Huygens and Blaise Pascal worked out expected values for games of chance, and in 1763 Richard Price published, posthumously, Thomas Bayes's Essay towards solving a Problem in the Doctrine of Chances134. Bayes asked how to reason backward from observed events to an unknown underlying chance, and gave a solution for a special case using a uniform prior35. Pierre-Simon Laplace, in a series of memoirs from the 1770s onward and in his Théorie analytique des probabilités (1812), generalized the idea into a working method: treat an unknown parameter as having a probability distribution, condition on the data, and read off the answer136. Laplace used it for everything from the sex ratio at birth to the mass of Saturn, and he and others applied it to jurisprudence, estimating the reliability of testimony and of jury verdicts63.
The method was called inverse probability, because it inverted the usual direction of probability statements: instead of asking for the chance of data given a hypothesis, it asked for the chance of a hypothesis given data65. The engine was what we now call Bayes's theorem, a consequence of the definition of conditional probability: the posterior is proportional to the prior times the likelihood1. The prior—the probability distribution expressing belief before seeing data—was the controversial ingredient, and in Laplace's hands it was usually uniform, an expression of "insufficient reason" to prefer one value over another67.
The frequentist revolt
Through the nineteenth century inverse probability was the standard tool of mathematical statistics, used by Siméon Denis Poisson, Adolphe Quetelet and Francis Galton, among others8. The revolt came around 1900. Karl Pearson, in founding the modern discipline of mathematical statistics, objected that a parameter such as a population mean is a fixed number, not a random variable, so it cannot sensibly carry a probability distribution; probability should describe the long-run behavior of repeated sampling1. Ronald Fisher sharpened this into a research program in the 1920s: likelihood, sufficiency, and the analysis of variance, all built on the sampling distribution of estimates rather than on priors8. Jerzy Neyman and Egon Pearson (Karl's son) added hypothesis testing and confidence intervals in the 1930s8. By mid-century the dominant textbooks taught that inverse probability was a historical curiosity, and "Bayesian" was often a term of reproach1.
The frequentist objection had real force: a prior is a choice, and different analysts with different priors can reach different posteriors from the same data. The Bayesian reply, developed over decades, was that this is a feature—it makes assumptions explicit—and that in large samples the data swamp the prior anyway.
Subjective probability and the decision-theoretic foundation
The modern revival began with a rethinking of what probability means. In 1931 the Italian actuary Bruno de Finetti published a memoir arguing that probability is a personal degree of belief, coherently held, and that exchangeable observations—whose joint distribution is unchanged by permuting them—behave as if drawn from a common unknown parameter5. Frank Ramsey had sketched a similar view in Truth and Probability (written 1926, published 1931)5. In 1937 John Maynard Keynes's A Treatise on Probability had already treated probability as a logical relation of partial entailment, a different flavor of the same move.
The decisive synthesis was Leonard Savage's The Foundations of Statistics (1954)5. Savage showed that a rational agent's preferences among gambles, if they satisfy a few axioms, are representable as maximizing expected utility with respect to a personal probability distribution. This gave subjective Bayesianism a decision-theoretic foundation: probability is the betting odds you would accept, and Bayes's theorem is how a coherent agent must change those odds on learning. The same period produced Abraham Wald's statistical decision theory (1950), which reframed estimation and testing as choosing among actions under loss, and gave Bayesian procedures a natural home: the Bayes estimator minimizes expected posterior loss59. Bayesian decision theory, as this body of work is called, made the prior a modeling choice with a clear operational meaning.
Priors, and the argument over objectivity
If priors are personal, what should a scientist who wants to be objective do? One answer was to find priors that let the data speak with minimal distortion. Harold Jeffreys, in Theory of Probability (1939, with later editions), developed priors that are invariant under reparameterization—the same prior whether you measure a scale or its logarithm—derived from the Fisher information510. This objective Bayesianism (also called reference analysis) was extended by José Bernardo, James Berger and others from the 1970s onward into a systematic theory of reference priors. The opposing camp, subjective Bayesianism, held that priors should encode genuine expert knowledge and that pretending otherwise merely hides assumptions; de Finetti and Savage are its patron figures, and Dennis Lindley its most forceful mid-century advocate5.
A third route, empirical Bayes, sidestepped the debate by estimating the prior from the data itself—typically from many parallel cases11. Herbert Robbins's 1956 paper on compound decision problems and Bradley Efron and Carl Morris's 1970s work on baseball batting averages made the idea concrete: with many players each having few at-bats, shrinking each player's average toward the group mean predicts future performance better than the raw average does. The prior is not a belief but a quantity estimated from the ensemble.
Hierarchy and shrinkage
Empirical Bayes pointed toward a deeper structure. In many problems the unknowns are not exchangeable in a flat way but organized in groups: students within classrooms, repeated measurements within patients, trials within centers. Hierarchical modeling treats the group-level parameters as themselves drawn from a population distribution, so that information is shared across groups—each group's estimate is pulled toward the population mean, with the amount of pull determined by how much the groups actually differ112. The idea was articulated in the 1960s and 1970s by Lindley and Smith, by David Blackwell and others, and became a standard tool through the 1980s and 1990s, especially in the social and biological sciences. Its practical payoff is shrinkage: small, noisy groups borrow strength from large ones, and the resulting estimates are more stable than either the raw group means or a single pooled mean11.
Making the posterior computable
For most realistic models the posterior has no closed form; the normalizing constant—the probability of the data—is an integral over many dimensions12. For a century this limited Bayesian analysis to conjugate cases, where the prior and likelihood combine into a known family13. The escape came from simulation.
Markov chain Monte Carlo (MCMC) builds a random walk whose long-run distribution is the posterior, so that averages over the walk approximate posterior expectations14. The Metropolis algorithm dates to 1953, when Nicholas Metropolis and colleagues at Los Alamos used it for statistical mechanics; the Metropolis–Hastings generalization came in 19701415. The statistical world took notice after Stuart Geman and Donald Geman's 1984 paper on image analysis and, decisively, after the 1990 paper of Alan Gelfand and Adrian Smith, which showed how to apply the technique to a wide class of Bayesian models14. The Gibbs sampler, a special case, was popularized in the same period by Gelfand and Smith and by the BUGS software project (from 1989), which let users specify a model graphically and have the computer do the sampling14. By the late 1990s MCMC had made Bayesian methods routine in fields from genetics to epidemiology65.
Checking the model
A posterior is only as good as the model that produced it, and the Bayesian tradition had long been criticized for lacking a way to test fit1. The answer that emerged was posterior predictive checking: simulate data from the fitted model and compare the simulations with the observed data on features of interest. The idea was developed in the 1980s and 1990s, notably by Donald Rubin and by Andrew Gelman, John Carlin, Hal Stern and Donald Rubin in their influential Bayesian Data Analysis (first edition 1995)16. Related work on calibration—whether stated probabilities match observed frequencies—and on model comparison through Bayes factors and information criteria gave the field a toolkit for diagnosing and choosing among models, rather than only fitting them.
Computation goes mainstream
The 2000s and 2010s brought two further shifts. The first was probabilistic programming: languages in which a model is written as ordinary code with random choices, and the system compiles it into an inference algorithm17. BUGS was an early ancestor; Stan, released from 2012, and later PyMC, Edward and others made it possible to specify a hierarchical model in a few lines and fit it with automatic differentiation and Hamiltonian Monte Carlo1819. The second was variational inference, which replaces sampling with optimization: approximate the posterior by a simpler distribution chosen to be as close as possible to the true one, and optimize20. The idea goes back to statistical physics in the 1980s and to Michael Jordan and colleagues' work in the late 1990s; it became central to machine learning in the 2010s because it scales to enormous datasets and to deep generative models, at the cost of a biased approximation21.
Where the field stands
Bayesian statistics today is a mainstream part of the discipline, taught alongside frequentist methods and often used in combination with them65. Hierarchical models are standard in the social, environmental and biomedical sciences; MCMC and its variants are the default engine for complex models; probabilistic programming has lowered the barrier to entry; and variational methods have carried Bayesian ideas into machine learning at scale65. The old arguments persist in new form: objective versus subjective priors, the calibration of Bayesian procedures, the reliability of posterior inference when the model is wrong, and the computational cost of scaling exact inference to the largest problems. The field's history is the record of a single idea—that uncertainty about unknowns can be handled by the ordinary rules of probability—being repeatedly declared impractical and repeatedly made practical by new mathematics and new machines.
References
- Bayesian statistics (introduction, Bayes' theorem, Bayesian inference, Statistical modeling, Exploratory analysis of Bayesian models), Wikipedia
- Posterior probability, Wikipedia
- Bayes' theorem (History), Wikipedia
- Decision theory (History), Wikipedia
- Bayesian probability (History, Personal probabilities and objective methods for constructing priors, Objective and subjective Bayesian probabilities), Wikipedia
- Bayesian inference (History), Wikipedia
- Principle of indifference (History), Wikipedia
- Inverse probability, Wikipedia
- Bayes estimator, Wikipedia
- Jeffreys prior, Wikipedia
- Empirical Bayes method, Wikipedia
- Bayesian hierarchical modeling, Wikipedia
- Conjugate prior, Wikipedia
- Markov chain Monte Carlo (introduction, History), Wikipedia
- Metropolis–Hastings algorithm (History), Wikipedia
- Approximate Bayesian computation (History), Wikipedia
- Probabilistic programming, Wikipedia
- Stan (software), Wikipedia
- Hamiltonian Monte Carlo, Wikipedia
- Variational Bayesian methods, Wikipedia
- Reparameterization trick, Wikipedia
The frameworks of Bayesian Statistics
Each framework is a way of working the field has used: when it held, and the question it answers.
- 1763-1900
- 1900-Present
- 1931-Present
- 1939-Present
- 1950-Present
- 1956-Present
- 1965-Present
- 1984-Present
- 1990-Present
- 1998-Present
- 2012-Present
How the ideas connect
Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.
Markov Chain Monte Carlo grew out of Subjective Bayesianism
MCMC methods were developed to allow for the computation of complex posterior distributions that arise when the prior and likelihood do not have a closed-form solution, a common issue in advanced Bayesian modeling.
Carried by Alan Gelfand; Adrian Smith.
Frequentist Statistics reacted against Inverse Probability
The frequentist movement reacted against the notion of assigning probability distributions to fixed parameters, insisting probability describes long-run sampling behavior.
Subjective Bayesianism grew out of Inverse Probability
The concept of probability as a personal degree of belief, central to subjective Bayesianism, was directly proposed by Bruno de Finetti, building on earlier ideas about reasoning backward from chance.
Carried by Bruno de Finetti.
Objective Bayesianism grew out of Inverse Probability
Objective Bayesianism, or reference analysis, sought to address the subjectivity of priors by developing invariance criteria based on information theory, stemming from the general framework of inverse probability.
Carried by Harold Jeffreys.
Bayesian Decision Theory grew out of Inverse Probability
Statistical decision theory, which provided a formal framework for choosing actions under uncertainty, incorporated Bayesian concepts by framing estimation as minimizing expected posterior loss.
Carried by Abraham Wald.
Empirical Bayes method grew out of Inverse Probability
The empirical Bayes method arose as a practical way to address the prior specification problem by estimating the prior distribution from data, building upon the general idea of updating beliefs.
Carried by Herbert Robbins.
Subjective Bayesianism reacted against Frequentist Statistics
The subjective revival reacted against the frequentist view that parameters are fixed constants, reintroducing the idea of probability over unknown quantities.
Objective Bayesianism reacted against Frequentist Statistics
Objective Bayesianism was a direct philosophical response to the perceived arbitrariness of subjective prior choices in the dominant frequentist paradigm.
Bayesian hierarchical modeling grew out of Subjective Bayesianism
Hierarchical modeling extends the idea of modeling unknown parameters by assuming they are drawn from a higher-level distribution, a concept rooted in treating parameters as random variables.
Variational Bayesian methods grew out of Subjective Bayesianism
Variational methods offer an alternative to simulation by approximating the true posterior distribution with a simpler one, building upon the computational challenges of exact inference.
Probabilistic Programming grew out of Subjective Bayesianism
Probabilistic Programming allows users to specify models and run inference automatically, synthesizing the need for flexible modeling with computational tools.
Probabilistic Programming influenced by Markov Chain Monte Carlo
The development of efficient simulation techniques like MCMC provided the necessary computational engine for the practical realization of probabilistic programming languages.
Bayesian Decision Theory grew out of Subjective Bayesianism
Savage's work provided a decision-theoretic foundation for subjective probability, linking personal degrees of belief to rational choice under uncertainty.
Carried by Leonard Savage.
Posterior Predictive Checking grew out of Subjective Bayesianism
Posterior Predictive Checking is a method for validating a model by comparing simulated future data to observed data, extending the core idea of posterior inference.
Bayesian hierarchical modeling influenced by Empirical Bayes method
The idea of estimating parameters from groups of data, central to empirical Bayes, laid groundwork for modeling group-level variation in hierarchical structures.
Posterior Predictive Checking influenced by Bayesian hierarchical modeling
Hierarchical models, by structuring data in groups, naturally lead to the need for methods to check the validity of the resulting group-level inferences.
Markov Chain Monte Carlo influenced by Bayesian hierarchical modeling
The complexity of hierarchical models often necessitated the development of simulation techniques like MCMC to compute their intractable posteriors.
Probabilistic Programming influenced by Variational Bayesian methods
Variational methods provide a deterministic, often faster alternative to sampling methods, influencing the design of automated probabilistic programming tools.
Beyond the field
Where Bayesian Statistics's ideas came from in other fields, and what they led to there: the strongest 12 of 73 links.
Inverse Probability led to Bayesian probability in Measure Theoretic Probability
Bayesian probability is the direct refinement of inverse probability, extending the eighteenth- and nineteenth-century rule for posterior probabilities into a general inferential framework.
Carried by Harold Jeffreys.
Inverse Probability provoked Frequentist Inference in Probability Theory
Frequentist inference defined itself against inverse probability, replacing posterior reasoning with confidence and significance procedures that avoid priors.
Carried by Ronald Fisher; Jerzy Neyman.
Inverse Probability provoked Frequentist Inference in Statistical Inference
Frequentist inference was built as an attack on inverse probability: Fisher dismissed the uniform prior and posterior reasoning of Laplace and Bayes, replacing them with likelihood and sampling-distribution methods.
Carried by Ronald A. Fisher.
Inverse Probability revived by Bayesian Inversion in Inverse Problems
Bayesian inversion deliberately revived the older inverse-probability tradition, recasting ill-posed inverse problems in the Bayesian posterior framework. The earlier inverse-probability formulation of inference from data to causes is the direct ancestor of Bayesian inversion.
Frequentist Statistics influenced Statistical Flood Frequency Analysis in Hydrology
Frequentist statistics supplied the estimation machinery of flood frequency analysis: distribution fitting, confidence intervals and hypothesis tests applied to peak-flow samples. Gumbel's 1941 work on extreme-value distributions and later fitting methods carried this into flood estimation.
Carried by Emil Julius Gumbel.
Frequentist Statistics influenced Statistical Decision Theory in Statistical Inference
Statistical decision theory formalised the frequentist programme by treating tests and estimators as decision rules with risk functions. Wald's work generalised the Neyman-Pearson and Fisherian machinery into a decision framework.
Carried by Abraham Wald.
Frequentist Statistics reacted against Inverse Probability in Statistical Inference
Frequentist statistics defined itself against inverse probability, rejecting priors and posterior distributions in favour of sampling distributions. Fisher's criticism of Bayesian methods and his introduction of likelihood made the rejection explicit.
Carried by Ronald A. Fisher.
Frequentist Statistics led to Robust Statistics in Statistical Inference
Robust statistics grew out of frequentist estimation theory, asking how classical estimators and tests behave under departures from the assumed model. Huber's and Tukey's work refined frequentist estimation with breakdown points and influence functions.
Carried by Peter J. Huber; John Tukey.
Frequentist Statistics influenced Multivariable Modeling in Epidemiology
Frequentist statistics, built on Fisher's maximum likelihood and the Neyman-Pearson framework from the 1920s and 1930s, supplied the estimation and hypothesis-testing machinery that multivariable epidemiologic models use. Regression coefficients, standard errors and confidence intervals in epidemiology are frequentist constructs.
Carried by Ronald A. Fisher; Jerzy Neyman; Egon Pearson.
Subjective Bayesianism led to Bayesian decision theory in Formal Epistemology
Ramsey's and de Finetti's subjective Bayesianism, with its betting-odds account of credence, is the direct basis of Bayesian decision theory; Savage built his 1954 system on that foundation.
Carried by Frank Ramsey; Bruno de Finetti; Leonard Savage.
Subjective Bayesianism influenced Statistical Decision Theory in Statistical Inference
Statistical decision theory takes subjective Bayesian probability and utility as its account of prior belief and loss. Savage's 1954 Foundations of Statistics carried subjective Bayesianism into the decision-theoretic framework.
Carried by Leonard J. Savage; The Foundations of Statistics (1954).
Bayesian Decision Theory influenced Bayesian and decision-analytic diagnosis in Clinical Reasoning
Bayesian decision theory supplied the prior-likelihood-posterior updating and expected-utility framework that clinical decision analysis applied to diagnosis and test interpretation. The medical framework is a direct application of that statistical theory.
Carried by Leonard J. Savage.

Bayesian Statistics is a planet in Nousfarer
Its city has a tower for each of these 11 frameworks. Land, walk in, and climb one concept at a time.
Land on Bayesian StatisticsRelated fields
Fields its ideas are linked to
- Statistical Inference 22 links
- Probability Theory 6 links
- Bayesian Econometrics 5 links
- Probabilistic Ai 3 links
- Biostatistics 3 links
- Measure Theoretic Probability 3 links
- Clinical Decision Science 2 links
- Psychometrics 2 links
- Demographic History 2 links
- Machine Learning 1 link
- Inverse Problems 1 link
- Perception 1 link
More in Statistics
- Causal Inference
- Experimental Design
- Multivariate Statistics
- Nonparametric Statistics
- Regression Analysis
- Spatial Statistics
- Statistical Computing
- Statistical Inference
- Statistical Learning
- Statistics
- Survey Sampling
- Survival Analysis
- Time Series Analysis
See where Bayesian Statistics sits on the Map of Human Knowledge