Survey Sampling

Survey sampling is the branch of statistics that decides which members of a population to measure, and then turns what is measured into estimates of the whole.

12 frameworks, from 1900 to today · Statistics · Mathematics

still a working view ended · each leads to its entry below

Overview

Survey sampling is the branch of statistics that decides which members of a population to measure, and then turns what is measured into estimates of the whole12. Its central problem is practical and unavoidable: you cannot ask every household in a country how much it spends, test every manufactured part, or interview every voter2. You must choose a part, and the way you choose it determines whether your conclusions about the whole are trustworthy. The field's history is the story of how that choice was made rigorous—how a "representative" sample became a probability sample with calculable error, how the machinery of estimation and variance grew around it, and how the field has kept adapting as the ways of reaching people changed from face-to-face visits to random-digit telephone dialing to web panels.

Representative sampling before probability

For most of the nineteenth and early twentieth centuries, the dominant idea was the representative sample: choose units so that the sample mirrors the population's known composition—so many from each region, income band, or occupation. This was the logic behind the quota sample, in which an interviewer is given targets (say, 200 women and 200 men, spread across age groups) and fills them by whatever means. Quota sampling was cheap and, in the hands of skilled practitioners, often produced plausible results; it underpinned much market research and early opinion polling.

Its weakness was structural. Because the interviewer chooses whom to approach within each quota, the probability that any given person enters the sample is unknown, and so the sampling error cannot be calculated. The most famous demonstration came in 1936, when the Literary Digest poll, drawing on millions of returned ballots from telephone and automobile-registration lists, predicted a landslide for Alf Landon; George Gallup's much smaller quota-based poll predicted Roosevelt correctly. The lesson drawn at the time was about sample selection (the Digest's lists skewed wealthy), but the deeper point—that without known selection probabilities you cannot quantify uncertainty—became the foundation of the modern field.

The probability-sample revolution

The decisive turn was the idea of probability sampling: every unit in the population is given a known, nonzero chance of selection, and the sample is drawn by a random mechanism12. This single requirement makes inference possible, because the statistician can compute how often a given sampling procedure would produce estimates far from the truth1. The intellectual architect was the Polish mathematician Jerzy Neyman, whose 1934 paper argued that the representative method should be replaced by random selection and that the confidence interval—a range that would contain the true value in a stated fraction of repeated samples—was the proper way to express uncertainty. Neyman's argument reframed the whole enterprise: randomness, not resemblance, is what licenses inference.

The institutional home of the new field was the United States Department of Agriculture and, later, the Census Bureau and the Bureau of Labor Statistics. In the 1930s and 1940s, statisticians there confronted large national problems—estimating farm acreage, unemployment, crop yields—and built the tools that became standard. Multistage area sampling was one such tool: rather than listing every household, the sampler divides the country into nested units (counties, then blocks, then dwellings) and samples at each stage, so that a manageable number of field visits yields a sample with known probabilities3. This made national surveys feasible without a complete population list3.

The theoretical synthesis came with Morris Hansen and William Hurwitz, whose work at the U.S. Census Bureau in the 1940s and 1950s formalized the design-based approach. In design-based inference, the population values are treated as fixed, and randomness comes entirely from the sampling design; the estimator's properties are evaluated over repeated samples from that design. Hansen and Hurwitz developed the ratio and regression estimators, the theory of sampling with unequal probabilities, and the variance formulas that go with them. Their 1953 textbook, written with William Madow, became the field's standard reference and taught a generation how to design a survey and estimate its error.

Estimating the variance

A probability sample gives an estimate, but a survey is useless without an honest statement of its precision. The classical variance formulas depend on the design—stratification, clustering, unequal weights—and for complex designs they become unwieldy. The practical solution was replication variance estimation: instead of deriving a formula, split the sample into subsamples (replicates), compute the estimate from each, and use the spread among them to estimate the variance4. The two dominant forms are the jackknife, which repeatedly drops one unit (or one group) and recomputes, and the bootstrap, which resamples from the observed data567. These methods, developed from the mid-twentieth century onward, let survey statisticians estimate uncertainty for almost any estimator under almost any design, including the ratio and regression estimators that had no simple closed-form variance.

Weighting and adjustment

Real samples are never perfectly balanced. Some groups are over- or under-represented, some units fail to respond, and the sampler must correct. Weighting assigns each sampled unit an influence proportional to how many population units it stands for—the inverse of its selection probability—and adjustment modifies those weights to compensate for nonresponse and to align the sample with known population totals (a procedure called post-stratification or calibration). The tension is real: weighting corrects bias but inflates variance, and the field developed a body of theory on how to trade the two. This machinery is what lets a survey of a few thousand people speak, however imperfectly, for a nation.

The total survey error framework

By the mid-twentieth century, practitioners recognized that sampling error was only one of many ways a survey can go wrong. The total survey error framework, articulated most influentially by Robert Groves and colleagues, organized the field's thinking around two families of error: sampling error (from measuring only a part) and nonsampling error (coverage, nonresponse, measurement, and processing error). The framework's value was diagnostic—it forced designers to weigh the errors against each other, since spending more to reduce one (say, a larger sample) might leave another (say, nonresponse bias) untouched. It reframed survey quality as a budget-allocation problem across all sources of error, not a single number.

Telephone, then the internet

The rise of the telephone reshaped sampling practice. Early telephone surveys suffered from coverage bias, since phone ownership was uneven1. Random-digit dialing (RDD), developed in the 1960s and refined through the 1970s and 1980s, solved the listing problem: by generating random telephone numbers within known area-code and exchange prefixes, it reached unlisted households and gave every residential number a known chance of selection. RDD became the workhorse of political polling and social research for decades.

Its decline is one of the field's clearest recent turning points. As households abandoned landlines for mobile phones and increasingly screened calls, RDD's coverage and response rates fell, and the cost of reaching a representative sample rose sharply. The field turned to address-based sampling (drawing from postal address frames) and, increasingly, to online panels and other non-probability sources. This has reopened the oldest question in the field: when selection probabilities are unknown, can weighting and modeling still deliver trustworthy estimates? Much current methodological work—on calibration, on propensity adjustment, on combining probability and non-probability samples—is an attempt to answer it.

Model-based and model-assisted inference

The design-based orthodoxy was never the only option. Model-based inference treats the population values as random draws from a statistical model and bases estimates and their uncertainty on that model rather than on the sampling design. Its appeal is that it can borrow strength across variables and handle small samples where design-based variance is unstable. The risk is model misspecification: if the model is wrong, the inference is wrong, and the design offers no protection.

Model-assisted estimation is the pragmatic middle ground, developed especially from the 1980s onward. It uses a model to improve an estimator—typically a regression or ratio estimator—but calibrates the result so that it remains approximately design-unbiased even if the model is imperfect. The generalized regression (GREG) estimator is the canonical example: it uses auxiliary data (say, known population totals from a census) to sharpen estimates while retaining design-based guarantees. This synthesis—models for efficiency, design for validity—is now standard.

Small-area estimation and adaptive designs

Two further developments extended the field's reach. Small-area estimation addresses the problem of producing estimates for domains (a small county, a demographic subgroup) where the sample is too thin for direct estimation8. The solution is to borrow strength from related areas and auxiliary data through models—often hierarchical or Bayesian—yielding estimates with acceptable precision where direct design-based estimates would be hopeless. This has become essential for official statistics and health and demographic research.

Adaptive and network sampling tackle populations that are rare, hidden, or clustered—people with a rare disease, or members of a social network19. In adaptive designs, the sampling rule changes as data accumulate (for example, sampling more intensively where cases are found). In network or respondent-driven sampling, participants recruit others from their social network, and the resulting referral chains are used to estimate the size and characteristics of the hidden population19. These designs trade the clean probabilities of classical sampling for reach into populations no list can capture, and their theory—how to weight and estimate from such chains—remains an active research frontier.

The field today

Contemporary survey sampling is a plural enterprise. Design-based inference remains the backbone of official statistics and large government surveys, where known selection probabilities and replication-based variance estimation are the norm. Model-assisted methods are routine wherever auxiliary data exist. Model-based and Bayesian approaches dominate small-area estimation and are increasingly used to combine disparate data sources. The rise of non-probability data—web panels, administrative records, and "big data" streams—has pushed the field toward methods that diagnose and correct selection bias through modeling and calibration, and toward total survey error as the organizing framework for weighing every source of error at once. The open questions are the old ones in new dress: how to quantify uncertainty when selection probabilities are unknown, how to integrate probability and non-probability sources without fooling oneself, and how to keep inference honest as the ways of reaching people keep changing.

References

  1. Survey sampling (introduction, Probability sampling, Non-sampling error in probability sampling, Non-probability sampling), Wikipedia
  2. Sampling (statistics) (introduction, Sampling frame), Wikipedia
  3. Multistage sampling, Wikipedia
  4. Imputation and Variance Estimation Software, Wikipedia
  5. Resampling (statistics), Wikipedia
  6. Jackknife resampling, Wikipedia
  7. Bootstrapping (statistics), Wikipedia
  8. Small area estimation, Wikipedia
  9. Snowball sampling, Wikipedia

The frameworks of Survey Sampling

Each framework is a way of working the field has used: when it held, and the question it answers.

  1. 1900-1948

    Representative and Quota Sampling

    How should a sample mirror the population?

    Fly to its tower in Nousfarer

  2. 1930-Present

    Multistage sampling

    How can national samples be drawn without a full list?

    Fly to its tower in Nousfarer

  3. 1934-Present

    Design-Based Inference

    How does random selection license inference?

    Fly to its tower in Nousfarer

  4. 1940-Present

    Weighting and Calibration Adjustment

    How should weights correct for imbalance and nonresponse?

    Fly to its tower in Nousfarer

  5. 1950-Present

    Replication Variance Estimation

    How can variance be estimated for complex designs?

    Fly to its tower in Nousfarer

  6. 1960-2005

    Random-Digit Dialing

    How can telephone households be sampled with known chances?

    Fly to its tower in Nousfarer

  7. 1970-Present

    Model-Based Inference

    When should inference rest on a statistical model?

    Fly to its tower in Nousfarer

  8. 1970-Present

    Small-Area Estimation

    How can estimates be made for small domains?

    Fly to its tower in Nousfarer

  9. 1970-Present

    Total Survey Error

    What are all the ways a survey can go wrong?

    Fly to its tower in Nousfarer

  10. 1980-Present

    Model-Assisted Estimation

    How can models improve estimates without biasing them?

    Fly to its tower in Nousfarer

  11. 1990-Present

    Adaptive and Network Sampling

    How should sampling adapt as data are collected?

    Fly to its tower in Nousfarer

  12. 2005-Present

    Non-Probability and Blended-Source Inference

    Can non-probability samples yield trustworthy estimates?

    Fly to its tower in Nousfarer

How the ideas connect

Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.

Representative and Quota SamplingMultistage samplingDesign-Based InferenceWeighting and Calibration AdjustmentReplication Variance EstimationRandom-Digit DialingModel-Based InferenceSmall-Area EstimationTotal Survey ErrorModel-Assisted EstimationAdaptive and Network SamplingNon-Probability and Blended-Source In…

Beyond the field

Where Survey Sampling's ideas came from in other fields, and what they led to there: the strongest 12 of 43 links.

A city in Nousfarer: a red-carpeted stair climbing terraces to a crowned tower on an acropolis, more towers round it, forest and snowy mountains beyond.
A city in Nousfarer, the game played in the browser.

Survey Sampling is a planet in Nousfarer

Its city has a tower for each of these 12 frameworks. Land, walk in, and climb one concept at a time.

Land on Survey Sampling

Related fields

Fields its ideas are linked to

More in Statistics

See where Survey Sampling sits on the Map of Human Knowledge