Panel Data Methods
Panel data are observations of the same units—people, firms, countries, schools—followed over several periods.
9 frameworks, from 1940 to today · Econometrics · Economics
- Random effects model 1940-1975
- Fixed Effects and Random Effects Estimation 1968-Present
- Random Effects and Correlated Random Effects 1971-Present
- Nonlinear Panel Models with Unobserved Heterogeneity 1980-Present
- Dynamic Panel Instrumental-Variable Estimation 1981-Present
- Clustered standard errors 1984-Present
- Difference in differences 1985-Present
- Interactive Fixed Effects and Factor-Augmented Panels 2000-Present
- Machine Learning and Regularized Panel Estimation 2015-Present
Overview
Panel data are observations of the same units—people, firms, countries, schools—followed over several periods12. The same household is interviewed in 1990, 1991, and 1992; the same firm files accounts year after year; the same country appears in cross-national statistics decade after decade. What makes such data special is that they let a researcher compare a unit with itself over time, so that fixed, unmeasured differences between units—a person's innate ability, a firm's corporate culture, a country's geography—can be stripped away. That is the central promise of the field, and most of its history is a long effort to make good on it: to separate what changes within units from what differs between them, and to do so without fooling oneself.
The stakes are practical. If you want to know whether a job-training program raises earnings, comparing trainees with non-trainees confounds the program with the kinds of people who enroll. If you want to know whether a minimum wage destroys jobs, comparing states with different wage floors confounds policy with the states' underlying economies. Panel methods are the toolkit econometricians built to answer such questions with observational data, and their influence spread from labor economics into macroeconomics, finance, development, and public policy.
The between-and-within decomposition and the error-components idea
The statistical foundations were laid in the mid-twentieth century. In the 1940s and 1950s, work on the analysis of variance and on "error components"—models in which an observation's error is split into a unit-specific part and a transitory part—gave economists a formal way to think about repeated observations. The key insight is simple to state: variation in panel data has two sources, variation between units (differences in their averages) and variation within units (deviations from each unit's own average). A regression that uses only within variation is immune to any unmeasured characteristic of the unit that does not change over time1.
By the late 1960s and early 1970s this idea had become an estimation strategy. The fixed effects estimator—often called the within estimator—transforms the data by subtracting each unit's time-average, so that all time-invariant differences vanish, and then runs ordinary least squares on the transformed data. The random effects estimator instead treats the unit-specific term as a random draw from a distribution, uncorrelated with the regressors, and exploits both within and between variation, gaining efficiency if that assumption holds. The trade-off is the heart of the field: fixed effects are robust but discard between-unit information and cannot estimate the effect of anything that never changes (a person's sex, a country's legal origin); random effects use more information but are biased if the unobserved unit effect is correlated with the regressors—which, in economics, it usually is. A specification test proposed by Jerry Hausman in 1978 gave practitioners a formal way to choose, by checking whether the two estimators differ significantly. The fixed-versus-random debate, and the habit of treating fixed effects as the default in microeconomic applications, shaped how a generation was taught.
Dynamic panels and the Nickell bias
A harder problem appeared when researchers wanted to include a lagged dependent variable—yesterday's outcome as a predictor of today's—which is natural for questions about persistence: does a firm's investment this year depend on last year's1? Does poverty persist? With fixed effects and short panels, the within transformation induces a mechanical correlation between the transformed lagged outcome and the transformed error, biasing the coefficient. Stephen Nickell documented this in 1981: the bias is substantial when the number of time periods is small, and it does not disappear as the number of units grows.
The response, developed through the 1980s and 1990s, was a family of instrumental-variable approaches that use deeper lags and lagged differences as instruments for the endogenous lagged variable. The Arellano–Bond estimator (1991) generalized earlier work by Manuel Arellano and Stephen Bond using moment conditions from the differenced equations; the Arellano–Bover/Blundell–Bond system estimator (1995, 1998) added conditions from the levels equations to improve performance when the series are highly persistent. These methods became standard in empirical growth and labor research, though they are fragile when instruments are weak—a recurring caution in the literature.
Nonlinear models with unobserved heterogeneity
Economic outcomes are often not continuous: whether a person works, whether a firm exits, how many children a household has. Extending panel methods to nonlinear models—logit, probit, count, and duration models—raised a new difficulty. In a linear model, differencing or within-transformation eliminates the fixed effect cleanly. In a nonlinear model, the fixed effect enters nonlinearly, and the transformation no longer cancels it. The incidental parameters problem, analyzed by J. Neyman and Elizabeth Scott in 1948, is that estimating a separate parameter for each unit in a short panel produces inconsistent estimates of the common parameters.
Econometricians found conditional and marginal solutions. For the fixed-effects logit, Gary Chamberlain (1980) showed that conditioning on the total number of successes across periods eliminates the unit effect, yielding consistent estimates. For other models, random-effects specifications with distributional assumptions, or correlated random-effects approaches that model the unit effect as a function of the regressors' averages (associated with Chamberlain, 1982, and later Mundlak), offered practical routes. The literature on nonlinear panels remains technically demanding, and no single method dominates; the choice depends on the model and the question.
Difference-in-differences and the credibility revolution
Parallel to these statistical developments, a design-based tradition grew up around a simple comparison. Difference-in-differences (DiD) compares the change in an outcome for a group affected by a policy with the change for an unaffected group, differencing out both fixed group differences and common time trends3. Its modern econometric formalization is often traced to Ashenfelter and Card's 1985 study of training programs, building on earlier applications in the 1970s. The method's appeal is transparency: under the "parallel trends" assumption—that the two groups would have moved in parallel absent the policy—a simple regression identifies the effect.
DiD became the workhorse of the "credibility revolution" in applied economics from the 1990s onward, when researchers sought natural experiments and quasi-experimental designs. Its assumptions were later scrutinized: when units are treated at different times, the standard two-way fixed-effects regression can weight comparisons in unintended ways, and a literature from the late 2010s (notably work by Andrew Goodman-Bacon, and by Brantly Callaway and Pedro Sant'Anna) developed alternative estimators and diagnostics. This remains one of the most active areas of the field.
Interactive fixed effects and factor structure
Standard fixed effects assume each unit's unobserved term is constant over time1. But some unobservables evolve—a country's political climate, a firm's managerial quality—and may do so in ways shared across units. Interactive fixed effects models allow the unit effect to interact with time-varying factors: the unobserved term is a product of unit-specific "loadings" and common time factors, much like a factor model. This framework, developed in the 2000s by Jushan Bai and others, generalizes both fixed effects and common time effects and connects panel methods to the factor-structure literature in time series. Estimation typically proceeds by iterating between regression and factor extraction. These models are used where unobserved heterogeneity is suspected to be multidimensional and time-varying, such as in cross-country growth or firm productivity studies.
Robust and clustered inference
As panel methods spread, so did concern about standard errors. Observations within a unit are typically correlated over time, so the classical assumption of independent errors is wrong, and conventional standard errors are too small. The remedy is clustered standard errors, which allow arbitrary correlation within a cluster (usually the unit) and are computed from the empirical covariance of the moment conditions. The approach was formalized in the 1980s and 1990s, with influential treatments by Halbert White and by Jeffrey Wooldridge, and became routine practice. A parallel literature on the finite-sample behavior of clustered errors—how many clusters are enough, and what to do when there are few—continues to matter, especially in DiD and policy evaluation with a modest number of treated groups.
Machine learning enters the panel
The most recent turn applies machine-learning methods to panel data. The motivation is twofold: high-dimensional controls (many covariates relative to observations) and flexible functional forms. Regularization methods such as the lasso can select among many potential confounders; tree-based and ensemble methods can capture nonlinearities; and double-machine-learning approaches combine flexible prediction with causal estimation. A specific panel problem is that the unit fixed effect must be handled within a machine-learning procedure, which is not automatic. Work from the 2010s and 2020s has developed debiased and orthogonal estimators for panel settings, and the area is growing rapidly, though its practical dominance is still emerging relative to the classical estimators.
The field today
Contemporary panel econometrics is a layered toolkit. Fixed and random effects remain the baseline, taught in every graduate sequence and applied in thousands of papers. Dynamic-panel instrumental-variable methods are standard for persistence questions, with ongoing work on weak instruments and bias correction. Nonlinear panel models continue to be refined, often through correlated random-effects and bias-reduction techniques. Difference-in-differences, now with modern estimators for staggered adoption and heterogeneous effects, is central to policy evaluation. Interactive fixed effects and factor-augmented panels address time-varying unobservables. Clustered inference is universal, and machine-learning methods are being integrated into the classical framework.
Open questions remain: how to do credible inference with few treated clusters; how to combine flexible machine learning with the identifying assumptions panel methods rely on; how to handle nonrandom attrition and missing data; and how to extend these tools to settings with many units and many periods, where the asymptotics of short panels no longer apply. The field's history is one of successive refinements of a single powerful idea—compare units with themselves—and of learning, often painfully, exactly when that comparison is trustworthy.
References
- Panel data (introduction, Analysis, Dynamic panel data), Wikipedia
- Panel analysis, Wikipedia
- Difference in differences, Wikipedia
The frameworks of Panel Data Methods
Each framework is a way of working the field has used: when it held, and the question it answers.
- 1940-1975
Random effects model
How should repeated observations be split into unit and transitory parts?
- 1968-Present
Fixed Effects and Random Effects Estimation
Should unmeasured unit differences be treated as fixed or random?
- 1980-Present
Nonlinear Panel Models with Unobserved Heterogeneity
How can unobserved unit effects be removed in nonlinear models?
- 1981-Present
Dynamic Panel Instrumental-Variable Estimation
How can a lagged outcome be used without inducing bias?
- 1984-Present
Clustered standard errors
How should standard errors account for correlation within units?
- 1985-Present
Difference in differences
What is the effect of a policy when trends would have been parallel?
- 2000-Present
Interactive Fixed Effects and Factor-Augmented Panels
How can unobserved influences that change over time be modeled?
- 2015-Present
Machine Learning and Regularized Panel Estimation
How can flexible models be estimated without overfitting in panels?
How the ideas connect
Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.
Fixed Effects and Random Effects Estimation grew out of Random effects model
The error-components model of the 1940s–50s supplied the between/within decomposition that the fixed- and random-effects estimators formalized in the late 1960s. The within transformation and the random-effects GLS estimator are direct operationalizations of that decomposition.
Carried by Pietro Balestra; Marc Nerlove.
Dynamic Panel Instrumental-Variable Estimation grew out of Fixed Effects and Random Effects Estimation
Nickell's 1981 analysis showed that the within (fixed-effects) transformation induces bias when a lagged dependent variable is included. The instrumental-variable estimators of the 1980s–90s were built specifically to correct that bias in dynamic fixed-effects panels.
Carried by Stephen Nickell; Manuel Arellano; Stephen Bond.
Random Effects and Correlated Random Effects grew out of Fixed Effects and Random Effects Estimation
The fixed-versus-random debate and Hausman's 1978 specification test made the correlation between unit effects and regressors the central issue. Mundlak's and Chamberlain's correlated random-effects approach grew out of that debate by modeling the unit effect as a function of the regressors' averages.
Carried by Jerry A. Hausman; Yair Mundlak; Gary Chamberlain.
Nonlinear Panel Models with Unobserved Heterogeneity grew out of Fixed Effects and Random Effects Estimation
The fixed-effects logit of Chamberlain (1980) extended the within/fixed-effects logic to nonlinear models by conditioning on sufficient statistics to eliminate the unit effect. The broader nonlinear-panel literature grew out of that extension and the incidental-parameters problem it confronted.
Carried by Gary Chamberlain.
Clustered standard errors grew out of Fixed Effects and Random Effects Estimation
As fixed- and random-effects estimation spread, the within-unit correlation of panel errors made conventional standard errors too small. Clustered standard errors were developed to allow arbitrary serial correlation within the unit, the cluster that panel estimation had made salient.
Difference in differences grew out of Fixed Effects and Random Effects Estimation
The two-way fixed-effects regression used in difference-in-differences is a direct application of the panel fixed-effects estimator to group-by-time data. Ashenfelter and Card's 1985 training study formalized DiD in exactly that regression form.
Carried by Orley Ashenfelter; David Card.
Interactive Fixed Effects and Factor-Augmented Panels grew out of Fixed Effects and Random Effects Estimation
Interactive fixed-effects models generalize the standard additive unit effect by letting unit-specific loadings interact with common time factors. Bai's factor-augmented panel work built directly on the fixed-effects framework it generalizes.
Carried by Jushan Bai.
Nonlinear Panel Models with Unobserved Heterogeneity grew out of Random Effects and Correlated Random Effects
Chamberlain's correlated random-effects approach modeled the unit effect as a function of the regressors' averages. That device was carried into nonlinear panel models, where it offered a practical route around the incidental-parameters problem.
Carried by Gary Chamberlain.
Interactive Fixed Effects and Factor-Augmented Panels influenced by Dynamic Panel Instrumental-Variable Estimation
The instrumental-variable machinery for dynamic panels, using lagged values as instruments, informed the moment-based estimation of factor-augmented panel models. The interactive fixed-effects literature drew on that IV tradition when constructing estimators.
Carried by Jushan Bai.
Beyond the field
Where Panel Data Methods's ideas came from in other fields, and what they led to there: the strongest 12 of 23 links.
Difference in differences grew out of Panel data in Microeconometrics
Difference-in-differences estimation is built directly on panel/longitudinal data structures, using repeated observations of the same units to difference out fixed effects. The panel data literature supplied the fixed-effects and within-estimator machinery that DiD exploits.
Carried by Ashenfelter and Card (1985).
Interactive Fixed Effects and Factor-Augmented Panels grew out of Panel data in Microeconometrics
Interactive fixed effects and factor-augmented panels are a direct generalization of the panel data model, replacing additive unit and time effects with multiplicative factor structures. The panel framework's fixed-effects setup is the base case the interactive model extends.
Machine Learning and Regularized Panel Estimation influenced by Panel data in Microeconometrics
Regularized panel estimation applies machine-learning shrinkage and selection methods to the panel data regression framework, using its fixed-effects and within-transformation structure. The panel model is the object being estimated with these new tools.
Machine Learning and Regularized Panel Estimation influenced by Regularization and High-Dimensional Methods in ML Econometrics
Regularized panel estimation takes the lasso, ridge and high-dimensional selection techniques from the regularization literature and applies them to panel models with many covariates or many units. The shrinkage machinery is the core method transferred.
Carried by Robert Tibshirani.
Random effects model influenced Frailty and Multilevel Survival Models in Survival Analysis
The random effects model gave frailty models their form: an unobserved random term shared within a group, carried over from panel-data variance-components models into survival hazards.
Random effects model influenced by Randomized Block and Factorial Design in Experimental Design
Randomized block and factorial designs introduced the idea of partitioning variation among random blocks, which the random effects model carried over to panel units treated as random draws. The variance-components logic of blocked experiments was adapted to longitudinal data.
Carried by Ronald A. Fisher.
Random effects model influenced by Analysis of Variance and Experimental Design in Regression Analysis
The random effects model for panel data grew out of the variance-components analysis of designed experiments, treating individual effects as random draws in an ANOVA-style decomposition. Fisher's analysis of variance and experimental design supplied the variance-components framework that early panel random-effects estimators adapted.
Carried by Ronald A. Fisher.
Random effects model influenced Microeconometrics in Econometrics
The random effects model, developed in statistics for variance components, was taken up by microeconometrics as a standard estimator for panel data with individual-specific effects. It became one of the core tools of the panel data literature.
Fixed Effects and Random Effects Estimation influenced Natural-Experiment and Quasi-Experimental Estimation in Labor Supply Theory
Fixed-effects panel estimators, formalized by Mundlak, Wallace and Hussain, and Swamy in the late 1960s and 1970s, became a core tool of the credibility revolution's quasi-experimental work, where within-unit variation is used to control for unobserved heterogeneity. Difference-in-differences and panel designs in labor economics rest directly on these estimators.
Carried by Yair Mundlak; Pietro Balestra; Marc Nerlove.
Fixed Effects and Random Effects Estimation influenced Estimating Equations and Semiparametric Inference in Statistical Inference
Estimating equations for panel data, including generalised estimating equations for correlated observations, drew on fixed- and random-effects estimation in econometrics. The clustered-data estimating-equation literature took up that panel framework.
Fixed Effects and Random Effects Estimation influenced by Ordinary Least Squares Regression in Microeconometrics
Fixed effects estimation is implemented as least squares on demeaned or dummy-variable data, so ordinary least squares is the computational engine the fixed effects estimator takes up. The within transformation is an application of OLS to transformed panel data.
Random Effects and Correlated Random Effects influenced by Ordinary Least Squares Regression in Microeconometrics
The correlated random effects approach (Mundlak) is built by augmenting a random effects regression with group means, an application of least squares to a transformed specification. OLS supplies the estimation machinery the framework takes up.
Carried by Yair Mundlak.

Panel Data Methods is a planet in Nousfarer
Its city has a tower for each of these 9 frameworks. Land, walk in, and climb one concept at a time.
Land on Panel Data MethodsRelated fields
Fields its ideas are linked to
- Microeconometrics 6 links
- Statistical Inference 4 links
- Econometrics 2 links
- Design Theory 1 link
- ML Econometrics 1 link
- Structural Estimation 1 link
- Time Series Econometrics 1 link
- Labor Supply Theory 1 link
- Public Health Program Evaluation 1 link
- Causal Inference 1 link
- Experimental Design 1 link
- Regression Analysis 1 link
More in Econometrics
- Bayesian Econometrics
- Econometrics
- Microeconometrics
- ML Econometrics
- Structural Estimation
- Time Series Econometrics
See where Panel Data Methods sits on the Map of Human Knowledge