- Atlas
- Mathematics
- Numerical Analysis
Numerical Analysis
Numerical analysis is the study of how to get reliable answers to mathematical problems using arithmetic that a machine—or a person with a table—can actually carry out.
18 frameworks, from 1670 to today · Applied Mathematics · Mathematics
- Root-finding algorithm 1670-Present
- Interpolation and Approximation Theory 1750-Present
- Numerical Solution of Differential Equations 1768-Present
- Runge–Kutta methods 1900-Present
- Finite difference method 1920-Present
- Numerical Optimization 1947-Present
- Backward Error Analysis 1948-Present
- Numerical Linear Algebra 1950-Present
- Stability Theory for Differential Equations 1950-Present
- Krylov Subspace and Iterative Linear Solvers 1952-Present
- Finite element method 1955-Present
- Computational Complexity of Numerical Algorithms 1965-Present
- Fast Fourier transform 1965-Present
- Multigrid method 1965-Present
- Scientific Software Libraries 1970-Present
- Adaptive Mesh Refinement 1975-Present
- Uncertainty Quantification 1990-Present
- Numerical Analysis of Machine Learning 2010-Present
Overview
Numerical analysis is the study of how to get reliable answers to mathematical problems using arithmetic that a machine—or a person with a table—can actually carry out1. Its questions are practical: how close is an approximation to the truth, how much does rounding distort it, how many operations will it take, and will the answer survive the wobble of finite precision1? What is at stake is nothing less than whether the equations of science and engineering can be turned into numbers you can trust. A bridge design, a weather forecast, and a spacecraft trajectory all rest on methods whose error can be bounded and whose cost can be estimated.
Counting on fingers, tables, and finite differences
Numerical work is as old as commerce. Babylonian clay tablets from roughly 1800 BCE contain tables of squares and reciprocals for calculation, and Chinese, Indian, and Islamic astronomers built trigonometric and astronomical tables over many centuries. The decisive early modern turn came with calculus. Newton and Leibniz gave formulas for derivatives and integrals, but a formula is not a number. Newton's method for solving an equation by successive correction, and the systematic use of finite differences—approximating a function's slope or curvature from values at nearby points—turned calculus into a computational instrument123. By the eighteenth century, Euler, Lagrange, and others were interpolating from tables and integrating differential equations step by step14.
Interpolation, the art of drawing a smooth curve through known points, became a discipline in its own right15. Lagrange's formula (published in the 1790s) writes the interpolating polynomial explicitly; Newton's divided-difference form made it practical to add points one at a time. The recurring warning was already visible: high-degree polynomials through many equally spaced points can oscillate wildly between them, a phenomenon later named for Runge, who demonstrated it in 1901. The lesson—that more data does not always mean a better approximation—shaped approximation theory for a century.
Solving equations and the birth of error analysis
Finding where a function equals zero is the oldest computational problem in analysis. Newton's method converges fast when it works, but it can fail, cycle, or diverge. The nineteenth and early twentieth centuries added robust bracketing methods—the bisection method, which halves an interval known to contain a root, and the secant and false-position methods—that trade speed for guaranteed convergence. Iterative equation solving thus settled into a permanent tension: fast methods with uncertain behavior versus slow methods with proofs.
The deeper problem was rounding16. Hand computation with log tables and slide rules carried errors that could compound invisibly. The modern understanding arrived with the electronic computer1. James Wilkinson, working at the National Physical Laboratory in Britain in the late 1940s and 1950s, analyzed Gaussian elimination—the schoolbook method of solving simultaneous linear equations—and showed that in floating-point arithmetic it does not compute the exact answer to the given problem, but rather the exact answer to a slightly perturbed problem. This idea, backward error analysis, reframed the question from "how wrong is my answer?" to "what problem did I actually solve?"7 If the perturbation is comparable to the uncertainty already in the data, the answer is as good as the inputs allow7. Wilkinson's Rounding Errors in Algebraic Processes (1963) made this the standard way to judge numerical algorithms, and it remains so.
Matrices, stability, and the machinery of computation
Linear algebra became the backbone of the field because so many problems—differential equations, least-squares fitting, optimization—reduce to solving systems of equations or finding eigenvalues8. The 1950s and 1960s produced the core toolkit: Householder reflections and Givens rotations for stable matrix factorization, the QR algorithm for eigenvalues (independently developed by Francis and by Kublanovskaya around 1961), and the singular value decomposition as a diagnostic for conditioning. Conditioning measures how much a problem amplifies small input changes; a well-conditioned problem is one whose answer is not hypersensitive9. A stable algorithm on a well-conditioned problem gives a trustworthy result; either alone is not enough91.
Stability theory grew alongside10. For differential equations, the question is whether a numerical step-by-step scheme reproduces the qualitative behavior of the true solution or blows up spuriously10. The Dahlquist stability theory of the 1950s and 1960s classified linear multistep methods by their stability regions, and the concept of A-stability—methods that stay stable on stiff problems, where some components change far faster than others—became central. Stiff equations, common in chemical kinetics and circuit simulation, forced the development of implicit methods such as the backward differentiation formulas.
Differential equations and the finite element revolution
Solving ordinary and partial differential equations numerically is the field's largest application. Early twentieth-century work by Runge and Kutta produced the Runge–Kutta family of one-step integrators; later, adaptive step-size control let programs choose their own resolution. For partial differential equations, the finite difference method—replacing derivatives by differences on a grid—was standard by the mid-twentieth century, but it struggled with irregular geometry.
The finite element method changed that. Developed in the 1950s and 1960s by engineers (notably at Boeing and in civil engineering circles) and given mathematical foundations by Courant earlier and by mathematicians such as Babuška, Zienkiewicz, and Ciarlet later, it divides a complicated domain into simple pieces—triangles or tetrahedra—and builds a global solution from local ones. Its variational formulation, resting on weak solutions of differential equations, gave it both flexibility and rigorous error estimates. It is now the default method for structural, thermal, and fluid analysis in engineering.
Two further ideas attacked cost. Multigrid methods, developed from the 1960s onward (Brandt's work in the 1970s was pivotal), solve problems on a hierarchy of grids, using coarse grids to fix errors that fine grids correct only slowly; for many elliptic problems they achieve optimal cost, proportional to the number of unknowns. Adaptive mesh refinement concentrates computational effort where the solution changes rapidly, refining locally rather than everywhere11. Both descend from the same insight: resolution should follow need.
Fast transforms and the cost of computation
The fast Fourier transform, published by Cooley and Tukey in 1965 (with earlier roots in Gauss's unpublished work and in Runge's era), reduced the cost of computing a discrete Fourier transform from roughly n² operations to n log n. The effect was transformative: signal processing, spectral methods for differential equations, and later image compression all became feasible at scale. It also crystallized a new question—the computational complexity of numerical algorithms—asking not just whether a method converges but how its work grows with problem size. The distinction between n³ and n²·⁷⁵ for matrix multiplication, or between direct and iterative solvers, became a design criterion in itself.
For large linear systems, direct factorization is often too expensive, and iterative methods take over1. The conjugate gradient method, introduced by Hestenes and Stiefel in 1952 and largely ignored for two decades, solves symmetric positive-definite systems by building up a solution in a sequence of optimal directions. Its generalization to Krylov subspace methods—GMRES, BiCGSTAB, and their descendants—made large sparse systems tractable, and preconditioning (transforming a hard system into an easier one) became an art. These methods are the engine of modern large-scale simulation8.
Optimization, uncertainty, and the software that carries it all
Numerical optimization—finding minima and maxima—grew from the calculus of variations and from practical needs in operations research. The simplex method (Dantzig, 1947) solved linear programs; nonlinear optimization developed gradient-based methods, quasi-Newton updates such as BFGS, and later interior-point methods. Numerical analysis contributed the crucial concerns: convergence rates, conditioning of the Hessian, and the reliability of stopping criteria.
As models grew, so did awareness of their inputs' uncertainty12. Uncertainty quantification treats parameters as random and asks how their variability propagates to outputs, using Monte Carlo sampling, polynomial chaos expansions, and stochastic Galerkin methods. It is now standard in climate modeling, engineering design, and risk assessment.
None of this would reach users without scientific software libraries113. From the early Fortran libraries (EISPACK and LINPACK for linear algebra, later LAPACK) through Netlib and the Numerical Recipes books to today's open-source ecosystems, the field has built shared, tested implementations. This is not a minor convenience: a well-tested library encodes decades of stability analysis, and its availability determines what scientists can actually compute.
The present: machine learning and the old questions
Numerical analysis now underpins machine learning1415. Training a neural network is numerical optimization at scale—stochastic gradient methods, adaptive step sizes, and the conditioning of very high-dimensional problems are all classical concerns in new dress. Automatic differentiation computes gradients exactly by the chain rule, and the numerical analysis of machine learning studies generalization, stability of training, and the floating-point behavior of large models. Meanwhile, the field's oldest themes persist: rounding error, conditioning, convergence, and cost. The questions still open are familiar ones—how to certify that a computed answer is correct, how to handle problems in very high dimensions where classical methods stall, and how to make algorithms reliable on the parallel and heterogeneous hardware that now defines computing. The history of numerical analysis is, in the end, the history of making mathematics answerable to the machine.
References
- Numerical analysis (introduction, Applications, Generation and propagation of errors, Round-off, Truncation and discretization error, History, Computing values of functions, Numerical stability and well-posed problems, Direct and iterative methods, Solving equations and systems of equations, Software), Wikipedia
- Finite difference, Wikipedia
- Newton's method, Wikipedia
- Numerical methods for ordinary differential equations (History), Wikipedia
- Interpolation, Wikipedia
- Round-off error, Wikipedia
- Error analysis (mathematics) (Backward error analysis), Wikipedia
- Numerical linear algebra, Wikipedia
- Condition number, Wikipedia
- Numerical stability, Wikipedia
- Adaptive mesh refinement, Wikipedia
- Uncertainty quantification, Wikipedia
- Basic Linear Algebra Subprograms (Background), Wikipedia
- Gradient descent, Wikipedia
- Stochastic gradient descent, Wikipedia
The frameworks of Numerical Analysis
Each framework is a way of working the field has used: when it held, and the question it answers.
- 1670-Present
- 1750-Present
Interpolation and Approximation Theory
How do we draw a curve through known points?
- 1768-Present
Numerical Solution of Differential Equations
How do we compute solutions of differential equations step by step?
- 1900-Present
Runge–Kutta methods
Which one-step formulas integrate equations accurately?
- 1920-Present
Finite difference method
How do we replace derivatives by differences on a grid?
- 1947-Present
- 1948-Present
- 1950-Present
- 1950-Present
Stability Theory for Differential Equations
When does a numerical scheme stay stable or blow up?
- 1952-Present
Krylov Subspace and Iterative Linear Solvers
How do we solve huge linear systems by iteration?
- 1955-Present
- 1965-Present
Computational Complexity of Numerical Algorithms
How does the work of an algorithm grow with size?
- 1965-Present
- 1965-Present
- 1970-Present
Scientific Software Libraries
How do we package numerical methods for reliable reuse?
- 1975-Present
Adaptive Mesh Refinement
Where should we concentrate computational effort?
- 1990-Present
Uncertainty Quantification
How does input uncertainty affect our answers?
- 2010-Present
Numerical Analysis of Machine Learning
How do numerical methods behave in machine learning?
How the ideas connect
Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.
Runge–Kutta methods grew out of Numerical Solution of Differential Equations
Runge and Kutta built their one-step integrators directly as refinements of earlier step-by-step methods for ordinary differential equations.
Carried by Carl Runge; Wilhelm Kutta.
Finite difference method grew out of Numerical Solution of Differential Equations
The finite difference method for partial differential equations grew out of the same step-by-step discretization tradition used for ordinary differential equations.
Stability Theory for Differential Equations influenced by Runge–Kutta methods
Dahlquist's stability theory for linear multistep methods was framed against the behavior of one-step Runge–Kutta schemes, classifying when step-by-step integrators remain stable.
Carried by Germund Dahlquist.
Finite element method influenced by Finite difference method
The finite element method was developed partly because finite differences struggled with irregular geometry; it replaced grid-based differences with local piecewise polynomial approximations on unstructured meshes.
Carried by Richard Courant; Ivo Babuška.
Multigrid method grew out of Finite difference method
Multigrid methods were built on finite difference discretizations, solving the same grid equations on a hierarchy of coarser and finer grids.
Carried by Achi Brandt.
Adaptive Mesh Refinement grew out of Finite difference method
Adaptive mesh refinement extended finite difference grids by locally refining cells where the solution changes rapidly.
Numerical Analysis of Machine Learning influenced by Numerical Optimization
Gradient-based and stochastic optimization methods from numerical optimization became the core algorithms for training machine learning models.
Numerical Analysis of Machine Learning influenced by Numerical Linear Algebra
The numerical analysis of machine learning applies matrix conditioning, stability, and factorization results from numerical linear algebra to the training and analysis of learning algorithms.
Multigrid method influenced by Finite element method
Multigrid techniques were extended from finite difference grids to finite element discretizations, using hierarchies of meshes to accelerate convergence.
Carried by Achi Brandt.
Uncertainty Quantification influenced by Numerical Optimization
Uncertainty quantification uses numerical optimization, for example in calibration and in finding worst-case parameter values, as part of propagating input variability.
Numerical Analysis of Machine Learning influenced by Scientific Software Libraries
Scientific software libraries provided the numerical building blocks, such as linear algebra and optimization routines, on which machine learning frameworks were built.
Beyond the field
Where Numerical Analysis's ideas came from in other fields, and what they led to there: the strongest 12 of 87 links.
Interpolation and Approximation Theory led to Numerical analysis in Applied Mathematics
Interpolation and approximation theory, from Newton and Lagrange through Chebyshev, is a direct parent of numerical analysis, supplying its core techniques for representing and approximating functions.
Carried by Joseph-Louis Lagrange; Pafnuty Chebyshev.
Numerical Solution of Differential Equations influenced Step-by-step time integration in Structural Dynamics
Step-by-step time integration is a direct application of numerical methods for ordinary differential equations; the Newmark and similar schemes are specific integrators for the second-order structural equations built on that tradition.
Carried by Nathan Newmark.
Numerical Solution of Differential Equations led to Numerical analysis in Applied Mathematics
Numerical solution of differential equations, from Euler and Runge through finite-difference schemes, grew into the central branch of numerical analysis as computing made large-scale computation feasible.
Carried by Leonhard Euler; Carl Runge.
Runge–Kutta methods grew out of Numerical methods for ordinary differential equations in Ordinary Differential Equations
Runge–Kutta methods are a direct refinement of the earlier one-step numerical methods for ordinary differential equations, generalising Euler's method and the Taylor-series schemes of Runge and Heun. Kutta's 1901 paper built explicitly on Runge's work to construct higher-order schemes.
Carried by Carl Runge; Wilhelm Kutta.
Numerical Linear Algebra led to Randomized numerical linear algebra in Randomized Algorithms
The field emerged from numerical linear algebra as practiced in numerical analysis, adding randomization to standard matrix algorithms.
Numerical Linear Algebra led to Sparse and Iterative Methods in Linear Algebra
Sparse and iterative methods are a branch of numerical linear algebra, arising from the need to solve large sparse systems where direct factorization is impractical.
Carried by Magnus Hestenes; Eduard Stiefel.
Fast Fourier transform grew out of Fourier analysis in Applied Mathematics
The Cooley-Tukey algorithm reduced the discrete Fourier transform to O(N log N), building directly on Fourier's decomposition of functions into sinusoidal components. The FFT is a fast algorithm for computing exactly the transform Fourier analysis defines.
Carried by James Cooley; John Tukey; An algorithm for the machine calculation of complex Fourier series (1965).
Fast Fourier transform grew out of Fourier analysis in Harmonic Analysis
The fast Fourier transform is a computational algorithm for the Fourier transform studied in harmonic analysis, exploiting its symmetries and periodicity to reduce complexity.
Carried by James Cooley; John Tukey.
Fast Fourier transform grew out of Fourier analysis in Signals And Systems
The FFT computes the discrete Fourier transform that signals-and-systems Fourier analysis defines; the 1965 Cooley-Tukey paper turned that transform into a practical engineering tool.
Carried by James Cooley; John Tukey.
Fast Fourier transform led to FFT-based spectral analysis in Signal Processing
FFT-based spectral analysis exists only because the Cooley-Tukey algorithm made Fourier spectral estimation computationally cheap; it is the direct application of the FFT to spectrum estimation.
Carried by James Cooley; John Tukey.
Scientific Software Libraries grew out of Numerical analysis in Applied Mathematics
Scientific software libraries package numerical analysis algorithms — linear algebra, quadrature, root finding, ODE solvers — as reusable code; they are the direct software embodiment of the numerical analysis tradition.
Root-finding algorithm influenced by Theory of equations in Algebra
Newton's root-finding work grew out of the early modern theory of equations: his Newton's method and his classification of cubics were framed within the algebraic equation-solving tradition of Descartes, Wallis and the 17th-century algebraists. The theory of equations supplied the objects (polynomial equations) and the question of locating and approximating roots that numerical root-finding took up.
Carried by Isaac Newton; John Wallis.

Numerical Analysis is a planet in Nousfarer
Its city has a tower for each of these 18 frameworks. Land, walk in, and climb one concept at a time.
Land on Numerical AnalysisRelated fields
Fields its ideas are linked to
- Applied Mathematics 15 links
- Ordinary Differential Equations 10 links
- Partial Differential Equations 9 links
- Linear Algebra 6 links
- Computational Physics 4 links
- Statistical Computing 4 links
- Inverse Problems 3 links
- Randomized Algorithms 2 links
- Asymptotic Analysis 2 links
- Mathematical Optimization 2 links
- Structural Dynamics 2 links
- Systems Simulation 2 links
More in Applied Mathematics
- Applied Mathematics
- Asymptotic Analysis
- Continuum Mechanics
- Control Theory
- Financial Mathematics
- Game Theory
- Inverse Problems
- Mathematical Biology
- Mathematical Optimization
See where Numerical Analysis sits on the Map of Human Knowledge