Numerical Analysis

Numerical analysis is the study of how to get reliable answers to mathematical problems using arithmetic that a machine—or a person with a table—can actually carry out.

18 frameworks, from 1670 to today · Applied Mathematics · Mathematics

still a working view ended · each leads to its entry below

Overview

Numerical analysis is the study of how to get reliable answers to mathematical problems using arithmetic that a machine—or a person with a table—can actually carry out1. Its questions are practical: how close is an approximation to the truth, how much does rounding distort it, how many operations will it take, and will the answer survive the wobble of finite precision1? What is at stake is nothing less than whether the equations of science and engineering can be turned into numbers you can trust. A bridge design, a weather forecast, and a spacecraft trajectory all rest on methods whose error can be bounded and whose cost can be estimated.

Counting on fingers, tables, and finite differences

Numerical work is as old as commerce. Babylonian clay tablets from roughly 1800 BCE contain tables of squares and reciprocals for calculation, and Chinese, Indian, and Islamic astronomers built trigonometric and astronomical tables over many centuries. The decisive early modern turn came with calculus. Newton and Leibniz gave formulas for derivatives and integrals, but a formula is not a number. Newton's method for solving an equation by successive correction, and the systematic use of finite differences—approximating a function's slope or curvature from values at nearby points—turned calculus into a computational instrument123. By the eighteenth century, Euler, Lagrange, and others were interpolating from tables and integrating differential equations step by step14.

Interpolation, the art of drawing a smooth curve through known points, became a discipline in its own right15. Lagrange's formula (published in the 1790s) writes the interpolating polynomial explicitly; Newton's divided-difference form made it practical to add points one at a time. The recurring warning was already visible: high-degree polynomials through many equally spaced points can oscillate wildly between them, a phenomenon later named for Runge, who demonstrated it in 1901. The lesson—that more data does not always mean a better approximation—shaped approximation theory for a century.

Solving equations and the birth of error analysis

Finding where a function equals zero is the oldest computational problem in analysis. Newton's method converges fast when it works, but it can fail, cycle, or diverge. The nineteenth and early twentieth centuries added robust bracketing methods—the bisection method, which halves an interval known to contain a root, and the secant and false-position methods—that trade speed for guaranteed convergence. Iterative equation solving thus settled into a permanent tension: fast methods with uncertain behavior versus slow methods with proofs.

The deeper problem was rounding16. Hand computation with log tables and slide rules carried errors that could compound invisibly. The modern understanding arrived with the electronic computer1. James Wilkinson, working at the National Physical Laboratory in Britain in the late 1940s and 1950s, analyzed Gaussian elimination—the schoolbook method of solving simultaneous linear equations—and showed that in floating-point arithmetic it does not compute the exact answer to the given problem, but rather the exact answer to a slightly perturbed problem. This idea, backward error analysis, reframed the question from "how wrong is my answer?" to "what problem did I actually solve?"7 If the perturbation is comparable to the uncertainty already in the data, the answer is as good as the inputs allow7. Wilkinson's Rounding Errors in Algebraic Processes (1963) made this the standard way to judge numerical algorithms, and it remains so.

Matrices, stability, and the machinery of computation

Linear algebra became the backbone of the field because so many problems—differential equations, least-squares fitting, optimization—reduce to solving systems of equations or finding eigenvalues8. The 1950s and 1960s produced the core toolkit: Householder reflections and Givens rotations for stable matrix factorization, the QR algorithm for eigenvalues (independently developed by Francis and by Kublanovskaya around 1961), and the singular value decomposition as a diagnostic for conditioning. Conditioning measures how much a problem amplifies small input changes; a well-conditioned problem is one whose answer is not hypersensitive9. A stable algorithm on a well-conditioned problem gives a trustworthy result; either alone is not enough91.

Stability theory grew alongside10. For differential equations, the question is whether a numerical step-by-step scheme reproduces the qualitative behavior of the true solution or blows up spuriously10. The Dahlquist stability theory of the 1950s and 1960s classified linear multistep methods by their stability regions, and the concept of A-stability—methods that stay stable on stiff problems, where some components change far faster than others—became central. Stiff equations, common in chemical kinetics and circuit simulation, forced the development of implicit methods such as the backward differentiation formulas.

Differential equations and the finite element revolution

Solving ordinary and partial differential equations numerically is the field's largest application. Early twentieth-century work by Runge and Kutta produced the Runge–Kutta family of one-step integrators; later, adaptive step-size control let programs choose their own resolution. For partial differential equations, the finite difference method—replacing derivatives by differences on a grid—was standard by the mid-twentieth century, but it struggled with irregular geometry.

The finite element method changed that. Developed in the 1950s and 1960s by engineers (notably at Boeing and in civil engineering circles) and given mathematical foundations by Courant earlier and by mathematicians such as Babuška, Zienkiewicz, and Ciarlet later, it divides a complicated domain into simple pieces—triangles or tetrahedra—and builds a global solution from local ones. Its variational formulation, resting on weak solutions of differential equations, gave it both flexibility and rigorous error estimates. It is now the default method for structural, thermal, and fluid analysis in engineering.

Two further ideas attacked cost. Multigrid methods, developed from the 1960s onward (Brandt's work in the 1970s was pivotal), solve problems on a hierarchy of grids, using coarse grids to fix errors that fine grids correct only slowly; for many elliptic problems they achieve optimal cost, proportional to the number of unknowns. Adaptive mesh refinement concentrates computational effort where the solution changes rapidly, refining locally rather than everywhere11. Both descend from the same insight: resolution should follow need.

Fast transforms and the cost of computation

The fast Fourier transform, published by Cooley and Tukey in 1965 (with earlier roots in Gauss's unpublished work and in Runge's era), reduced the cost of computing a discrete Fourier transform from roughly n² operations to n log n. The effect was transformative: signal processing, spectral methods for differential equations, and later image compression all became feasible at scale. It also crystallized a new question—the computational complexity of numerical algorithms—asking not just whether a method converges but how its work grows with problem size. The distinction between n³ and n²·⁷⁵ for matrix multiplication, or between direct and iterative solvers, became a design criterion in itself.

For large linear systems, direct factorization is often too expensive, and iterative methods take over1. The conjugate gradient method, introduced by Hestenes and Stiefel in 1952 and largely ignored for two decades, solves symmetric positive-definite systems by building up a solution in a sequence of optimal directions. Its generalization to Krylov subspace methods—GMRES, BiCGSTAB, and their descendants—made large sparse systems tractable, and preconditioning (transforming a hard system into an easier one) became an art. These methods are the engine of modern large-scale simulation8.

Optimization, uncertainty, and the software that carries it all

Numerical optimization—finding minima and maxima—grew from the calculus of variations and from practical needs in operations research. The simplex method (Dantzig, 1947) solved linear programs; nonlinear optimization developed gradient-based methods, quasi-Newton updates such as BFGS, and later interior-point methods. Numerical analysis contributed the crucial concerns: convergence rates, conditioning of the Hessian, and the reliability of stopping criteria.

As models grew, so did awareness of their inputs' uncertainty12. Uncertainty quantification treats parameters as random and asks how their variability propagates to outputs, using Monte Carlo sampling, polynomial chaos expansions, and stochastic Galerkin methods. It is now standard in climate modeling, engineering design, and risk assessment.

None of this would reach users without scientific software libraries113. From the early Fortran libraries (EISPACK and LINPACK for linear algebra, later LAPACK) through Netlib and the Numerical Recipes books to today's open-source ecosystems, the field has built shared, tested implementations. This is not a minor convenience: a well-tested library encodes decades of stability analysis, and its availability determines what scientists can actually compute.

The present: machine learning and the old questions

Numerical analysis now underpins machine learning1415. Training a neural network is numerical optimization at scale—stochastic gradient methods, adaptive step sizes, and the conditioning of very high-dimensional problems are all classical concerns in new dress. Automatic differentiation computes gradients exactly by the chain rule, and the numerical analysis of machine learning studies generalization, stability of training, and the floating-point behavior of large models. Meanwhile, the field's oldest themes persist: rounding error, conditioning, convergence, and cost. The questions still open are familiar ones—how to certify that a computed answer is correct, how to handle problems in very high dimensions where classical methods stall, and how to make algorithms reliable on the parallel and heterogeneous hardware that now defines computing. The history of numerical analysis is, in the end, the history of making mathematics answerable to the machine.

References

  1. Numerical analysis (introduction, Applications, Generation and propagation of errors, Round-off, Truncation and discretization error, History, Computing values of functions, Numerical stability and well-posed problems, Direct and iterative methods, Solving equations and systems of equations, Software), Wikipedia
  2. Finite difference, Wikipedia
  3. Newton's method, Wikipedia
  4. Numerical methods for ordinary differential equations (History), Wikipedia
  5. Interpolation, Wikipedia
  6. Round-off error, Wikipedia
  7. Error analysis (mathematics) (Backward error analysis), Wikipedia
  8. Numerical linear algebra, Wikipedia
  9. Condition number, Wikipedia
  10. Numerical stability, Wikipedia
  11. Adaptive mesh refinement, Wikipedia
  12. Uncertainty quantification, Wikipedia
  13. Basic Linear Algebra Subprograms (Background), Wikipedia
  14. Gradient descent, Wikipedia
  15. Stochastic gradient descent, Wikipedia

The frameworks of Numerical Analysis

Each framework is a way of working the field has used: when it held, and the question it answers.

  1. 1670-Present

    Root-finding algorithm

    How do we find where a function equals zero?

    Fly to its tower in Nousfarer

  2. 1750-Present
  3. 1768-Present

    Numerical Solution of Differential Equations

    How do we compute solutions of differential equations step by step?

    Fly to its tower in Nousfarer

  4. 1900-Present

    Runge–Kutta methods

    Which one-step formulas integrate equations accurately?

    Fly to its tower in Nousfarer

  5. 1920-Present

    Finite difference method

    How do we replace derivatives by differences on a grid?

    Fly to its tower in Nousfarer

  6. 1947-Present

    Numerical Optimization

    How do we find the best value of a function?

    Fly to its tower in Nousfarer

  7. 1948-Present

    Backward Error Analysis

    What problem did my computation actually solve?

    Fly to its tower in Nousfarer

  8. 1950-Present

    Numerical Linear Algebra

    How do we solve matrix equations reliably?

    Fly to its tower in Nousfarer

  9. 1950-Present

    Stability Theory for Differential Equations

    When does a numerical scheme stay stable or blow up?

    Fly to its tower in Nousfarer

  10. 1952-Present

    Krylov Subspace and Iterative Linear Solvers

    How do we solve huge linear systems by iteration?

    Fly to its tower in Nousfarer

  11. 1955-Present

    Finite element method

    How do we solve equations on complicated shapes?

    Fly to its tower in Nousfarer

  12. 1965-Present
  13. 1965-Present

    Fast Fourier transform

    How do we compute Fourier transforms quickly?

    Fly to its tower in Nousfarer

  14. 1965-Present

    Multigrid method

    How do we solve grid problems at optimal cost?

    Fly to its tower in Nousfarer

  15. 1970-Present

    Scientific Software Libraries

    How do we package numerical methods for reliable reuse?

    Fly to its tower in Nousfarer

  16. 1975-Present

    Adaptive Mesh Refinement

    Where should we concentrate computational effort?

    Fly to its tower in Nousfarer

  17. 1990-Present

    Uncertainty Quantification

    How does input uncertainty affect our answers?

    Fly to its tower in Nousfarer

  18. 2010-Present

    Numerical Analysis of Machine Learning

    How do numerical methods behave in machine learning?

    Fly to its tower in Nousfarer

How the ideas connect

Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.

Root-finding algorithmInterpolation and Approximation TheoryNumerical Solution of Differential Eq…Runge–Kutta methodsFinite difference methodNumerical OptimizationBackward Error AnalysisNumerical Linear AlgebraStability Theory for Differential Equ…Krylov Subspace and Iterative Linear …Finite element methodComputational Complexity of Numerical…Fast Fourier transformMultigrid methodScientific Software LibrariesAdaptive Mesh RefinementUncertainty QuantificationNumerical Analysis of Machine Learning

Beyond the field

Where Numerical Analysis's ideas came from in other fields, and what they led to there: the strongest 12 of 87 links.

A city in Nousfarer: a red-carpeted stair climbing terraces to a crowned tower on an acropolis, more towers round it, forest and snowy mountains beyond.
A city in Nousfarer, the game played in the browser.

Numerical Analysis is a planet in Nousfarer

Its city has a tower for each of these 18 frameworks. Land, walk in, and climb one concept at a time.

Land on Numerical Analysis

Related fields

Fields its ideas are linked to

More in Applied Mathematics

See where Numerical Analysis sits on the Map of Human Knowledge