Loading field map
Numerical analysis is the study of algorithms that use numerical approximation for problems of continuous mathematics. Its subject matter is the set of techniques by which mathematical problems involving real or complex numbers—problems that in principle require infinitely many operations to solve exactly—are converted into finite sequences of arithmetic operations that yield approximate answers with controlled error. The field sits at the intersection of mathematics and computer science: it draws on mathematical analysis to understand why methods work and how accurate they are, and it draws on computer science to design and implement efficient algorithms.
The central questions of numerical analysis are deceptively simple. Given a mathematical problem—solve an equation, compute an integral, approximate a derivative, solve a differential equation—one asks: Is there a practical algorithm that produces an answer? How accurate is that answer, and can the error be bounded or estimated? How much computational work is required, and does the algorithm remain stable as the problem size grows? The field's enduring challenge is that exact answers are almost never available. Even the simplest operations, such as evaluating a square root or a trigonometric function, require infinite processes to compute exactly; numerical analysis provides the finite approximations that make computation possible.
Numerical analysis is distinguished from other branches of applied mathematics by its focus on the process of computation rather than on the mathematical objects themselves. A mathematician studying differential equations might ask whether a solution exists and whether it is unique; a numerical analyst asks how to compute that solution to a specified tolerance using a finite number of arithmetic operations. This focus gives the field its characteristic concerns: error, stability, and efficiency.
Error in numerical computation comes from several sources. Discretization error arises when a continuous problem is replaced by a discrete one—for example, when an integral is approximated by a finite sum, or a derivative by a difference quotient. Rounding error arises because computers represent real numbers with finite precision, typically using floating-point arithmetic. A central insight of the field is that these two sources of error interact in ways that must be carefully managed. A method that reduces discretization error by using a very fine grid may amplify rounding error to the point where the computed result is meaningless. The study of this interaction leads to the concept of numerical stability: an algorithm is stable if small perturbations in the input—including rounding errors introduced during computation—produce only small perturbations in the output.
The field is organized less around a single grand theory than around a set of core problems, each with its own family of methods. These problems include solving linear systems, finding roots of equations, approximating functions, computing integrals and derivatives, and solving differential equations. Each area has developed its own techniques, but they share common mathematical tools—particularly Taylor's theorem, which underlies most error analysis, and linear algebra, which underlies most computational methods.
The practice of numerical approximation is as old as mathematics itself. Ancient Babylonian tablets contain methods for approximating square roots; Greek mathematicians developed methods for approximating π; medieval Islamic and Indian mathematicians refined techniques for interpolation and root-finding. These early efforts were not, however, numerical analysis in the modern sense. They were isolated techniques developed for specific problems, without a general theory of error or a systematic approach to algorithm design.
The modern field began to take shape in the seventeenth and eighteenth centuries, as calculus provided a unified framework for continuous mathematics. Isaac Newton's method for solving equations, published in the late seventeenth century, remains a cornerstone of numerical computation. Leonhard Euler and other eighteenth-century mathematicians developed methods for approximating integrals and solving differential equations. These methods were motivated by practical needs in astronomy, navigation, and ballistics, but they were still typically used by hand, with the mathematician performing the arithmetic.
The nineteenth century brought a crucial development: the systematic study of error. Mathematicians such as Augustin-Louis Cauchy and Carl Friedrich Gauss began to analyze rigorously how approximations converge to exact answers and how errors propagate through computations. This period also saw the development of the method of least squares by Gauss and Adrien-Marie Legendre, which remains fundamental to data fitting and many other numerical problems.
The invention of electronic computers in the mid-twentieth century transformed the field. Problems that had been computationally intractable—solving large systems of linear equations, simulating fluid flow, computing the trajectories of spacecraft—became feasible. Numerical analysis emerged as a distinct discipline, with its own journals, departments, and professional societies. The field's development since then has been driven by the interplay between mathematical theory and computational practice: new applications demand new methods, and new methods demand new mathematical analysis to understand their behavior.
Solving systems of linear equations is the most fundamental problem in numerical analysis, both because it arises directly in countless applications and because it underlies methods for nearly every other problem in the field. The classical method is Gaussian elimination, which transforms a system into triangular form and then solves by back-substitution. The method is ancient—it appears in Chinese mathematical texts from the first centuries CE—but its modern analysis dates to the mid-twentieth century, when the concept of pivoting was developed to control rounding error.
The key insight of modern linear algebra is that the difficulty of solving a linear system depends on the condition number of the coefficient matrix, which measures how sensitive the solution is to perturbations in the input. A well-conditioned system can be solved accurately; an ill-conditioned system cannot, no matter how good the algorithm. This distinction between the problem's inherent difficulty and the algorithm's quality is fundamental to numerical analysis.
For large systems, direct methods like Gaussian elimination become prohibitively expensive, and iterative methods are used instead. These methods generate a sequence of approximate solutions that converge to the exact answer. The Jacobi and Gauss–Seidel methods, developed in the nineteenth century, are simple but often slow; the conjugate gradient method, developed in the 1950s, is far more efficient for symmetric positive-definite systems. The choice between direct and iterative methods depends on the size and structure of the problem, and modern practice often combines them.
Finding the roots of an equation f(x) = 0 is one of the oldest numerical problems. The bisection method, which repeatedly halves an interval known to contain a root, is simple and reliable but slow. Newton's method, which uses the derivative to construct a linear approximation at each step, converges much faster when it converges at all—but it can fail to converge if the initial guess is poor or the function is badly behaved. The secant method, which approximates the derivative using finite differences, offers a compromise between speed and reliability.
The modern theory of root-finding is built on the concept of convergence order: a method has order p if the error at each step is roughly proportional to the p-th power of the previous error. Newton's method has order 2 (quadratic convergence) under favorable conditions; bisection has order 1 (linear convergence). This theory allows numerical analysts to compare methods rigorously and to understand the trade-offs between speed and robustness.
Given a set of data points, interpolation constructs a function that passes exactly through them; approximation constructs a function that comes close to them in some sense. The simplest interpolation methods use polynomials, and the theory of polynomial interpolation is classical. The Lagrange form and the Newton form provide explicit constructions, and the error can be bounded using derivatives of the underlying function.
A central result is that polynomial interpolation at equally spaced points can fail spectacularly: the interpolating polynomial may oscillate wildly between data points, a phenomenon known as Runge's phenomenon. This led to the development of better approaches: interpolation at specially chosen points (such as Chebyshev nodes) that minimize the maximum error, and piecewise polynomial methods (splines) that use low-degree polynomials on each interval between data points. Splines, developed in the mid-twentieth century, have become the standard tool for many applications because they combine smoothness with stability.
When data are noisy, exact interpolation is undesirable because it fits the noise as well as the signal. The method of least squares, developed by Gauss and Legendre, finds the function that minimizes the sum of squared errors. This approach is fundamental to statistics and data science, and its numerical implementation requires solving the normal equations or, more stably, using orthogonal decomposition methods.
Numerical integration, also called quadrature, approximates definite integrals by finite sums. The simplest methods—the rectangle rule, the trapezoidal rule, Simpson's rule—approximate the integrand by polynomials of increasing degree. The error in these methods depends on the smoothness of the integrand and the spacing of the evaluation points.
The classical theory of quadrature was developed in the nineteenth century, culminating in the Gauss quadrature formulas, which choose evaluation points optimally to achieve the highest possible accuracy for a given number of function evaluations. For integrands that are smooth but expensive to evaluate, Gauss quadrature remains the method of choice. For integrands with singularities or other irregularities, adaptive methods that concentrate evaluation points where the integrand varies rapidly are often more effective.
Numerical differentiation is a more delicate problem than integration. The obvious approach—using difference quotients—suffers from catastrophic cancellation: as the step size decreases, the discretization error decreases but the rounding error increases, so there is an optimal step size that balances the two. This phenomenon illustrates a general principle of numerical analysis: making a discretization finer does not always improve accuracy, because rounding error grows as the step size shrinks.
Solving differential equations numerically is the largest and most active area of numerical analysis, driven by applications in physics, engineering, biology, and finance. The methods fall into two broad families. Finite difference methods replace derivatives by difference quotients on a grid of points. Finite element methods divide the domain into small pieces and approximate the solution by piecewise polynomials on each piece. A third family, spectral methods, represents the solution as a sum of global basis functions such as trigonometric polynomials.
For ordinary differential equations, the classical methods are the Euler method and its refinements: the Runge–Kutta family, developed in the late nineteenth and early twentieth centuries, and the linear multistep methods, developed in the mid-twentieth century. The theory of these methods addresses two questions: convergence (does the approximate solution approach the exact solution as the step size decreases?) and stability (do small perturbations remain small?). A subtlety is that some methods that converge for well-behaved problems become unstable for stiff problems—those with solutions that vary on very different time scales. The development of methods for stiff problems, particularly the implicit methods, was a major achievement of the mid-twentieth century.
For partial differential equations, the situation is more complex because the solution depends on several variables and the geometry of the domain matters. The finite element method, developed in the 1950s and 1960s for structural engineering, has become the dominant approach for elliptic and parabolic problems because it handles complex geometries naturally. Finite difference methods remain important for hyperbolic problems and for problems on simple domains. The analysis of these methods involves subtle questions of stability and convergence that are the subject of active research.
The methods of numerical analysis are not organized into rival schools in the way that, say, competing theories in physics might be. Rather, the field is characterized by a pragmatic pluralism: different methods are suited to different problems, and practitioners choose among them based on the problem's structure, size, and required accuracy. Nevertheless, several broad approaches can be distinguished.
The most fundamental distinction is between direct and iterative methods. Direct methods, such as Gaussian elimination, produce an answer in a fixed number of operations; iterative methods, such as Newton's method or the conjugate gradient method, produce a sequence of improving approximations. Direct methods are preferred when the problem is small or when high accuracy is required; iterative methods are preferred for large problems, where direct methods would be too expensive, and for problems where a rough answer is acceptable.
A second distinction is between discretization-based methods and approximation-based methods. Discretization methods replace a continuous problem by a discrete one—a differential equation by a system of algebraic equations, an integral by a sum. Approximation methods represent the solution as a combination of simple functions—polynomials, trigonometric functions, splines—and determine the coefficients by requiring the approximation to satisfy the problem as closely as possible. These approaches often overlap: finite element methods are both discretization and approximation methods, since they discretize the domain and approximate the solution by piecewise polynomials.
A third distinction concerns the treatment of error. A priori error analysis provides bounds on the error in terms of the problem's data and the method's parameters, before any computation is performed. A posteriori error analysis uses the computed solution to estimate the error, allowing the algorithm to adapt—for example, by refining the grid where the error is large. Adaptive methods, which use a posteriori estimates to concentrate computational effort where it is needed, have become increasingly important as problems have grown larger and more complex.
These approaches are not competitors but complementary tools. A typical modern computation might use an iterative method (because the problem is large), with a preconditioner to accelerate convergence, and adaptive refinement based on a posteriori error estimates. The numerical analyst's skill lies in choosing and combining methods appropriately for the problem at hand.
Contemporary numerical analysis is shaped by several developments. The explosive growth of scientific computing has made numerical methods essential to virtually every branch of science and engineering, from climate modeling to drug design to financial risk assessment. This has driven the development of specialized methods for particular classes of problems and the creation of sophisticated software libraries that implement these methods reliably.
The rise of parallel and distributed computing has transformed the field's constraints. Methods that are efficient on a single processor may be poorly suited to parallel architectures, and vice versa. The development of algorithms that scale well to thousands or millions of processors is an active area of research. Similarly, the growth of data science has created new demand for numerical methods for large-scale optimization, low-rank approximation, and randomized algorithms.
A persistent theme is the tension between mathematical elegance and computational practicality. The best method in theory—the one with the fastest convergence or the tightest error bounds—may be impractical because it requires too much computation or too much memory. Numerical analysis is ultimately an engineering discipline as much as a mathematical one: its goal is not just to understand methods but to make them work in practice.
The field's future is likely to be shaped by the same forces that have shaped its past: new applications demanding new methods, new hardware demanding new algorithms, and new mathematical insights demanding new theories. What will not change is the field's central mission: to make the continuous mathematics of the physical world computable, reliably and efficiently, in the discrete world of digital computers.