Loading field map
Computational imaging is the discipline within computer graphics and computer vision concerned with forming images through computation rather than through optics alone. In a conventional camera, a lens focuses light onto a sensor, and the sensor records the resulting irradiance pattern. In computational imaging, the optical system is deliberately designed to capture a coded, multiplexed, or otherwise incomplete measurement of the scene, and a reconstruction algorithm later inverts the optical encoding to produce the final image. The field therefore sits at the intersection of optical design, signal processing, and inverse problems, and its central question is: what information about a scene can be recovered from measurements that are not themselves direct pictures of the scene?
The stakes are practical as well as conceptual. Computational imaging makes it possible to capture information that a conventional camera cannot record at all—such as the depth of every pixel, the spectral reflectance of surfaces, or light that has scattered through tissue—and to do so with hardware that is smaller, cheaper, or more robust than a purely optical solution would require. The same principles also allow images to be refocused after capture, extended in dynamic range, or reconstructed from a handful of noisy photon detections. The field is not a single technique but a family of approaches united by a common architecture: a physical forward model describing how light becomes measurements, and a computational inverse model that recovers the desired scene representation from those measurements.
Every computational imaging system shares the same logical structure, though the specific components vary widely. The forward model describes how the scene, the illumination, and the optical system combine to produce the raw sensor readings. This model may be as simple as a linear blur (as in a defocused lens) or as complex as a full simulation of light transport through scattering media, including polarization, wavelength, and time of flight. The measurement is the raw data recorded by the sensor—often a single two-dimensional array of intensity values, but sometimes a sequence of frames, a set of differently coded exposures, or a volumetric stack of focal planes. The inverse problem is the computational step that estimates the scene from the measurements, given the forward model. Because the measurement is almost always lower-dimensional than the scene information being recovered, the inverse problem is ill-posed: many different scenes could have produced the same measurements. Computational imaging therefore relies on priors—assumptions about the statistical or structural properties of natural scenes, such as smoothness, sparsity, or piecewise constancy—to select among the possible reconstructions.
This pipeline distinguishes computational imaging from two neighboring activities. It is not merely digital image processing, because the optical system is designed jointly with the algorithm; the encoding is intentional, not an artifact to be corrected. Nor is it conventional computational photography in the narrow sense of software features added to a standard camera, because the hardware itself is part of the encoding scheme. In practice the boundary is porous: a smartphone’s multi-frame HDR mode, which captures several exposures and merges them, is a simple computational imaging system, while a lensless camera that replaces the lens with a coded amplitude mask is an extreme one.
The intellectual roots of computational imaging lie in several older fields that developed independently. Holography, invented in the mid-twentieth century, records the interference pattern between a reference beam and light scattered from an object; the resulting hologram bears no resemblance to the object, but illuminating it with the reference beam reconstructs the original wavefront. Holography established the crucial idea that a seemingly unintelligible measurement could contain complete information about a light field, recoverable through a known physical transformation. Coherent diffraction imaging, developed later, extended this idea to the case where no reference beam exists, reconstructing an object from the intensity of its diffraction pattern alone using iterative phase-retrieval algorithms.
A second precursor is tomography, which reconstructs a three-dimensional volume from a set of line-integral projections. Computed tomography (CT) scanning, developed for medical imaging in the 1970s, introduced the mathematical framework of the Radon transform and its inverse, and established the general pattern of recovering a higher-dimensional object from lower-dimensional measurements through a carefully characterized forward model. Tomography remains a major branch of computational imaging, particularly in the form of optical projection tomography and diffuse optical tomography.
A third tradition is astronomical image reconstruction, where atmospheric turbulence, imperfect optics, and photon noise corrupt measurements of distant objects. Techniques developed for this domain—deconvolution, speckle interferometry, and later lucky imaging and adaptive optics—demonstrated that computational correction could rival or exceed the performance of perfect optics. These traditions did not use the term "computational imaging," and they developed largely in isolation from one another. The modern field emerged when advances in digital sensors, computational power, and optimization algorithms made it possible to treat these scattered ideas as instances of a single design philosophy.
The field is best organized not by chronology but by the kind of information each approach seeks to encode and recover. Five broad approaches dominate, and they overlap and combine freely.
The oldest and most conceptually direct approach is to replace or modify the lens with a mask that encodes angular information about incoming light. A coded aperture—a patterned occluder placed in the optical path—causes each point in the scene to cast a unique shadow pattern onto the sensor. The measured image is a superposition of these shadows, and reconstruction deconvolves the scene from the known mask pattern. This idea originated in astronomy, where coded masks allowed X-ray and gamma-ray imaging with no focusing optics, and was later adapted for visible light.
Lensless cameras take this to its logical extreme: they remove the lens entirely and place a thin amplitude or phase mask directly above the sensor. Each sensor pixel then receives light from many scene points, and the measurement is a highly multiplexed, seemingly random pattern. Reconstruction requires solving a large inverse problem, typically using the known point-spread function of the mask and a sparsity or smoothness prior. The advantage is a camera that is flat, thin, and potentially cheap, at the cost of substantial computation and sensitivity to calibration errors. The approach remains an active research area rather than a commercial product, but it demonstrates the core computational-imaging trade-off: optical simplicity for computational complexity.
A light field is the complete description of light rays in a region of space—not just their intensity at each point, but their direction of travel. A conventional photograph integrates over all ray directions at each sensor position, discarding directional information. Plenoptic cameras insert a microlens array between the main lens and the sensor, so that the sensor records not one image but a set of sub-images, each corresponding to a different view of the scene. From this single exposure, one can computationally refocus the image at different depths, shift the viewpoint slightly, or estimate scene depth.
The central limitation of plenoptic imaging is the fundamental trade-off between spatial and angular resolution: the sensor has a fixed number of pixels, and dividing them among views reduces the resolution of each. Later designs, such as the focused plenoptic camera, improved resolution by sacrificing some angular range, and hybrid approaches combine plenoptic capture with depth estimation algorithms. Light field imaging is now a standard feature in some professional cameras and in computational photography research, and it illustrates a recurring theme: the measurement captures more dimensions of the light field than a conventional photograph, and the algorithm selects which dimensions to emphasize in reconstruction.
A third approach encodes information in the time dimension. Time-of-flight cameras illuminate the scene with a modulated light source and measure the phase delay of the reflected light, yielding a depth map at every pixel. More advanced transient imaging systems use ultrafast detectors to record the arrival time of light with picosecond precision, capturing not just the first reflection but the full temporal profile of light scattering through and around objects. From such measurements, algorithms can reconstruct scenes hidden around corners, see through scattering media, or separate direct from indirect illumination.
The forward model for time-of-flight imaging is the propagation of light pulses through the scene, including multiple scattering events. The inverse problem is to recover the scene geometry and reflectance from the temporal measurements. This is one of the most computationally demanding branches of the field, because the space of possible light paths is enormous and the measurements are noisy. The approach has demonstrated remarkable results in controlled laboratory settings—such as imaging a scene hidden from direct view by analyzing light that scatters off a wall—but remains far from practical deployment. Its importance lies in showing that the computational-imaging framework can extend beyond what any lens-based system could ever capture.
Conventional cameras record intensity integrated over broad wavelength bands and over all polarization states. Hyperspectral imaging captures a full spectrum at each pixel, enabling material identification, vegetation analysis, and medical diagnostics. Polarimetric imaging records the polarization state of light, which reveals surface orientation, material properties, and stress patterns invisible to intensity measurements.
Both approaches face the same fundamental problem: a sensor that records the full spectral or polarimetric information at every pixel would be prohibitively expensive or slow. Computational imaging addresses this through compressive sensing and coded sampling. A coded aperture snapshot spectral imager (CASSI) uses a dispersive element and a random mask to multiplex spectral information into a single sensor frame, then reconstructs the full spectral cube using sparsity priors. Similarly, a polarimetric camera can use a patterned micro-polarizer array that samples different polarization states at different pixels, with reconstruction filling in the missing information. These systems embody the field’s central bargain: the measurement is deliberately incomplete, but the known structure of natural scenes makes the missing information recoverable.
The most ambitious branch of computational imaging treats the entire image formation process—including illumination, geometry, and material properties—as the forward model, and seeks to recover the full scene description from one or more photographs. Inverse rendering estimates the three-dimensional geometry, surface reflectance, and lighting of a scene from images. Computational illumination actively controls the light source, projecting structured patterns or sweeping a light source across the scene, to make the inverse problem more tractable.
This approach differs from the others in that the "measurement" is often a conventional photograph or a small set of photographs, and the computational burden lies almost entirely in the inverse model. The forward model must account for global illumination effects—shadows, interreflections, subsurface scattering—which makes the inverse problem highly nonlinear and computationally expensive. Modern methods increasingly use differentiable rendering, in which the rendering pipeline is implemented as a differentiable program, allowing gradients to flow from the image error back to the scene parameters. This has enabled high-quality reconstruction of geometry and materials from a handful of images, and it connects computational imaging directly to the broader field of neural rendering and inverse graphics.
These five approaches are not rival schools in the sense of mutually exclusive research programs; they are complementary strategies for encoding different dimensions of scene information. Coded aperture and lensless imaging encode angular information through spatial masks; light field imaging encodes the same information through microlens arrays; time-of-flight encodes temporal information; spectral and polarimetric methods encode wavelength and polarization; inverse rendering encodes the full scene model. Many modern systems combine approaches: a time-of-flight camera can also capture a conventional image; a light field camera can be combined with a coded mask; a hyperspectral imager can use structured illumination.
The deeper unity lies in the mathematical structure. Nearly every computational imaging system can be described as a linear or nonlinear operator applied to a scene, followed by a reconstruction that inverts the operator under constraints. This has led to a productive exchange of methods across subfields: compressed sensing theory, developed for signal acquisition, now informs lensless camera design; deep learning, developed for image classification, now provides learned priors for inverse problems; differentiable rendering, developed for computer graphics, now powers inverse rendering. The field is therefore characterized less by a stable set of canonical techniques than by a shared design methodology: choose an optical encoding, write the forward model, and solve the inverse problem.
The most significant recent development in computational imaging is the integration of learned methods into the reconstruction pipeline. Traditional reconstruction algorithms rely on hand-designed priors—total variation, sparsity in a wavelet basis, smoothness—that are generic and often weak. Deep neural networks can learn priors directly from large datasets of natural images, and they can also learn the inverse mapping itself, from measurements to reconstructions, without an explicit forward model.
Learned methods have produced dramatic improvements in reconstruction quality for nearly every branch of the field, from lensless imaging to spectral reconstruction to time-of-flight depth estimation. However, they introduce a new set of concerns. A network trained on one type of scene may fail unpredictably on another; the reconstruction may be plausible but not faithful to the actual scene; and the network may exploit artifacts of the training data rather than the physical information in the measurement. The field has therefore developed a spectrum of approaches, from model-based deep learning, which unrolls an iterative optimization algorithm and replaces the prior with a learned network, to end-to-end learning, which treats the entire system—optics and algorithm—as a differentiable pipeline that can be optimized jointly. The former preserves the interpretability and guarantees of the forward model; the latter achieves higher performance at the cost of physical transparency.
Computational imaging is an active and expanding field, driven by the continued improvement of sensors, the falling cost of computation, and the demand for imaging capabilities that conventional optics cannot provide. In consumer devices, computational imaging is now the default: smartphone cameras routinely use multi-frame fusion, depth estimation, and learned enhancement. In scientific and medical imaging, computational methods enable microscopy that exceeds the diffraction limit, imaging through scattering tissue, and non-line-of-sight imaging for safety and inspection. In remote sensing, spectral and polarimetric computational imagers provide information that conventional cameras cannot.
The field’s open problems are largely problems of robustness and trust. Reconstruction algorithms must be reliable across the full range of scenes and conditions, not just on benchmark datasets; they must provide uncertainty estimates so that a user knows when a reconstruction is trustworthy; and they must be calibrated to the physical system, since small errors in the forward model can produce large errors in the inverse solution. A second class of problems concerns the design space itself: given a desired imaging task, what is the optimal optical encoding, and how should it be co-designed with the reconstruction algorithm? This is a joint optimization problem over hardware and software, and it remains largely unsolved in general form.
Computational imaging has not replaced conventional photography, nor is it likely to. But it has permanently changed the understanding of what an image is. An image is no longer necessarily a direct record of light; it is the output of a computational process that infers scene properties from measurements. The field’s enduring contribution is the demonstration that the boundary between optics and computation is not fixed—it is a design choice, and the best place to draw that boundary depends on the task, the hardware, and the available computation.