- Atlas
- Computer Science
- Computational Imaging
Computational Imaging
Computational imaging is the practice of building cameras and imaging systems in which computation is not an afterthought but part of the optics.
8 frameworks, from 1965 to today · Computer Graphics · Computer Science
- Inverse problem 1965-Present
- Coded aperture 1968-Present
- Light field camera 1996-Present
- Active Depth Sensing 2000-Present
- Compressed Sensing 2006-Present
- Transient and Non-Line-of-Sight Imaging 2012-Present
- Learned Reconstruction and Optics–Network Co-Design 2015-Present
- Neural radiance field 2020-Present
Overview
Computational imaging is the practice of building cameras and imaging systems in which computation is not an afterthought but part of the optics1. A conventional camera is designed so that a lens focuses light onto a sensor and the sensor records a picture that is already, more or less, the image you want1. Computational imaging breaks that arrangement apart: the optics may deliberately blur or scramble the light, the sensor may capture a coded measurement rather than a picture, and an algorithm reconstructs the image that the hardware never formed1. The field sits between computer graphics, computer vision, and optics, and its central question is a design question: given a task, what should the physical measurement be, and what computation turns it into the answer12? What is at stake is the reach of imaging itself—seeing around corners, capturing a whole light field in one exposure, imaging at wavelengths where lenses are impossible, and making cameras smaller, faster, or cheaper than optics alone allows.
The inverse-problem foundation
The intellectual root of the field is the idea that forming an image is solving an inverse problem13. A forward model predicts what a sensor will measure given a scene; reconstruction inverts that model13. In the 1960s and 1970s, radio astronomers had already made this idea central. Because a radio telescope cannot focus radio waves the way a lens focuses light, aperture-synthesis arrays measure sparse samples of the scene's spatial frequencies, and images are computed from those samples. The 1974 book The Fourier Transform and Its Applications by Ronald Bracewell, and the wider literature on tomography that grew from Godfrey Hounsfield's X-ray computed tomography (announced in the early 1970s, with a Nobel Prize in 1979), taught a generation that a "camera" could be a measurement device plus a reconstruction algorithm. In graphics and vision, the same logic appeared in shape-from-shading, photometric stereo, and structure-from-motion: the picture on the sensor is evidence, and the scene is inferred45.
The practical consequence was that reconstruction quality depended on regularization—extra assumptions that make an underdetermined inversion well-posed6. Early work used smoothness priors and least-squares formulations. This tradition is the backbone on which nearly everything later in computational imaging rests.
Coded apertures and coded exposure
One of the first distinctly computational cameras changed the aperture rather than the lens. In a pinhole camera, a small hole gives a sharp but dim image; a large hole gives a bright but blurred one1. A coded aperture replaces the single hole with a pattern of holes, so that each scene point casts a copy of the pattern onto the sensor1. The overlapping copies are then deconvolved—divided out using knowledge of the pattern—to recover a sharper image at greater light throughput1. The idea has roots in X-ray and gamma-ray astronomy, where lenses are impractical; the mathematician R. H. Dicke proposed coded-aperture imaging for X-rays in 1968. In visible-light photography, the MIT group around Ramesh Raskar brought the concept into computer graphics in the mid-2000s, demonstrating coded-aperture cameras that traded a computable blur for depth information and light efficiency.
A close relative is coded exposure. In 2006 Raskar, Amit Agrawal, and Jack Tumblin showed that by opening and closing the shutter in a known pseudo-random pattern during an exposure, motion blur becomes invertible: the blur is no longer a smear but a code that can be deconvolved to recover a sharp image of a fast-moving object. Both techniques share the field's signature move—deliberately degrade the raw measurement in a structured, known way so that computation can undo it.
Active depth sensing
A parallel line of work asked the camera to send out light and time its return. Time-of-flight cameras modulate a light source and measure the phase shift of the reflected signal, converting a delay into a distance. The principle is old—radar and lidar—but the 2000s brought it to consumer scale. The Swiss company MESA Imaging (spun out of CSEM) sold time-of-flight sensors, and Microsoft's Kinect, launched in 2010, put structured-light depth sensing in living rooms: an infrared projector cast a known dot pattern, and a camera triangulated depth from how the pattern deformed on the scene. The Kinect's success made depth cameras a standard graphics input and drove a wave of research on denoising, calibration, and fusion of depth with color.
Structured light itself descends from earlier industrial metrology, and the graphics community's contribution was to treat the projected pattern as a code to be designed and decoded—again, optics plus computation. Later systems moved to continuous-wave modulation and, eventually, to single-photon avalanche diode arrays that could time individual photons, pushing depth sensing toward the transient regime described below.
Light fields and plenoptic cameras
A conventional photograph collapses the four-dimensional light field—the set of rays arriving at every point from every direction—into two dimensions. In 1996 Marc Levoy and Pat Hanrahan, and independently Steven Gortler and colleagues, showed that capturing many views of a scene allows new views to be synthesized by resampling the light field, a technique that became image-based rendering. The question then became whether a single exposure could capture the light field directly.
The answer was the plenoptic camera7. Ren Ng's 2005 Stanford dissertation, and the Lytro camera commercialized in 2012, placed a microlens array in front of the sensor so that each microlens recorded the rays arriving from a small patch of the main lens. The raw sensor image is a grid of tiny sub-images; computation reassembles them into a focal stack or a set of viewpoints, allowing focus to be changed after the fact2. The trade-off was resolution: spatial resolution is spent to gain angular resolution. Later designs, including focused plenoptic cameras and light-field microscopy, refined that balance, and light-field capture became a tool in microscopy and in virtual-reality capture.
Compressed sensing and single-pixel imaging
In 2006 Emmanuel Candès, Justin Romberg, and Terence Tao, and separately David Donoho, proved a striking result: a signal that is sparse in some basis can be recovered exactly from far fewer measurements than its nominal dimension, provided the measurements are sufficiently incoherent with that basis and reconstruction solves an L1-minimization problem. This was compressed sensing, and it gave computational imaging a general theory for why coded, undersampled measurements can still yield good images8.
The most literal demonstration is the single-pixel camera, built by Richard Baraniuk's group at Rice University around 2006–2007. Instead of a sensor array, it used one photodetector and a digital micromirror device that projected a sequence of random binary patterns onto the scene; each measurement was the total light passing through one pattern. From a few thousand such scalar measurements, compressed sensing reconstructed an image. The approach was slow but showed that a camera need not have a pixel grid at all. Compressed sensing also reshaped magnetic resonance imaging (fast acquisition), and it supplied the mathematical language—sparsity, incoherence, L1 recovery—that later reconstruction work in the field would use18.
Transient and non-line-of-sight imaging
If a camera can time photons to picoseconds, it can watch light propagate. Streak cameras and single-photon detectors made it possible to record a time-resolved image of a short laser pulse bouncing off a scene. In 2012, Andreas Velten, Ramesh Raskar, and colleagues at MIT published "Recovering Three-Dimensional Shape around a Corner using Ultrafast Time-of-Flight Imaging," which used a laser and a streak camera to detect light that had scattered off a wall, hit a hidden object, and returned. By analyzing the arrival times, they reconstructed the shape of the object around the corner—non-line-of-sight imaging. The same transient data supported seeing through scattering media, since early-arriving photons carry less scattered information.
This line of work turned the camera into a device for measuring light transport itself, and it connected computational imaging to the graphics community's Monte Carlo light-transport machinery: reconstruction of hidden scenes is, in effect, an inverse rendering problem. Later methods replaced streak cameras with cheaper single-photon avalanche diode arrays and added priors from scene geometry, but the founding demonstration established the transient regime as a distinct branch of the field.
Learned reconstruction and optics–network co-design
From the 2010s, deep learning changed how reconstruction was done. Instead of hand-designed priors, a neural network could be trained to map measurements to images, learning the regularizer from data1. Early work applied convolutional networks to deconvolution, denoising, and compressed-sensing recovery. The more distinctive development was co-design: rather than fixing the optics and learning the reconstruction, researchers optimized the physical measurement and the reconstruction network together1. A coded aperture pattern, a diffractive optical element, or a lensless mask could be chosen by gradient descent so that the subsequent network reconstructs well. This produced lensless cameras in which a thin mask replaces the lens and a network computes the image, and diffractive optical systems in which a stack of patterned surfaces performs computation optically.
The claim behind co-design is that the best measurement depends on the reconstruction algorithm, so the two should be designed jointly1. Its supporters point to systems that are smaller, cheaper, or faster than lens-based designs; skeptics note that learned reconstructions can fail on out-of-distribution scenes and that the physical realizability of optimized optics must be checked. The debate is unresolved and active.
Neural radiance fields and the inverse-rendering turn
In 2020 Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng published NeRF, which represented a scene as a neural network mapping position and viewing direction to density and color, and rendered it by volume rendering along camera rays. Fitting the network to a set of photographs is an inverse-rendering problem, and NeRF made that problem tractable with a positional encoding that let a small network represent high-frequency detail. The result was photorealistic novel-view synthesis from ordinary photographs.
NeRF belongs to computational imaging in the broad sense: it recovers scene structure from measurements by inverting a rendering model, and it descends directly from the light-field and inverse-problem traditions9. Its rapid successors—instant neural graphics primitives, 3D Gaussian splatting (2023), and a family of feed-forward reconstruction networks—made capture-and-reconstruct pipelines fast enough for practical use, and they now connect computational imaging to neural rendering in graphics.
The present
Today the field's main approaches are these: coded measurement plus regularized or learned inversion, as in coded apertures, lensless cameras, and compressed sensing; time-resolved sensing for transient and non-line-of-sight imaging; light-field and plenoptic capture; active depth sensing with structured light and single-photon detectors; and joint optics–algorithm co-design, increasingly with neural networks on both sides110118. NeRF-style inverse rendering has become a common reconstruction framework across them. The open questions are the ones the history keeps returning to: how to make learned reconstructions reliable outside their training distribution, how to design optics that are both optimal and manufacturable, how to scale single-photon and transient sensing beyond laboratory demonstrations, and how far computation can substitute for physical aperture and lens quality.
References
- Computational imaging (introduction, Algorithms, Coded-aperture and lensless imaging, History), Wikipedia
- Computational photography, Wikipedia
- Inverse problem, Wikipedia
- Structure from motion, Wikipedia
- Photometric stereo, Wikipedia
- Regularization (mathematics), Wikipedia
- Light field camera, Wikipedia
- Compressed sensing, Wikipedia
- Neural radiance field, Wikipedia
- Single-pixel imaging, Wikipedia
- Time-of-flight camera, Wikipedia
The frameworks of Computational Imaging
Each framework is a way of working the field has used: when it held, and the question it answers.
- 1965-Present
Inverse problem
How can a scene be recovered from indirect measurements?
- 1968-Present
Coded aperture
Can a patterned aperture yield a sharper, brighter image?
- 1996-Present
- 2000-Present
Active Depth Sensing
How can a camera measure distance by sending light?
- 2006-Present
Compressed Sensing
Can a sparse signal be recovered from few measurements?
- 2012-Present
Transient and Non-Line-of-Sight Imaging
Can we see objects hidden around a corner?
- 2015-Present
Learned Reconstruction and Optics–Network Co-Design
Can networks learn to invert coded measurements?
- 2020-Present
How the ideas connect
Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.
Compressed Sensing influenced by Inverse problem
Compressed sensing supplied a general theory of when undersampled linear measurements can be inverted, and the single-pixel camera realized it as a coded measurement plus L1 reconstruction. The inverse-problem framing of a forward model and regularized inversion carried directly into this work.
Carried by Emmanuel Candès; David Donoho; Richard Baraniuk.
Transient and Non-Line-of-Sight Imaging influenced by Inverse problem
Non-line-of-sight reconstruction inverts a light-transport forward model: measured transient responses are related to hidden geometry through a model of scattering, and the scene is recovered by inversion with priors. The MIT demonstration framed the problem in exactly this inverse-rendering way.
Carried by Andreas Velten; Ramesh Raskar.
Learned Reconstruction and Optics–Network Co-Design influenced by Inverse problem
Learned reconstruction keeps the forward model of the imaging system and replaces the hand-designed regularizer with a trained network, often unrolled as iterations of a proximal inversion. The inverse-problem formulation is the skeleton on which learned reconstruction is built.
Compressed Sensing influenced by Coded aperture
The single-pixel camera is a coded-aperture idea taken to its limit: a digital micromirror device imposes a sequence of binary patterns on the scene and one detector records the total light. Coded-aperture imaging established the practice of structured, known modulation before detection that the single-pixel camera adopted.
Carried by Richard Baraniuk.
Transient and Non-Line-of-Sight Imaging influenced by Active Depth Sensing
Time-of-flight depth sensing established the practice of timing light to measure distance, and single-photon avalanche diode arrays developed for depth sensing were later used for transient and non-line-of-sight capture. The photon-timing hardware and the delay-to-distance logic passed directly into the transient regime.
Learned Reconstruction and Optics–Network Co-Design influenced by Compressed Sensing
Learned reconstruction inherited the compressed-sensing problem setting—undersampled, coded measurements recovered with a sparsity prior—and replaced the L1 solver with a trained network or an unrolled sparse-recovery iteration. The measurement models and benchmarks came from compressed sensing.
Light field camera influenced by Inverse problem
Plenoptic reconstruction is posed as an inverse problem: the microlens raw image is a coded measurement of the light field, and computation resamples it into focal stacks or viewpoints. The framing of capture-plus-reconstruction follows the inverse-problem tradition.
Carried by Ren Ng.
Active Depth Sensing influenced by Inverse problem
Structured-light depth sensing treats the projected pattern as a code and triangulation as decoding, and the graphics community's work on denoising and calibrating Kinect depth framed depth recovery as an estimation problem with priors.
Carried by Microsoft Kinect.
Neural radiance field influenced by Inverse problem
Neural radiance fields fit a volumetric scene model to posed images by optimizing a rendering loss, which is an inverse rendering problem in the same forward-model-and-invert tradition. The reconstruction-from-measurements logic carries over even though the representation is a network.
Active Depth Sensing influenced by Coded aperture
Structured-light depth sensing projects a known pattern and decodes its deformation, the same design move as coded apertures: impose a known code on the light so computation can recover what the raw sensor does not show. The graphics community's coded-aperture work fed into this pattern-as-code approach.
Carried by Ramesh Raskar.
Neural radiance field influenced by Light field camera
Light-field capture and image-based rendering established that a scene can be represented by rays and resynthesized from many views, a view-dependent representation. Neural radiance fields generalize this ray-based, view-dependent scene representation with a learned continuous function.
Carried by Marc Levoy; Pat Hanrahan.
Transient and Non-Line-of-Sight Imaging influenced by Compressed Sensing
Transient reconstruction from sparse, noisy photon arrivals uses sparsity and regularized inversion in the compressed-sensing style, and the single-photon detection methods share the coded-measurement-plus-recovery logic. The mathematical language of sparse recovery carried into non-line-of-sight work.
Beyond the field
Where Computational Imaging's ideas came from in other fields, and what they led to there: the strongest 12 of 35 links.
Inverse problem influenced Computational and Model-Based Imaging in Imaging Physics
The inverse-problem framework, formalised from the 1960s in applied mathematics, supplied the regularization theory and ill-posedness analysis on which model-based image reconstruction rests.
Inverse problem influenced Tomographic reconstruction in Imaging Physics
The theory of inverse problems, especially the Radon transform and its inversion, is the mathematical core of tomographic reconstruction, and Cormack and Hounsfield drew on it directly. The later framework is the application of this earlier mathematics to medical imaging.
Carried by Johann Radon; Allan M. Cormack; Godfrey Hounsfield.
Inverse problem led to Iterative reconstruction in Image Reconstruction
Iterative reconstruction is a computational strategy for solving inverse problems, and the inverse-problem formulation (ill-posedness, regularization, data fitting) supplied its mathematical framing. The algebraic reconstruction and expectation-maximization algorithms of the early 1970s were presented explicitly as inverse-problem solutions.
Carried by Richard Gordon; Gabor Herman.
Learned Reconstruction and Optics–Network Co-Design grew out of Computational and Model-Based Imaging in Imaging Physics
Learned reconstruction and optics-network co-design is a direct continuation of model-based computational imaging: the forward model and inversion machinery of model-based imaging were kept and the priors and regularisers were replaced by learned networks. The later framework generalises the earlier one rather than replacing its physics.
Neural radiance field influenced Neural rendering in Virtual Reality
Neural radiance fields, introduced by Mildenhall and colleagues in 2020, became the central technique of neural rendering: a multilayer perceptron maps position and view direction to density and colour, and volume rendering integrates it. Neural rendering frameworks built directly on this representation and its follow-ups.
Carried by Ben Mildenhall; NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (2020).
Inverse problem influenced by Well-posed problem in Inverse Problems
Hadamard's distinction between well-posed and ill-posed problems, formulated around 1902, supplied the vocabulary and the problem class that the modern inverse-problems literature organized itself around. Tikhonov and later regularization work framed imaging inversion explicitly as restoring well-posedness to Hadamard-ill-posed problems.
Carried by Jacques Hadamard.
Inverse problem influenced by Integral Equation Inversion in Inverse Problems
Fredholm and Volterra integral-equation inversion provided the mathematical machinery that imaging inverse problems are written in, since image formation is modeled as a convolution or integral operator to be inverted. The inverse-problem framework inherited this formulation directly.
Inverse problem influenced Inverse Optics and Constraint-Based Reconstruction in Computer Vision
Vision as inverse optics took the general formulation of ill-posed inverse problems and applied it to recovering scene properties from images. The regularization methods of inverse-problem theory were carried into shape-from-shading and related vision work.
Inverse problem influenced Compressed sensing in Inverse Problems
Compressed sensing was framed as a linear inverse problem with sparsity regularization, inheriting the inverse-problem formulation of ill-posed recovery. The inverse-problem literature supplied the regularization viewpoint that compressed sensing adopted.
Inverse problem influenced Model-Based Reconstruction in Image Reconstruction
Model-based reconstruction applies inverse-problem theory, formulating imaging as regularized minimization of a data-fit term with a forward model. The regularization and ill-posedness concepts came from that literature.
Carried by Jeffrey Fessler.
Inverse problem led to Global Optimization Inversion in Inverse Problems
Global optimization inversion is a strategy within the inverse-problem field, addressing nonconvex data-fitting objectives. The inverse-problem literature defined the misfit-minimization problem that global methods solve.
Inverse problem led to Learning-Based Inversion in Inverse Problems
Learning-based inversion is a branch of the inverse-problem field, replacing hand-designed regularizers with learned priors and networks. The inverse-problem formulation of recovering a model from measurements remains its frame.

Computational Imaging is a planet in Nousfarer
Its city has a tower for each of these 8 frameworks. Land, walk in, and climb one concept at a time.
Land on Computational ImagingRelated fields
Fields its ideas are linked to
- Inverse Problems 8 links
- Image Reconstruction 5 links
- Computer Vision 4 links
- Imaging Physics 4 links
- Optics 4 links
- Graphics Rendering 3 links
- Virtual Reality 2 links
- Radio Astronomy 1 link
- Coding Theory 1 link
- Computer Graphics 1 link
- Geometric Modeling 1 link
- Medical Imaging 1 link
More in Computer Graphics
- Computer Animation
- Computer Graphics
- Game Technology
- Geometric Modeling
- Graphics Rendering
- Physical Simulation
- Virtual Reality
See where Computational Imaging sits on the Map of Human Knowledge