Computational Imaging

Computational imaging is the practice of building cameras and imaging systems in which computation is not an afterthought but part of the optics.

8 frameworks, from 1965 to today · Computer Graphics · Computer Science

still a working view ended · each leads to its entry below

Overview

Computational imaging is the practice of building cameras and imaging systems in which computation is not an afterthought but part of the optics1. A conventional camera is designed so that a lens focuses light onto a sensor and the sensor records a picture that is already, more or less, the image you want1. Computational imaging breaks that arrangement apart: the optics may deliberately blur or scramble the light, the sensor may capture a coded measurement rather than a picture, and an algorithm reconstructs the image that the hardware never formed1. The field sits between computer graphics, computer vision, and optics, and its central question is a design question: given a task, what should the physical measurement be, and what computation turns it into the answer12? What is at stake is the reach of imaging itself—seeing around corners, capturing a whole light field in one exposure, imaging at wavelengths where lenses are impossible, and making cameras smaller, faster, or cheaper than optics alone allows.

The inverse-problem foundation

The intellectual root of the field is the idea that forming an image is solving an inverse problem13. A forward model predicts what a sensor will measure given a scene; reconstruction inverts that model13. In the 1960s and 1970s, radio astronomers had already made this idea central. Because a radio telescope cannot focus radio waves the way a lens focuses light, aperture-synthesis arrays measure sparse samples of the scene's spatial frequencies, and images are computed from those samples. The 1974 book The Fourier Transform and Its Applications by Ronald Bracewell, and the wider literature on tomography that grew from Godfrey Hounsfield's X-ray computed tomography (announced in the early 1970s, with a Nobel Prize in 1979), taught a generation that a "camera" could be a measurement device plus a reconstruction algorithm. In graphics and vision, the same logic appeared in shape-from-shading, photometric stereo, and structure-from-motion: the picture on the sensor is evidence, and the scene is inferred45.

The practical consequence was that reconstruction quality depended on regularization—extra assumptions that make an underdetermined inversion well-posed6. Early work used smoothness priors and least-squares formulations. This tradition is the backbone on which nearly everything later in computational imaging rests.

Coded apertures and coded exposure

One of the first distinctly computational cameras changed the aperture rather than the lens. In a pinhole camera, a small hole gives a sharp but dim image; a large hole gives a bright but blurred one1. A coded aperture replaces the single hole with a pattern of holes, so that each scene point casts a copy of the pattern onto the sensor1. The overlapping copies are then deconvolved—divided out using knowledge of the pattern—to recover a sharper image at greater light throughput1. The idea has roots in X-ray and gamma-ray astronomy, where lenses are impractical; the mathematician R. H. Dicke proposed coded-aperture imaging for X-rays in 1968. In visible-light photography, the MIT group around Ramesh Raskar brought the concept into computer graphics in the mid-2000s, demonstrating coded-aperture cameras that traded a computable blur for depth information and light efficiency.

A close relative is coded exposure. In 2006 Raskar, Amit Agrawal, and Jack Tumblin showed that by opening and closing the shutter in a known pseudo-random pattern during an exposure, motion blur becomes invertible: the blur is no longer a smear but a code that can be deconvolved to recover a sharp image of a fast-moving object. Both techniques share the field's signature move—deliberately degrade the raw measurement in a structured, known way so that computation can undo it.

Active depth sensing

A parallel line of work asked the camera to send out light and time its return. Time-of-flight cameras modulate a light source and measure the phase shift of the reflected signal, converting a delay into a distance. The principle is old—radar and lidar—but the 2000s brought it to consumer scale. The Swiss company MESA Imaging (spun out of CSEM) sold time-of-flight sensors, and Microsoft's Kinect, launched in 2010, put structured-light depth sensing in living rooms: an infrared projector cast a known dot pattern, and a camera triangulated depth from how the pattern deformed on the scene. The Kinect's success made depth cameras a standard graphics input and drove a wave of research on denoising, calibration, and fusion of depth with color.

Structured light itself descends from earlier industrial metrology, and the graphics community's contribution was to treat the projected pattern as a code to be designed and decoded—again, optics plus computation. Later systems moved to continuous-wave modulation and, eventually, to single-photon avalanche diode arrays that could time individual photons, pushing depth sensing toward the transient regime described below.

Light fields and plenoptic cameras

A conventional photograph collapses the four-dimensional light field—the set of rays arriving at every point from every direction—into two dimensions. In 1996 Marc Levoy and Pat Hanrahan, and independently Steven Gortler and colleagues, showed that capturing many views of a scene allows new views to be synthesized by resampling the light field, a technique that became image-based rendering. The question then became whether a single exposure could capture the light field directly.

The answer was the plenoptic camera7. Ren Ng's 2005 Stanford dissertation, and the Lytro camera commercialized in 2012, placed a microlens array in front of the sensor so that each microlens recorded the rays arriving from a small patch of the main lens. The raw sensor image is a grid of tiny sub-images; computation reassembles them into a focal stack or a set of viewpoints, allowing focus to be changed after the fact2. The trade-off was resolution: spatial resolution is spent to gain angular resolution. Later designs, including focused plenoptic cameras and light-field microscopy, refined that balance, and light-field capture became a tool in microscopy and in virtual-reality capture.

Compressed sensing and single-pixel imaging

In 2006 Emmanuel Candès, Justin Romberg, and Terence Tao, and separately David Donoho, proved a striking result: a signal that is sparse in some basis can be recovered exactly from far fewer measurements than its nominal dimension, provided the measurements are sufficiently incoherent with that basis and reconstruction solves an L1-minimization problem. This was compressed sensing, and it gave computational imaging a general theory for why coded, undersampled measurements can still yield good images8.

The most literal demonstration is the single-pixel camera, built by Richard Baraniuk's group at Rice University around 2006–2007. Instead of a sensor array, it used one photodetector and a digital micromirror device that projected a sequence of random binary patterns onto the scene; each measurement was the total light passing through one pattern. From a few thousand such scalar measurements, compressed sensing reconstructed an image. The approach was slow but showed that a camera need not have a pixel grid at all. Compressed sensing also reshaped magnetic resonance imaging (fast acquisition), and it supplied the mathematical language—sparsity, incoherence, L1 recovery—that later reconstruction work in the field would use18.

Transient and non-line-of-sight imaging

If a camera can time photons to picoseconds, it can watch light propagate. Streak cameras and single-photon detectors made it possible to record a time-resolved image of a short laser pulse bouncing off a scene. In 2012, Andreas Velten, Ramesh Raskar, and colleagues at MIT published "Recovering Three-Dimensional Shape around a Corner using Ultrafast Time-of-Flight Imaging," which used a laser and a streak camera to detect light that had scattered off a wall, hit a hidden object, and returned. By analyzing the arrival times, they reconstructed the shape of the object around the corner—non-line-of-sight imaging. The same transient data supported seeing through scattering media, since early-arriving photons carry less scattered information.

This line of work turned the camera into a device for measuring light transport itself, and it connected computational imaging to the graphics community's Monte Carlo light-transport machinery: reconstruction of hidden scenes is, in effect, an inverse rendering problem. Later methods replaced streak cameras with cheaper single-photon avalanche diode arrays and added priors from scene geometry, but the founding demonstration established the transient regime as a distinct branch of the field.

Learned reconstruction and optics–network co-design

From the 2010s, deep learning changed how reconstruction was done. Instead of hand-designed priors, a neural network could be trained to map measurements to images, learning the regularizer from data1. Early work applied convolutional networks to deconvolution, denoising, and compressed-sensing recovery. The more distinctive development was co-design: rather than fixing the optics and learning the reconstruction, researchers optimized the physical measurement and the reconstruction network together1. A coded aperture pattern, a diffractive optical element, or a lensless mask could be chosen by gradient descent so that the subsequent network reconstructs well. This produced lensless cameras in which a thin mask replaces the lens and a network computes the image, and diffractive optical systems in which a stack of patterned surfaces performs computation optically.

The claim behind co-design is that the best measurement depends on the reconstruction algorithm, so the two should be designed jointly1. Its supporters point to systems that are smaller, cheaper, or faster than lens-based designs; skeptics note that learned reconstructions can fail on out-of-distribution scenes and that the physical realizability of optimized optics must be checked. The debate is unresolved and active.

Neural radiance fields and the inverse-rendering turn

In 2020 Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi, and Ren Ng published NeRF, which represented a scene as a neural network mapping position and viewing direction to density and color, and rendered it by volume rendering along camera rays. Fitting the network to a set of photographs is an inverse-rendering problem, and NeRF made that problem tractable with a positional encoding that let a small network represent high-frequency detail. The result was photorealistic novel-view synthesis from ordinary photographs.

NeRF belongs to computational imaging in the broad sense: it recovers scene structure from measurements by inverting a rendering model, and it descends directly from the light-field and inverse-problem traditions9. Its rapid successors—instant neural graphics primitives, 3D Gaussian splatting (2023), and a family of feed-forward reconstruction networks—made capture-and-reconstruct pipelines fast enough for practical use, and they now connect computational imaging to neural rendering in graphics.

The present

Today the field's main approaches are these: coded measurement plus regularized or learned inversion, as in coded apertures, lensless cameras, and compressed sensing; time-resolved sensing for transient and non-line-of-sight imaging; light-field and plenoptic capture; active depth sensing with structured light and single-photon detectors; and joint optics–algorithm co-design, increasingly with neural networks on both sides110118. NeRF-style inverse rendering has become a common reconstruction framework across them. The open questions are the ones the history keeps returning to: how to make learned reconstructions reliable outside their training distribution, how to design optics that are both optimal and manufacturable, how to scale single-photon and transient sensing beyond laboratory demonstrations, and how far computation can substitute for physical aperture and lens quality.

References

  1. Computational imaging (introduction, Algorithms, Coded-aperture and lensless imaging, History), Wikipedia
  2. Computational photography, Wikipedia
  3. Inverse problem, Wikipedia
  4. Structure from motion, Wikipedia
  5. Photometric stereo, Wikipedia
  6. Regularization (mathematics), Wikipedia
  7. Light field camera, Wikipedia
  8. Compressed sensing, Wikipedia
  9. Neural radiance field, Wikipedia
  10. Single-pixel imaging, Wikipedia
  11. Time-of-flight camera, Wikipedia

The frameworks of Computational Imaging

Each framework is a way of working the field has used: when it held, and the question it answers.

  1. 1965-Present

    Inverse problem

    How can a scene be recovered from indirect measurements?

    Fly to its tower in Nousfarer

  2. 1968-Present

    Coded aperture

    Can a patterned aperture yield a sharper, brighter image?

    Fly to its tower in Nousfarer

  3. 1996-Present

    Light field camera

    Can one exposure capture all rays of light?

    Fly to its tower in Nousfarer

  4. 2000-Present

    Active Depth Sensing

    How can a camera measure distance by sending light?

    Fly to its tower in Nousfarer

  5. 2006-Present

    Compressed Sensing

    Can a sparse signal be recovered from few measurements?

    Fly to its tower in Nousfarer

  6. 2012-Present
  7. 2015-Present
  8. 2020-Present

    Neural radiance field

    Can a network represent a scene from many views?

    Fly to its tower in Nousfarer

How the ideas connect

Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.

Inverse problemCoded apertureLight field cameraActive Depth SensingCompressed SensingTransient and Non-Line-of-Sight Imagi…Learned Reconstruction and Optics–Net…Neural radiance field

Beyond the field

Where Computational Imaging's ideas came from in other fields, and what they led to there: the strongest 12 of 35 links.

A city in Nousfarer: a red-carpeted stair climbing terraces to a crowned tower on an acropolis, more towers round it, forest and snowy mountains beyond.
A city in Nousfarer, the game played in the browser.

Computational Imaging is a planet in Nousfarer

Its city has a tower for each of these 8 frameworks. Land, walk in, and climb one concept at a time.

Land on Computational Imaging

Related fields

Fields its ideas are linked to

More in Computer Graphics

See where Computational Imaging sits on the Map of Human Knowledge