Computer Vision

Computer vision is the branch of artificial intelligence concerned with getting machines to extract meaning from images and video: to say what is where, to recognize a face or a road sign, to reconstruct a scene in three dimensions, or to generate a picture from a sentence.

13 frameworks, from 1958 to today · Artificial Intelligence · Computer Science

still a working view ended · each leads to its entry below

Overview

Computer vision is the branch of artificial intelligence concerned with getting machines to extract meaning from images and video: to say what is where, to recognize a face or a road sign, to reconstruct a scene in three dimensions, or to generate a picture from a sentence1. What is at stake is both practical and philosophical. Practically, cameras are everywhere, and most of the visual information in the world is never seen by a human eye. Philosophically, vision is the clearest test of whether perception can be understood as computation — whether seeing can be broken into steps a machine can carry out. The field's history is a long argument about what those steps are, and about how much of the answer must be learned from data rather than designed by hand.

Early Machines, Edges, and the Perceptron

Work on machine perception began in the 1950s and 1960s alongside the wider project of artificial intelligence. Early systems were small and hand-crafted. In 1959, the neurophysiologists David Hubel and Torsten Wiesel, recording from the visual cortex of cats, found neurons that responded to oriented edges and to moving bars — evidence that early vision decomposes the image into local features. That finding shaped how engineers thought about the problem for decades.

The perceptron, introduced by Frank Rosenblatt in 1958, was a simple learning machine: a layer of threshold units whose weights were adjusted from labelled examples2. It could be trained to classify simple patterns, and it was demonstrated on a machine built to recognize shapes on a photoelectric retina. In 1969 Marvin Minsky and Seymour Papert published Perceptrons, proving that a single-layer perceptron cannot compute certain functions, such as exclusive-or, and arguing that the approach was fundamentally limited2. The critique was mathematically correct for one layer, but its influence cooled neural-network research for years. The deeper problem it exposed — that useful vision needs internal representations, not just a mapping from pixels to labels — set the agenda for what followed.

Blocks World and the Geometry of Scenes

Through the 1960s and 1970s, a dominant research style treated vision as a problem of reconstructing a scene's geometry from an image, using explicit models of the world. The canonical testbed was the blocks world: a tabletop of simple polyhedra, lit and photographed, whose edges and faces a program had to recover. At MIT, work associated with Larry Roberts (his 1963 doctoral thesis on machine perception of three-dimensional solids) and later with David Marr and others built systems that detected edges, grouped them into lines, and fitted them to models of blocks and wedges.

These systems embodied a belief: that the world is simple enough to describe with hand-built geometric primitives, and that vision is largely a matter of matching image evidence to those descriptions. They worked on clean, controlled scenes and failed on anything messier. The lesson practitioners drew was that the hard part is not the geometry of a perfect block but the ambiguity of real images — shadows, texture, occlusion, and noise.

Reconstruction, Inverse Optics, and Marr's Program

A second strand asked how a machine could recover three-dimensional structure from two-dimensional images using the physics of image formation. This was sometimes called inverse optics: if you know how light, surfaces, and cameras interact, you can invert the process to infer depth and shape. Work on stereo (matching points between two views), shape from shading, shape from texture, and optical flow (the apparent motion of image points between frames) flourished in the 1970s and 1980s. Berthold Horn's work on shape from shading and on optical flow, and the development of the constraint style of reasoning — adding assumptions until an underdetermined problem becomes solvable — were central.

The most influential synthesis was David Marr's3. In Vision (published in 1982, after his death), Marr argued that vision should be studied at three levels: the computational level (what problem is being solved and why), the algorithmic level (what representations and procedures carry it out), and the implementational level (how it is realized in hardware or neurons). He proposed that early vision produces a primal sketch of edges and blobs, then a 2½-D sketch of visible surfaces oriented in depth, and finally a 3-D model of objects. Marr's framework gave the field a shared vocabulary and a research program: find the representations, then the algorithms. Its emphasis on explicit, staged representations was later challenged, but its insistence on asking what a computation is for remains a standard way to think about perception.

Statistical Appearance and the Turn to Learning

By the late 1980s, hand-built geometric models were hitting a wall on real images, and attention shifted to statistical methods that learn appearance from examples. A landmark was eigenfaces, introduced by Matthew Turk and Alex Pentland in 1991: face images were represented as combinations of a small set of basis images computed from a training set by principal component analysis, and faces were recognized by their coordinates in that space4. The idea — that a class of objects occupies a low-dimensional subspace of pixel space — generalized widely.

Probabilistic graphical models gave the field a language for uncertainty1. Hidden Markov models were applied to gesture and speech; later, conditional random fields and other structured models handled segmentation and labelling. The shift was methodological as much as technical: instead of asking what rule a human would write, researchers asked what distribution the data came from and how to estimate it. This was the beginning of vision as a statistical learning problem.

Features, Matching, and Hand-Engineered Pipelines

The 1990s and 2000s were the era of hand-engineered features: descriptors designed by researchers to capture useful image structure, paired with classifiers trained on labelled data. David Lowe's SIFT (scale-invariant feature transform, published 1999–2004) detected distinctive keypoints and described each by a histogram of local gradient orientations, making matching robust to changes in scale and rotation5. Similar detectors and descriptors — Harris corners, SURF, and others — made geometric feature matching practical: find corresponding points between two images, then estimate the transformation between them67. This powered panorama stitching, object recognition, and the structure-from-motion pipelines that reconstruct 3D scenes from many photographs.

For recognition, the bag-of-visual-words model treated image patches like words in a document, and support vector machines classified the resulting histograms. The histogram of oriented gradients (HOG) descriptor, combined with a sliding-window classifier, gave strong pedestrian detection8. The PASCAL VOC challenges, running from 2005, standardized datasets and metrics and drove steady progress. The pattern of this era was a pipeline: detect features, describe them, pool them, classify. Each stage was designed by a human, and each stage's design was a research contribution.

Deep Convolutional Networks

The decisive break came from neural networks, revived by advances in learning algorithms and computing power. A convolutional neural network (CNN) is a layered network that applies small learned filters across an image, sharing weights across positions — a design inspired by the receptive fields Hubel and Wiesel had found9. Yann LeCun and colleagues demonstrated in the late 1980s and 1990s that CNNs trained by backpropagation could read handwritten digits, and their LeNet system was deployed commercially for cheque reading. But for years CNNs were beaten on harder benchmarks by hand-engineered pipelines, and the field remained skeptical.

The turning point was the ImageNet challenge10. ImageNet, assembled by Fei-Fei Li and colleagues from 2007, was a dataset of millions of labelled images across a thousand categories. In 2012, Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton entered AlexNet, a deep CNN trained on graphics processors, and cut the top-5 error rate dramatically below the best hand-engineered entries10. The result convinced much of the field. The lesson was that features could be learned end-to-end from data, and that scale — in data and computation — mattered enormously. Within a few years, nearly every vision benchmark was led by deep networks, and the hand-engineered pipeline gave way to learned representations.

Detection, Segmentation, and the Architecture Boom

Deep learning then spread across vision's subproblems, each generating its own architectures. Object detection — locating and labelling multiple objects in an image — moved from region-proposal methods such as R-CNN (2014) and its faster successors to single-stage detectors such as YOLO (2015) and SSD, which predicted boxes and classes in one pass. Semantic segmentation — labelling every pixel — was transformed by fully convolutional networks (2015), which replaced the classifier head with convolutional layers that output a spatial map. Instance segmentation combined detection and pixel labelling, notably in Mask R-CNN (2017)11. These architectures shared a pattern: deep convolutional backbones, multi-scale feature pyramids, and task-specific heads, trained on large labelled datasets.

The period also brought generative models into vision. Generative adversarial networks, introduced by Ian Goodfellow and colleagues in 2014, trained a generator to fool a discriminator, producing increasingly realistic images12. Variational autoencoders and, later, diffusion models (which generate images by learning to reverse a gradual noising process) extended the field's reach from recognition to synthesis1314. By the late 2010s, a system could not only label an image but produce a plausible new one.

Transformers, Multimodal Models, and the Present

The most recent shift imported an architecture from natural language processing. The transformer, introduced in 2017, processes a sequence of tokens by attention: each token weighs how much to draw on every other token, rather than convolving over a fixed local neighbourhood15. In 2020, the Vision Transformer showed that an image could be cut into patches, treated as a sequence of tokens, and classified competitively — and, with enough data, better than CNNs. Attention's flexibility, and its ability to relate distant parts of an image, made it the backbone of a new generation.

The same years saw vision become multimodal. Models such as CLIP (2021) were trained on hundreds of millions of image–text pairs to place images and their captions in a shared embedding space, so that a text description could retrieve or classify images without task-specific training. This "foundation model" approach — pretrain broadly, adapt to many tasks — now underlies image generation from text, visual question answering, and open-vocabulary detection. The boundary between computer vision and language modelling has blurred, and many current systems are trained jointly on both.

The questions still open are recognizably the old ones in new clothes. How much can be learned from data versus designed? How do models generalize to situations unlike their training data — new viewpoints, lighting, or objects? How can a system explain why it produced a label, and how can it be made robust to adversarial or unusual inputs? And the deepest question, inherited from Marr, remains: what is the right level at which to describe what a seeing machine is actually doing3? The field's history is a record of answers that worked for a while, each replaced when the world turned out to be messier than the model assumed.

References

  1. Computer vision (introduction, Definition, History), Wikipedia
  2. Perceptrons (book) (Background, introduction), Wikipedia
  3. David Marr (scientist), Wikipedia
  4. Eigenface (introduction, History), Wikipedia
  5. Scale-invariant feature transform, Wikipedia
  6. Speeded up robust features, Wikipedia
  7. Harris corner detector, Wikipedia
  8. Histogram of oriented gradients, Wikipedia
  9. Convolutional neural network, Wikipedia
  10. AlexNet, Wikipedia
  11. Region-based convolutional neural network (History), Wikipedia
  12. Generative adversarial network, Wikipedia
  13. Variational autoencoder, Wikipedia
  14. Diffusion model, Wikipedia
  15. Transformer (deep learning), Wikipedia

The frameworks of Computer Vision

Each framework is a way of working the field has used: when it held, and the question it answers.

  1. 1958-1969

    Perceptron

    Can a simple network learn to classify patterns from examples?

    Fly to its tower in Nousfarer

  2. 1963-1980

    Blocks world

    How can a program recover the geometry of simple objects from an image?

    Fly to its tower in Nousfarer

  3. 1970-1990

    Inverse Optics and Constraint-Based Reconstruction

    How can depth and shape be inferred from the physics of image formation?

    Fly to its tower in Nousfarer

  4. 1982-1995

    Marr's Computational Theory of Vision

    What are the computational stages that turn an image into a description?

    Fly to its tower in Nousfarer

  5. 1985-2012

    Probabilistic Graphical Models for Vision

    How should uncertainty be represented and inferred in visual interpretation?

    Fly to its tower in Nousfarer

  6. 1991-2010

    Statistical Appearance Models and Subspace Learning

    Can object appearance be captured by a low-dimensional learned subspace?

    Fly to its tower in Nousfarer

  7. 1999-2012

    Geometric Feature Matching

    How can corresponding points be found between different views of a scene?

    Fly to its tower in Nousfarer

  8. 1999-2014

    Hand-Engineered Feature Pipelines

    Which hand-designed features and classifiers best recognize objects?

    Fly to its tower in Nousfarer

  9. 2012-Present

    Convolutional neural network

    Can useful visual features be learned end-to-end from raw pixels?

    Fly to its tower in Nousfarer

  10. 2014-Present

    Deep Generative Models

    How can a model learn to generate realistic images from data?

    Fly to its tower in Nousfarer

  11. 2014-Present

    Task-Specific Deep Architectures for Detection and Segmentation

    How can objects be located and segmented precisely in an image?

    Fly to its tower in Nousfarer

  12. 2020-Present

    Vision transformer

    Can attention over image patches replace convolution for visual recognition?

    Fly to its tower in Nousfarer

  13. 2021-Present

    Multimodal Foundation Models

    How can vision and language be learned together in one model?

    Fly to its tower in Nousfarer

How the ideas connect

Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.

PerceptronBlocks worldInverse Optics and Constraint-Based R…Marr's Computational Theory of VisionProbabilistic Graphical Models for Vi…Statistical Appearance Models and Sub…Geometric Feature MatchingHand-Engineered Feature PipelinesConvolutional neural networkDeep Generative ModelsTask-Specific Deep Architectures for …Vision transformerMultimodal Foundation Models

Beyond the field

Where Computer Vision's ideas came from in other fields, and what they led to there: the strongest 12 of 51 links.

A city in Nousfarer: a red-carpeted stair climbing terraces to a crowned tower on an acropolis, more towers round it, forest and snowy mountains beyond.
A city in Nousfarer, the game played in the browser.

Computer Vision is a planet in Nousfarer

Its city has a tower for each of these 13 frameworks. Land, walk in, and climb one concept at a time.

Land on Computer Vision

Related fields

Fields its ideas are linked to

More in Artificial Intelligence

See where Computer Vision sits on the Map of Human Knowledge