- Atlas
- Computer Science
- Computer Vision
Computer Vision
Computer vision is the branch of artificial intelligence concerned with getting machines to extract meaning from images and video: to say what is where, to recognize a face or a road sign, to reconstruct a scene in three dimensions, or to generate a picture from a sentence.
13 frameworks, from 1958 to today · Artificial Intelligence · Computer Science
- Perceptron 1958-1969
- Blocks world 1963-1980
- Inverse Optics and Constraint-Based Reconstruction 1970-1990
- Marr's Computational Theory of Vision 1982-1995
- Probabilistic Graphical Models for Vision 1985-2012
- Statistical Appearance Models and Subspace Learning 1991-2010
- Geometric Feature Matching 1999-2012
- Hand-Engineered Feature Pipelines 1999-2014
- Convolutional neural network 2012-Present
- Deep Generative Models 2014-Present
- Task-Specific Deep Architectures for Detection and Segmentation 2014-Present
- Vision transformer 2020-Present
- Multimodal Foundation Models 2021-Present
Overview
Computer vision is the branch of artificial intelligence concerned with getting machines to extract meaning from images and video: to say what is where, to recognize a face or a road sign, to reconstruct a scene in three dimensions, or to generate a picture from a sentence1. What is at stake is both practical and philosophical. Practically, cameras are everywhere, and most of the visual information in the world is never seen by a human eye. Philosophically, vision is the clearest test of whether perception can be understood as computation — whether seeing can be broken into steps a machine can carry out. The field's history is a long argument about what those steps are, and about how much of the answer must be learned from data rather than designed by hand.
Early Machines, Edges, and the Perceptron
Work on machine perception began in the 1950s and 1960s alongside the wider project of artificial intelligence. Early systems were small and hand-crafted. In 1959, the neurophysiologists David Hubel and Torsten Wiesel, recording from the visual cortex of cats, found neurons that responded to oriented edges and to moving bars — evidence that early vision decomposes the image into local features. That finding shaped how engineers thought about the problem for decades.
The perceptron, introduced by Frank Rosenblatt in 1958, was a simple learning machine: a layer of threshold units whose weights were adjusted from labelled examples2. It could be trained to classify simple patterns, and it was demonstrated on a machine built to recognize shapes on a photoelectric retina. In 1969 Marvin Minsky and Seymour Papert published Perceptrons, proving that a single-layer perceptron cannot compute certain functions, such as exclusive-or, and arguing that the approach was fundamentally limited2. The critique was mathematically correct for one layer, but its influence cooled neural-network research for years. The deeper problem it exposed — that useful vision needs internal representations, not just a mapping from pixels to labels — set the agenda for what followed.
Blocks World and the Geometry of Scenes
Through the 1960s and 1970s, a dominant research style treated vision as a problem of reconstructing a scene's geometry from an image, using explicit models of the world. The canonical testbed was the blocks world: a tabletop of simple polyhedra, lit and photographed, whose edges and faces a program had to recover. At MIT, work associated with Larry Roberts (his 1963 doctoral thesis on machine perception of three-dimensional solids) and later with David Marr and others built systems that detected edges, grouped them into lines, and fitted them to models of blocks and wedges.
These systems embodied a belief: that the world is simple enough to describe with hand-built geometric primitives, and that vision is largely a matter of matching image evidence to those descriptions. They worked on clean, controlled scenes and failed on anything messier. The lesson practitioners drew was that the hard part is not the geometry of a perfect block but the ambiguity of real images — shadows, texture, occlusion, and noise.
Reconstruction, Inverse Optics, and Marr's Program
A second strand asked how a machine could recover three-dimensional structure from two-dimensional images using the physics of image formation. This was sometimes called inverse optics: if you know how light, surfaces, and cameras interact, you can invert the process to infer depth and shape. Work on stereo (matching points between two views), shape from shading, shape from texture, and optical flow (the apparent motion of image points between frames) flourished in the 1970s and 1980s. Berthold Horn's work on shape from shading and on optical flow, and the development of the constraint style of reasoning — adding assumptions until an underdetermined problem becomes solvable — were central.
The most influential synthesis was David Marr's3. In Vision (published in 1982, after his death), Marr argued that vision should be studied at three levels: the computational level (what problem is being solved and why), the algorithmic level (what representations and procedures carry it out), and the implementational level (how it is realized in hardware or neurons). He proposed that early vision produces a primal sketch of edges and blobs, then a 2½-D sketch of visible surfaces oriented in depth, and finally a 3-D model of objects. Marr's framework gave the field a shared vocabulary and a research program: find the representations, then the algorithms. Its emphasis on explicit, staged representations was later challenged, but its insistence on asking what a computation is for remains a standard way to think about perception.
Statistical Appearance and the Turn to Learning
By the late 1980s, hand-built geometric models were hitting a wall on real images, and attention shifted to statistical methods that learn appearance from examples. A landmark was eigenfaces, introduced by Matthew Turk and Alex Pentland in 1991: face images were represented as combinations of a small set of basis images computed from a training set by principal component analysis, and faces were recognized by their coordinates in that space4. The idea — that a class of objects occupies a low-dimensional subspace of pixel space — generalized widely.
Probabilistic graphical models gave the field a language for uncertainty1. Hidden Markov models were applied to gesture and speech; later, conditional random fields and other structured models handled segmentation and labelling. The shift was methodological as much as technical: instead of asking what rule a human would write, researchers asked what distribution the data came from and how to estimate it. This was the beginning of vision as a statistical learning problem.
Features, Matching, and Hand-Engineered Pipelines
The 1990s and 2000s were the era of hand-engineered features: descriptors designed by researchers to capture useful image structure, paired with classifiers trained on labelled data. David Lowe's SIFT (scale-invariant feature transform, published 1999–2004) detected distinctive keypoints and described each by a histogram of local gradient orientations, making matching robust to changes in scale and rotation5. Similar detectors and descriptors — Harris corners, SURF, and others — made geometric feature matching practical: find corresponding points between two images, then estimate the transformation between them67. This powered panorama stitching, object recognition, and the structure-from-motion pipelines that reconstruct 3D scenes from many photographs.
For recognition, the bag-of-visual-words model treated image patches like words in a document, and support vector machines classified the resulting histograms. The histogram of oriented gradients (HOG) descriptor, combined with a sliding-window classifier, gave strong pedestrian detection8. The PASCAL VOC challenges, running from 2005, standardized datasets and metrics and drove steady progress. The pattern of this era was a pipeline: detect features, describe them, pool them, classify. Each stage was designed by a human, and each stage's design was a research contribution.
Deep Convolutional Networks
The decisive break came from neural networks, revived by advances in learning algorithms and computing power. A convolutional neural network (CNN) is a layered network that applies small learned filters across an image, sharing weights across positions — a design inspired by the receptive fields Hubel and Wiesel had found9. Yann LeCun and colleagues demonstrated in the late 1980s and 1990s that CNNs trained by backpropagation could read handwritten digits, and their LeNet system was deployed commercially for cheque reading. But for years CNNs were beaten on harder benchmarks by hand-engineered pipelines, and the field remained skeptical.
The turning point was the ImageNet challenge10. ImageNet, assembled by Fei-Fei Li and colleagues from 2007, was a dataset of millions of labelled images across a thousand categories. In 2012, Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton entered AlexNet, a deep CNN trained on graphics processors, and cut the top-5 error rate dramatically below the best hand-engineered entries10. The result convinced much of the field. The lesson was that features could be learned end-to-end from data, and that scale — in data and computation — mattered enormously. Within a few years, nearly every vision benchmark was led by deep networks, and the hand-engineered pipeline gave way to learned representations.
Detection, Segmentation, and the Architecture Boom
Deep learning then spread across vision's subproblems, each generating its own architectures. Object detection — locating and labelling multiple objects in an image — moved from region-proposal methods such as R-CNN (2014) and its faster successors to single-stage detectors such as YOLO (2015) and SSD, which predicted boxes and classes in one pass. Semantic segmentation — labelling every pixel — was transformed by fully convolutional networks (2015), which replaced the classifier head with convolutional layers that output a spatial map. Instance segmentation combined detection and pixel labelling, notably in Mask R-CNN (2017)11. These architectures shared a pattern: deep convolutional backbones, multi-scale feature pyramids, and task-specific heads, trained on large labelled datasets.
The period also brought generative models into vision. Generative adversarial networks, introduced by Ian Goodfellow and colleagues in 2014, trained a generator to fool a discriminator, producing increasingly realistic images12. Variational autoencoders and, later, diffusion models (which generate images by learning to reverse a gradual noising process) extended the field's reach from recognition to synthesis1314. By the late 2010s, a system could not only label an image but produce a plausible new one.
Transformers, Multimodal Models, and the Present
The most recent shift imported an architecture from natural language processing. The transformer, introduced in 2017, processes a sequence of tokens by attention: each token weighs how much to draw on every other token, rather than convolving over a fixed local neighbourhood15. In 2020, the Vision Transformer showed that an image could be cut into patches, treated as a sequence of tokens, and classified competitively — and, with enough data, better than CNNs. Attention's flexibility, and its ability to relate distant parts of an image, made it the backbone of a new generation.
The same years saw vision become multimodal. Models such as CLIP (2021) were trained on hundreds of millions of image–text pairs to place images and their captions in a shared embedding space, so that a text description could retrieve or classify images without task-specific training. This "foundation model" approach — pretrain broadly, adapt to many tasks — now underlies image generation from text, visual question answering, and open-vocabulary detection. The boundary between computer vision and language modelling has blurred, and many current systems are trained jointly on both.
The questions still open are recognizably the old ones in new clothes. How much can be learned from data versus designed? How do models generalize to situations unlike their training data — new viewpoints, lighting, or objects? How can a system explain why it produced a label, and how can it be made robust to adversarial or unusual inputs? And the deepest question, inherited from Marr, remains: what is the right level at which to describe what a seeing machine is actually doing3? The field's history is a record of answers that worked for a while, each replaced when the world turned out to be messier than the model assumed.
References
- Computer vision (introduction, Definition, History), Wikipedia
- Perceptrons (book) (Background, introduction), Wikipedia
- David Marr (scientist), Wikipedia
- Eigenface (introduction, History), Wikipedia
- Scale-invariant feature transform, Wikipedia
- Speeded up robust features, Wikipedia
- Harris corner detector, Wikipedia
- Histogram of oriented gradients, Wikipedia
- Convolutional neural network, Wikipedia
- AlexNet, Wikipedia
- Region-based convolutional neural network (History), Wikipedia
- Generative adversarial network, Wikipedia
- Variational autoencoder, Wikipedia
- Diffusion model, Wikipedia
- Transformer (deep learning), Wikipedia
The frameworks of Computer Vision
Each framework is a way of working the field has used: when it held, and the question it answers.
- 1958-1969
Perceptron
Can a simple network learn to classify patterns from examples?
- 1963-1980
Blocks world
How can a program recover the geometry of simple objects from an image?
- 1970-1990
Inverse Optics and Constraint-Based Reconstruction
How can depth and shape be inferred from the physics of image formation?
- 1982-1995
Marr's Computational Theory of Vision
What are the computational stages that turn an image into a description?
- 1985-2012
Probabilistic Graphical Models for Vision
How should uncertainty be represented and inferred in visual interpretation?
- 1991-2010
Statistical Appearance Models and Subspace Learning
Can object appearance be captured by a low-dimensional learned subspace?
- 1999-2012
Geometric Feature Matching
How can corresponding points be found between different views of a scene?
- 1999-2014
Hand-Engineered Feature Pipelines
Which hand-designed features and classifiers best recognize objects?
- 2012-Present
Convolutional neural network
Can useful visual features be learned end-to-end from raw pixels?
- 2014-Present
Deep Generative Models
How can a model learn to generate realistic images from data?
- 2014-Present
Task-Specific Deep Architectures for Detection and Segmentation
How can objects be located and segmented precisely in an image?
- 2020-Present
Vision transformer
Can attention over image patches replace convolution for visual recognition?
- 2021-Present
Multimodal Foundation Models
How can vision and language be learned together in one model?
How the ideas connect
Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.
Convolutional neural network revived Perceptron
The perceptron's layered threshold units and learning-by-weight-adjustment were the direct ancestor of backpropagation-trained multilayer networks. LeCun's convolutional networks of the late 1980s and 1990s, and later AlexNet in 2012, deliberately revived the connectionist program that Minsky and Papert's critique had cooled.
Carried by Frank Rosenblatt; Yann LeCun; AlexNet (2012).
Marr's Computational Theory of Vision grew out of Inverse Optics and Constraint-Based Reconstruction
Marr's computational theory grew directly out of the inverse-optics and constraint-based reconstruction work of the 1970s. He synthesized stereo, shape-from-shading and optical-flow research into a staged program with the primal sketch, 2½-D sketch and 3-D model, giving the reconstruction approach its canonical formulation.
Carried by David Marr; Vision (1982); Berthold Horn.
Convolutional neural network replaced Hand-Engineered Feature Pipelines
AlexNet's 2012 ImageNet win showed that features learned end-to-end from data beat hand-designed descriptors like SIFT and HOG. Within a few years the detect-describe-pool-classify pipeline was replaced by deep convolutional networks across vision benchmarks.
Carried by AlexNet (2012); Alex Krizhevsky; ImageNet (2009).
Task-Specific Deep Architectures for Detection and Segmentation grew out of Convolutional neural network
Task-specific detection and segmentation architectures were built directly on deep convolutional backbones. R-CNN, Fast R-CNN, Faster R-CNN and YOLO replaced hand-engineered region proposals and sliding-window classifiers with learned features and end-to-end training.
Carried by R-CNN (2014).
Inverse Optics and Constraint-Based Reconstruction replaced Blocks world
Blocks-world systems reconstructed clean polyhedral scenes by matching image edges to hand-built geometric models. When these failed on real images, the field turned to inverse-optics methods that recovered depth and surface orientation from physical image formation rather than from object models.
Carried by Larry Roberts; David Marr.
Probabilistic Graphical Models for Vision reacted against Marr's Computational Theory of Vision
Probabilistic graphical models reframed vision as inference over distributions estimated from data, rejecting Marr's insistence on hand-specified explicit representations and staged algorithms. Researchers asked what distribution generated the image rather than what rule a human would write.
Statistical Appearance Models and Subspace Learning replaced Marr's Computational Theory of Vision
Statistical appearance models such as eigenfaces abandoned Marr's hand-built geometric descriptions in favour of low-dimensional subspaces learned from example images. The shift was a deliberate move away from designed representations toward learning appearance from data.
Carried by Matthew Turk; Alex Pentland; Eigenfaces (1991).
Deep Generative Models grew out of Convolutional neural network
Deep generative models such as variational autoencoders and generative adversarial networks grew out of deep convolutional networks, using learned layered representations as the basis for image synthesis. They extended the deep-learning program from recognition to generation.
Carried by Ian Goodfellow; Generative Adversarial Networks (2014).
Multimodal Foundation Models influenced by Deep Generative Models
Multimodal foundation models such as CLIP and image generators built on deep generative and contrastive representation learning to align images with text. They extended generative and embedding methods to joint vision-language pretraining at scale.
Carried by CLIP (2021).
Vision transformer influenced by Task-Specific Deep Architectures for Detection and Segmentation
The Vision Transformer adapted the transformer architecture, which had already been applied to detection and segmentation tasks, to image recognition by treating image patches as tokens. It carried over the deep-learning training recipes and benchmark practices of the detection and segmentation era.
Carried by An Image is Worth 16x16 Words (2020).
Multimodal Foundation Models grew out of Vision transformer
Multimodal foundation models adopted the transformer architecture that the Vision Transformer had brought to vision. Models like CLIP and Flamingo combined vision transformers with language transformers for joint image-text pretraining.
Carried by CLIP (2021); Flamingo (2022).
Hand-Engineered Feature Pipelines influenced by Statistical Appearance Models and Subspace Learning
The idea that object classes occupy low-dimensional appearance subspaces fed into hand-engineered recognition pipelines. Bag-of-visual-words and HOG-plus-SVM systems pooled local descriptors into compact representations classified by learned models, extending the statistical-appearance program with richer features.
Beyond the field
Where Computer Vision's ideas came from in other fields, and what they led to there: the strongest 12 of 51 links.
Inverse Optics and Constraint-Based Reconstruction influenced by Regularization Theory in Inverse Problems
Constraint-based reconstruction in vision applied Tikhonov-style regularization theory to make ill-posed inverse problems solvable. Poggio and Torre explicitly imported regularization from the mathematics of inverse problems into computational vision.
Carried by Tomaso Poggio; Regularization Theory (1943).
Probabilistic Graphical Models for Vision influenced by Graphical model in Multivariate Statistics
Graphical models for vision were built directly on the multivariate-statistics formalism of conditional independence graphs and Markov random fields. Vision researchers imported that machinery for image and scene modelling.
Statistical Appearance Models and Subspace Learning influenced by Unsupervised Learning and Dimension Reduction in Statistical Learning
Statistical appearance models such as active shape and appearance models built directly on principal component analysis and classical dimension reduction. Cootes and Taylor used PCA over training shapes and textures to obtain compact subspaces.
Carried by Timothy Cootes; Active Shape Models (1995).
Statistical Appearance Models and Subspace Learning influenced Statistical Shape and Appearance Models in Medical Image Analysis
Statistical appearance models and subspace learning (eigenfaces, PCA appearance models) provided the direct methodological template for Active Appearance Models and statistical shape/appearance models in medical imaging. Cootes and Taylor's framework generalized these subspace techniques to medical anatomy.
Carried by Timothy F. Cootes; Christopher J. Taylor.
Convolutional neural network led to Deep learning in Artificial Intelligence
The 2012 convolutional networks were the event that launched modern deep learning: AlexNet's ImageNet result demonstrated that deep layered networks trained on large data and GPUs worked, and the broader deep-learning programme grew out of that success.
Carried by Alex Krizhevsky; Ilya Sutskever; Geoffrey Hinton.
Convolutional neural network led to Deep Learning in Machine Learning
Convolutional networks are the branch of machine-learning deep learning that made the paradigm dominant; the 2012 ImageNet result is the standard origin story of the deep-learning surge in machine learning.
Carried by Alex Krizhevsky; Geoffrey Hinton.
Deep Generative Models led to Diffusion model in Deep Learning
Diffusion models were developed within and against the deep generative modelling programme, using neural networks as the score or noise-prediction function. They generalised earlier deep generative approaches such as VAEs and GANs by replacing adversarial or variational training with a denoising objective.
Carried by Jonathan Ho; Denoising Diffusion Probabilistic Models (2020); Yang Song.
Vision transformer grew out of Transformer in Deep Learning
The vision transformer is a direct adaptation of the transformer architecture: it keeps self-attention blocks and replaces token sequences with image patches, so the vision model is a branch grown from the general transformer.
Carried by Alexey Dosovitskiy.
Vision transformer influenced by Transformer and pre-training paradigm in Natural Language Processing
The transformer and pre-training paradigm from natural language processing supplied both the self-attention architecture and the pretrain-then-finetune recipe that the vision transformer adopted for images.
Multimodal Foundation Models influenced by Multimodal language modeling in Natural Language Processing
Multimodal foundation models such as CLIP and the image-text models that followed paired a vision encoder with a large language model trained on web-scale text. The transformer language-modeling methods and pretraining recipes developed in multimodal language modeling were carried directly into these vision-language systems.
Carried by CLIP (2021); Alec Radford.
Perceptron influenced Connectionism in Philosophy Of Artificial Intelligence
Rosenblatt's perceptron became a central exhibit for philosophical connectionism, which treats distributed network learning as a model of mind. The perceptron supplied the concrete learning mechanism connectionists cited from the 1960s onward.
Carried by Frank Rosenblatt; Perceptron (1958).
Perceptron influenced Convolutional neural network in Deep Learning
The perceptron's weighted-sum-of-inputs unit and its layered pattern-recognition design were carried into convolutional networks, which replaced full connectivity with local receptive fields and shared weights.
Carried by Frank Rosenblatt; Yann LeCun.

Computer Vision is a planet in Nousfarer
Its city has a tower for each of these 13 frameworks. Land, walk in, and climb one concept at a time.
Land on Computer VisionRelated fields
Fields its ideas are linked to
- Deep Learning 6 links
- Computational Imaging 4 links
- Medical Image Analysis 4 links
- Computational Neuroscience 4 links
- Cognitive Science 3 links
- Perception 3 links
- ML Systems 3 links
- Statistical Learning 3 links
- Artificial Intelligence 2 links
- Machine Learning 2 links
- Natural Language Processing 2 links
- Computational Cognitive Science 2 links
More in Artificial Intelligence
- Ai Ethics And Fairness
- Ai Safety
- Artificial Intelligence
- Deep Learning
- Evolutionary Computation
- Knowledge Representation Reasoning
- Machine Learning
- Multiagent Systems
- Natural Language Processing
- Planning And Search
- Probabilistic Ai
- Reinforcement Learning
- Speech Recognition And Synthesis
See where Computer Vision sits on the Map of Human Knowledge