Deep Learning

Deep learning is the branch of artificial intelligence that builds machines to learn from raw data using neural networks: layered systems of simple computing units whose connections are adjusted by experience.

13 frameworks, from 1958 to today · Artificial Intelligence · Computer Science

still a working view ended · each leads to its entry below

Overview

Deep learning is the branch of artificial intelligence that builds machines to learn from raw data using neural networks: layered systems of simple computing units whose connections are adjusted by experience1. What is at stake is representation. Traditional programs need a human to specify which features of the input matter — the edges of a letter, the words that signal spam1. Deep networks instead learn their own internal features, layer by layer, so that early layers capture simple patterns and later layers combine them into abstract concepts1. That idea, that intelligence can emerge from many layers of learned representation, is what makes deep learning both powerful and controversial, and its history is a story of repeated rediscovery, of hardware catching up with theory, and of a few stubborn researchers who kept faith during the decades when the approach was unfashionable.

The Perceptron and the First Neural Networks

The intellectual roots lie in the 1940s. Warren McCulloch and Walter Pitts described an idealized artificial neuron in 1943: a unit that sums weighted inputs and fires if the total crosses a threshold2. This was a logical device, not a learning one, but it established the metaphor of computation as a network of neuron-like units. In 1949 Donald Hebb proposed that learning could occur by strengthening connections between neurons that fire together — a rule still called Hebbian learning3.

The first trainable network came in 1958, when Frank Rosenblatt built the Perceptron at Cornell. It learned to classify patterns by adjusting weights according to errors, and Rosenblatt demonstrated it recognizing simple shapes. The perceptron could only solve problems that are linearly separable — where a straight line (or flat plane) can divide the categories. In 1969 Marvin Minsky and Seymour Papert published Perceptrons, proving that single-layer perceptrons cannot compute certain simple functions, notably the exclusive-or (XOR), and arguing that multi-layer networks would be hard to train. The book is often blamed for freezing neural network research for a decade; in truth funding and interest were already shifting, but its authority made the limitations hard to ignore.

Connectionism and the Hidden-Layer Revival

The revival came from a different direction: psychology. In the 1980s, researchers including David Rumelhart, James McClelland, Geoffrey Hinton, and Terrence Sejnowski developed connectionism — the view that cognition is the parallel activity of many simple units, and that knowledge is stored in the strengths of their connections rather than in symbols. Their 1986 two-volume work Parallel Distributed Processing collected models of memory, language, and perception built this way. The key technical advance was backpropagation: an algorithm that computes how much each weight in a multi-layer network contributed to the final error, then adjusts it14. The idea had appeared earlier in different contexts — Paul Werbos described it in his 1974 doctoral thesis, and it resembles a method from control theory — but Rumelhart, Hinton, and Ronald Williams showed in 1986 that it could train hidden layers, the intermediate layers that give networks their power. This solved the problem Minsky and Papert had raised: a network with a hidden layer can learn XOR.

The 1980s also produced the first convolutional neural networks (CNNs), designed for images1. A convolution applies the same small filter across every position in an image, so the network detects a pattern wherever it appears and needs far fewer weights than a fully connected layer. Kunihiko Fukushima's Neocognitron (1980) introduced a hierarchy of such filters inspired by the visual cortex. Yann LeCun and colleagues, working at Bell Labs, combined convolutional layers with backpropagation and by 1989 had a system that read handwritten zip codes; deployed in the 1990s, it processed a substantial share of US mail. Recurrent neural networks (RNNs), which loop information through time to handle sequences, were also developed in this period, with early work by John Hopfield and others and influential formulations by Jeffrey Elman and Michael Jordan in the late 1980s and early 1990s.

The Winter and the Long Preparation

Despite these successes, the 1990s and early 2000s were lean years for neural networks. Backpropagation-trained networks overfit small datasets, were slow to train, and lacked theoretical guarantees. Support vector machines, which had strong generalization theory behind them, and other statistical methods took the spotlight. Yet the period was not idle. Sepp Hochreiter and Jürgen Schmidhuber's long short-term memory (LSTM, 1997) addressed a central weakness of RNNs — the vanishing gradient, where error signals shrink as they propagate back through many time steps — by adding gated memory cells that can preserve information across long spans. LeCun and others continued refining convolutional architectures. And a small community, including Hinton, Yoshua Bengio, and LeCun, kept working on the core problem: how to train deep networks, meaning networks with many layers, which were known to be hard to optimize.

Deep Belief Networks and Unsupervised Pretraining

The breakthrough came in 2006, when Hinton, Simon Osindero, and Yee-Whye Teh showed that a deep network could be trained layer by layer using unsupervised pretraining1. Each layer was first trained as a restricted Boltzmann machine (RBM), a generative model that learns to represent the probability distribution of its inputs, and the layers were then stacked into a deep belief network15. The pretrained weights gave the network a sensible starting point, after which supervised fine-tuning could refine them. This "greedy layer-wise" approach worked, and it persuaded the field that depth was trainable. Bengio and others extended the idea to autoencoders, networks trained to reconstruct their own input. The pretraining era was short-lived — within a few years, better activation functions, improved initialization, and larger datasets made plain end-to-end training work — but it changed the climate of opinion and gave the field its modern name.

The Deep Learning Boom

The decisive demonstration came in 2012, when Alex Krizhevsky, Ilya Sutskever, and Hinton entered a network called AlexNet in the ImageNet image-recognition competition16. Using convolutional layers trained on graphics processors (GPUs), which had been built for video games but turned out to be ideal for the matrix arithmetic of neural networks, AlexNet cut the error rate dramatically compared with the best hand-engineered methods. The result was widely reported and set off a wave of adoption. Within a few years, deep networks were beating classical methods in speech recognition, machine translation, and game playing.

The 2014 introduction of generative adversarial networks (GANs) by Ian Goodfellow and colleagues showed another face of deep learning: a generator network learns to produce realistic data (images, for instance) while a discriminator network learns to tell real from fake, and the two improve by competing17. GANs produced strikingly realistic images and became a major research direction, though training them proved unstable.

Attention and the Transformer

Sequence modeling with RNNs had a persistent limitation: processing one step at a time made training slow and made it hard to relate distant parts of a sequence. In 2014, Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio introduced attention for machine translation — a mechanism that lets the model weigh different parts of the input when producing each output word, rather than compressing everything into a single fixed vector. In 2017, a team at Google published "Attention Is All You Need," introducing the transformer, an architecture built entirely on attention, dispensing with recurrence89. Its self-attention mechanism lets every position in a sequence attend to every other position in parallel, which made training on large data far more efficient9. The transformer became the dominant architecture for language and beyond.

Scaling and Foundation Models

The transformer made scale the central lever. Researchers found that performance improved predictably as models grew in parameters and training data — a relationship documented in scaling laws by OpenAI researchers in 2020. Large language models such as GPT-2 (2019) and GPT-3 (2020) showed that a single model trained on vast text could perform many tasks from a prompt, without task-specific training. The term foundation model was coined around 2021 for such large pretrained systems, adaptable to many downstream uses. The same period saw deep learning spread into other domains: graph neural networks, which operate on data structured as nodes and edges (molecules, social networks), were formalized in influential work by Thomas Kipf and Max Welling in 2016 and by others, and deep reinforcement learning produced systems that mastered Atari games and the board game Go.

The Present

Deep learning today is the default approach across much of AI. Convolutional networks remain central to vision; transformers dominate language and increasingly vision and other modalities; graph networks handle relational data; and generative models — GANs, diffusion models, and autoregressive language models — produce text, images, and audio. The field's open questions are real: how to make models reliable and interpretable, how to train them without enormous energy cost, how to ensure they generalize beyond their training data, and whether scaling alone will continue to deliver. The history above explains why the field believes it might: each time the approach seemed stuck, a new idea or a new resource — a training trick, a GPU, a dataset, an architecture — reopened the path.

References

  1. Deep learning (introduction, Overview, Before 1980, 2000s, Deep learning revolution), Wikipedia
  2. Perceptron (History), Wikipedia
  3. Machine learning (History), Wikipedia
  4. Backpropagation, Wikipedia
  5. Restricted Boltzmann machine, Wikipedia
  6. AlexNet, Wikipedia
  7. Generative adversarial network, Wikipedia
  8. Attention Is All You Need, Wikipedia
  9. Transformer (deep learning), Wikipedia

The frameworks of Deep Learning

Each framework is a way of working the field has used: when it held, and the question it answers.

  1. 1958-1969

    Perceptron and Early Neural Networks

    Can a machine learn to classify patterns from examples?

    Fly to its tower in Nousfarer

  2. 1982-1995

    Connectionism

    Can cognition emerge from many simple connected units?

    About Connectionism
    Connectionism is a framework within cognitive science and artificial intelligence that explains mental phenomena as emerging from networks of simple, interconnected units. Rather than viewing the mind as a device that manipulates abstract symbols according to explicit rules, connectionism holds that knowledge and behavior arise from the strengths of connections between these units. In the context of deep learning, connectionism is the intellectual ancestor of modern neural networks: the core idea is that a large number of simple processing elements, working in parallel, can learn to perform complex tasks by adjusting the weights of their interconnections based on experience.
    The central problem connectionism addresses is the rigidity of classical, rule-based artificial intelligence. Traditional “good old-fashioned AI” (GOFAI) assumed that intelligence could be captured by explicit logical rules and symbolic representations, which required a human expert to hand-code every rule and fact. This approach struggled with tasks that are easy for humans but hard to formalize, such as recognizing a face, understanding spoken language, or generalizing from noisy, incomplete data. Connectionism offers an alternative: instead of programming rules, you provide a learning algorithm and a large amount of data, and the network discovers its own internal representations. This shift from hand-crafted symbols to learned, distributed representations is the framework’s most distinctive organizing idea.
    Connectionism differs from nearby approaches in several key ways. Unlike symbolic AI, it does not rely on explicit, localist representations where each concept corresponds to a single symbol. Instead, representations are distributed across many units, and a single unit can participate in representing many different concepts. Unlike early perceptrons, which were limited to linearly separable problems, connectionist networks with hidden layers can learn non-linear functions. And unlike purely behaviorist stimulus-response models, connectionism posits internal states and representations, albeit ones that are learned rather than innate. The emphasis is on emergence: complex behavior is not designed directly but emerges from the collective activity of many simple units.
    To understand connectionism, it is helpful to learn a few essential concepts in a logical order. First, the artificial neuron (or unit) is the basic building block: it receives inputs, multiplies each by a weight, sums them, and applies an activation function to produce an output. Second, a network architecture organizes these units into layers—an input layer, one or more hidden layers, and an output layer. Third, a learning rule specifies how weights are adjusted. The most important is backpropagation, which computes the error at the output and propagates it backward through the network to update weights. Fourth, distributed representation means that a concept is encoded as a pattern of activity across many units, rather than in a single dedicated unit. Finally, graceful degradation refers to the ability of a connectionist system to tolerate damage or noise: because knowledge is spread across many connections, the system fails gradually rather than catastrophically.
    The intellectual history of connectionism is often described in waves. The first wave began in the 1940s with Warren McCulloch and Walter Pitts, who showed that networks of simplified neurons could compute logical functions. Frank Rosenblatt’s perceptron in 1958 was a major milestone, demonstrating a simple learning rule for single-layer networks. However, Marvin Minsky and Seymour Papert’s 1969 book Perceptrons exposed the limitations of single-layer networks, particularly their inability to solve problems like XOR, which led to a period of reduced funding and interest. The second wave arrived in the mid-1980s with the publication of Parallel Distributed Processing (1986) by David Rumelhart, James McClelland, and the PDP Research Group. This work reintroduced hidden layers and popularized the backpropagation algorithm, showing that multi-layer networks could overcome the earlier limitations. This period, roughly from the mid-1980s to the mid-1990s, is the core era of connectionism as a distinct framework, marked by intense interest in its implications for psychology and neuroscience. The third wave, beginning around 2012, is what we now call deep learning, where much larger networks, more data, and faster hardware led to dramatic breakthroughs in image recognition, speech processing, and language modeling.
    Major figures in connectionism include McCulloch and Pitts for their foundational neuron model, Rosenblatt for the perceptron, and Rumelhart and McClelland for the PDP movement and backpropagation. John Hopfield contributed the Hopfield network, a recurrent network used for associative memory. Geoffrey Hinton, though more associated with the deep learning revival, was also a key figure in the 1980s connectionist era, working on distributed representations and Boltzmann machines. The continuing significance of connectionism is profound: it provides the theoretical foundation for modern deep learning, and its principles—learning from data, distributed representations, and emergent behavior—remain central to contemporary AI. Moreover, connectionism continues to influence cognitive science, offering a biologically plausible alternative to symbolic models of the mind, and it informs debates about the nature of concepts, language, and reasoning.
    For a reader approaching connectionism, the practical prerequisites are modest. A basic familiarity with linear algebra (vectors, matrices, dot products) and calculus (derivatives, especially the chain rule) is helpful for understanding how networks learn, but it is not strictly necessary for grasping the conceptual ideas. Some exposure to introductory machine learning or neural networks will make the material easier, but the core ideas—units, weights, learning, distributed representation—can be understood without any specialist background. If you have no prior knowledge, you can still follow the narrative by focusing on the intuitive metaphors: think of a network as a team of many small workers who coordinate by adjusting how strongly they listen to one another, gradually improving their collective performance through trial and error. No deep mathematical training is required to appreciate why connectionism matters, but a willingness to think in terms of patterns and probabilities rather than rules and symbols will serve you well.
    The 7 concepts its tower teaches
    • Artificial Neuron
    • Network Architecture
    • Learning Rule
    • Backpropagation
    • Distributed Representation
    • Graceful Degradation
    • Connectionist Model

    Fly to its tower in Nousfarer

  3. 1986-Present

    Backpropagation and gradient-based training

    How can a multi-layer network adjust its weights?

    Fly to its tower in Nousfarer

  4. 1986-2017

    Recurrent neural network

    How can a network remember sequences over time?

    Fly to its tower in Nousfarer

  5. 1989-Present

    Convolutional neural network

    How can a network learn to see images?

    About Convolutional neural network
    A convolutional neural network, or CNN, is a type of deep learning architecture designed to process data with a grid-like topology, most famously images. Its core organizing idea is that instead of treating every pixel as an independent input, the network learns a set of small, reusable filters that scan across the input to detect local patterns. This approach is a direct response to the limitations of fully connected networks, which are computationally wasteful and prone to overfitting when applied to high-dimensional data like photographs.
    The central problem CNNs address is the challenge of learning spatial hierarchies of features from raw pixel values. In a traditional fully connected network, an image of 100×100 pixels would require 10,000 weights per neuron in the first hidden layer, and the network would have no built-in assumption that nearby pixels are related. A CNN, by contrast, uses a small filter—say, 5×5 pixels—that slides across the entire image. This single filter is shared across all locations, meaning only 25 weights are needed for that layer. This weight sharing dramatically reduces the number of parameters and provides a form of regularization that helps prevent overfitting. More importantly, it builds in the inductive bias that a pattern learned in one part of the image is useful everywhere else.
    The essential concepts of a CNN can be learned in a logical sequence. First is the convolution operation itself, where a filter slides over the input and computes a dot product at each position, producing a feature map. Second is the activation function, typically ReLU (rectified linear unit), which introduces nonlinearity and helps mitigate the vanishing gradient problem that plagued earlier networks. Third is pooling, a downsampling operation that reduces the spatial size of feature maps, making the network more computationally efficient and providing a degree of translation invariance—though it is important to note that most CNNs are not truly invariant to translation because of this downsampling. Fourth is the fully connected classifier at the end of the network, which takes the high-level features learned by the convolutional layers and produces the final output, such as class probabilities. Finally, dropout is a regularization technique that randomly drops neurons during training to prevent co-adaptation and overfitting.
    The intellectual history of CNNs begins well before their dominant period. The foundational architecture was established by Yann LeCun and colleagues in the late 1980s and 1990s with LeNet, which was successfully applied to handwritten digit recognition for postal mail sorting. However, the framework remained a niche technique for years because training deep networks was slow and datasets were small. The pivotal moment came in 2012, when Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton won the ImageNet Large Scale Visual Recognition Challenge with AlexNet, a deep CNN that dramatically outperformed all traditional computer vision methods. This victory, made possible by the availability of large datasets and GPU computing, ignited the deep learning revolution and established CNNs as the de-facto standard for computer vision. Subsequent influential works in the following years, such as the deeper VGGNet and the ResNet architecture with its skip connections, refined the recipe and pushed performance further.
    CNNs differ from nearby approaches in several key ways. Compared to recurrent neural networks (RNNs), which process sequential data and maintain a hidden state over time, CNNs process spatial data in a feedforward manner and are better suited for images. RNNs were the dominant framework for sequence modeling before the rise of deep learning, but they struggled with long-range dependencies and were computationally expensive to train. CNNs, with their parallelizable convolution operations, were far more efficient on GPUs. Compared to the attention-based transformer architectures that emerged later, CNNs rely on local receptive fields and weight sharing, whereas transformers use global self-attention mechanisms that can relate distant elements directly. Transformers have since surpassed CNNs on many vision tasks, but CNNs remain widely used, especially in settings where computational efficiency and inductive bias for local structure are valuable.
    For a reader approaching this topic, the prerequisites are modest. A basic understanding of linear algebra (matrix multiplication) and calculus (derivatives) is helpful, as is familiarity with the general concept of gradient descent and backpropagation. No specialist knowledge of image processing is required, though intuition about how edges, textures, and shapes compose into objects will make the material more intuitive. The core ideas of convolution, pooling, and weight sharing can be grasped with high-school mathematics, and the practical implementation of CNNs is now accessible through modern deep learning libraries.
    The continuing significance of CNNs lies in their efficiency and their demonstrated effectiveness on structured spatial data. While transformers have become the dominant architecture for large-scale vision models, CNNs remain the backbone of countless deployed systems, from facial recognition and medical image analysis to autonomous driving and industrial inspection. Their architectural principles—local connectivity, weight sharing, and hierarchical feature learning—have also influenced the design of other neural network families. Understanding CNNs is essential not only because they represent a landmark achievement in deep learning, but because they embody the fundamental lesson that building the right inductive biases into a network is often more powerful than simply adding more data and computation.
    The 11 concepts its tower teaches
    • Convolution
    • Activation function
    • Pooling
    • Fully connected layer
    • Dropout
    • Feature hierarchy learning
    • Translation invariance
    • LeNet
    • AlexNet
    • VGGNet
    • ResNet

    Fly to its tower in Nousfarer

  6. 2006-2012

    Unsupervised Pretraining and Deep Belief Networks

    How can deep networks be trained without labelled data?

    Fly to its tower in Nousfarer

  7. 2012-Present

    Deep Learning at Scale

    What happens when neural networks are scaled up?

    Fly to its tower in Nousfarer

  8. 2013-Present

    Deep Reinforcement Learning

    How can an agent learn to act from reward?

    Fly to its tower in Nousfarer

  9. 2014-Present

    Generative adversarial network

    How can a network generate realistic new data?

    Fly to its tower in Nousfarer

  10. 2014-Present

    Transformer

    Can attention alone model sequences without recurrence?

    Fly to its tower in Nousfarer

  11. 2016-Present

    Graph neural network

    How can networks learn from graph-structured data?

    Fly to its tower in Nousfarer

  12. 2019-Present

    Scaling Laws and Foundation Models

    How does performance scale with model size and data?

    Fly to its tower in Nousfarer

  13. 2020-Present

    Diffusion model

    How can a network generate data by reversing noise?

    Fly to its tower in Nousfarer

How the ideas connect

Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.

Perceptron and Early Neural NetworksConnectionismBackpropagation and gradient-based tr…Recurrent neural networkConvolutional neural networkUnsupervised Pretraining and Deep Bel…Deep Learning at ScaleDeep Reinforcement LearningGenerative adversarial networkTransformerGraph neural networkScaling Laws and Foundation ModelsDiffusion model

Beyond the field

Where Deep Learning's ideas came from in other fields, and what they led to there: the strongest 12 of 63 links.

A city in Nousfarer: a red-carpeted stair climbing terraces to a crowned tower on an acropolis, more towers round it, forest and snowy mountains beyond.
A city in Nousfarer, the game played in the browser.

Deep Learning is a planet in Nousfarer

Its city has a tower for each of these 13 frameworks. Land, walk in, and climb one concept at a time.

Land on Deep Learning

Related fields

Fields its ideas are linked to

More in Artificial Intelligence

See where Deep Learning sits on the Map of Human Knowledge