- Atlas
- Computer Science
- Deep Learning
Deep Learning
Deep learning is the branch of artificial intelligence that builds machines to learn from raw data using neural networks: layered systems of simple computing units whose connections are adjusted by experience.
13 frameworks, from 1958 to today · Artificial Intelligence · Computer Science
- Perceptron and Early Neural Networks 1958-1969
- Connectionism 1982-1995
- Backpropagation and gradient-based training 1986-Present
- Recurrent neural network 1986-2017
- Convolutional neural network 1989-Present
- Unsupervised Pretraining and Deep Belief Networks 2006-2012
- Deep Learning at Scale 2012-Present
- Deep Reinforcement Learning 2013-Present
- Generative adversarial network 2014-Present
- Transformer 2014-Present
- Graph neural network 2016-Present
- Scaling Laws and Foundation Models 2019-Present
- Diffusion model 2020-Present
Overview
Deep learning is the branch of artificial intelligence that builds machines to learn from raw data using neural networks: layered systems of simple computing units whose connections are adjusted by experience1. What is at stake is representation. Traditional programs need a human to specify which features of the input matter — the edges of a letter, the words that signal spam1. Deep networks instead learn their own internal features, layer by layer, so that early layers capture simple patterns and later layers combine them into abstract concepts1. That idea, that intelligence can emerge from many layers of learned representation, is what makes deep learning both powerful and controversial, and its history is a story of repeated rediscovery, of hardware catching up with theory, and of a few stubborn researchers who kept faith during the decades when the approach was unfashionable.
The Perceptron and the First Neural Networks
The intellectual roots lie in the 1940s. Warren McCulloch and Walter Pitts described an idealized artificial neuron in 1943: a unit that sums weighted inputs and fires if the total crosses a threshold2. This was a logical device, not a learning one, but it established the metaphor of computation as a network of neuron-like units. In 1949 Donald Hebb proposed that learning could occur by strengthening connections between neurons that fire together — a rule still called Hebbian learning3.
The first trainable network came in 1958, when Frank Rosenblatt built the Perceptron at Cornell. It learned to classify patterns by adjusting weights according to errors, and Rosenblatt demonstrated it recognizing simple shapes. The perceptron could only solve problems that are linearly separable — where a straight line (or flat plane) can divide the categories. In 1969 Marvin Minsky and Seymour Papert published Perceptrons, proving that single-layer perceptrons cannot compute certain simple functions, notably the exclusive-or (XOR), and arguing that multi-layer networks would be hard to train. The book is often blamed for freezing neural network research for a decade; in truth funding and interest were already shifting, but its authority made the limitations hard to ignore.
Connectionism and the Hidden-Layer Revival
The revival came from a different direction: psychology. In the 1980s, researchers including David Rumelhart, James McClelland, Geoffrey Hinton, and Terrence Sejnowski developed connectionism — the view that cognition is the parallel activity of many simple units, and that knowledge is stored in the strengths of their connections rather than in symbols. Their 1986 two-volume work Parallel Distributed Processing collected models of memory, language, and perception built this way. The key technical advance was backpropagation: an algorithm that computes how much each weight in a multi-layer network contributed to the final error, then adjusts it14. The idea had appeared earlier in different contexts — Paul Werbos described it in his 1974 doctoral thesis, and it resembles a method from control theory — but Rumelhart, Hinton, and Ronald Williams showed in 1986 that it could train hidden layers, the intermediate layers that give networks their power. This solved the problem Minsky and Papert had raised: a network with a hidden layer can learn XOR.
The 1980s also produced the first convolutional neural networks (CNNs), designed for images1. A convolution applies the same small filter across every position in an image, so the network detects a pattern wherever it appears and needs far fewer weights than a fully connected layer. Kunihiko Fukushima's Neocognitron (1980) introduced a hierarchy of such filters inspired by the visual cortex. Yann LeCun and colleagues, working at Bell Labs, combined convolutional layers with backpropagation and by 1989 had a system that read handwritten zip codes; deployed in the 1990s, it processed a substantial share of US mail. Recurrent neural networks (RNNs), which loop information through time to handle sequences, were also developed in this period, with early work by John Hopfield and others and influential formulations by Jeffrey Elman and Michael Jordan in the late 1980s and early 1990s.
The Winter and the Long Preparation
Despite these successes, the 1990s and early 2000s were lean years for neural networks. Backpropagation-trained networks overfit small datasets, were slow to train, and lacked theoretical guarantees. Support vector machines, which had strong generalization theory behind them, and other statistical methods took the spotlight. Yet the period was not idle. Sepp Hochreiter and Jürgen Schmidhuber's long short-term memory (LSTM, 1997) addressed a central weakness of RNNs — the vanishing gradient, where error signals shrink as they propagate back through many time steps — by adding gated memory cells that can preserve information across long spans. LeCun and others continued refining convolutional architectures. And a small community, including Hinton, Yoshua Bengio, and LeCun, kept working on the core problem: how to train deep networks, meaning networks with many layers, which were known to be hard to optimize.
Deep Belief Networks and Unsupervised Pretraining
The breakthrough came in 2006, when Hinton, Simon Osindero, and Yee-Whye Teh showed that a deep network could be trained layer by layer using unsupervised pretraining1. Each layer was first trained as a restricted Boltzmann machine (RBM), a generative model that learns to represent the probability distribution of its inputs, and the layers were then stacked into a deep belief network15. The pretrained weights gave the network a sensible starting point, after which supervised fine-tuning could refine them. This "greedy layer-wise" approach worked, and it persuaded the field that depth was trainable. Bengio and others extended the idea to autoencoders, networks trained to reconstruct their own input. The pretraining era was short-lived — within a few years, better activation functions, improved initialization, and larger datasets made plain end-to-end training work — but it changed the climate of opinion and gave the field its modern name.
The Deep Learning Boom
The decisive demonstration came in 2012, when Alex Krizhevsky, Ilya Sutskever, and Hinton entered a network called AlexNet in the ImageNet image-recognition competition16. Using convolutional layers trained on graphics processors (GPUs), which had been built for video games but turned out to be ideal for the matrix arithmetic of neural networks, AlexNet cut the error rate dramatically compared with the best hand-engineered methods. The result was widely reported and set off a wave of adoption. Within a few years, deep networks were beating classical methods in speech recognition, machine translation, and game playing.
The 2014 introduction of generative adversarial networks (GANs) by Ian Goodfellow and colleagues showed another face of deep learning: a generator network learns to produce realistic data (images, for instance) while a discriminator network learns to tell real from fake, and the two improve by competing17. GANs produced strikingly realistic images and became a major research direction, though training them proved unstable.
Attention and the Transformer
Sequence modeling with RNNs had a persistent limitation: processing one step at a time made training slow and made it hard to relate distant parts of a sequence. In 2014, Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio introduced attention for machine translation — a mechanism that lets the model weigh different parts of the input when producing each output word, rather than compressing everything into a single fixed vector. In 2017, a team at Google published "Attention Is All You Need," introducing the transformer, an architecture built entirely on attention, dispensing with recurrence89. Its self-attention mechanism lets every position in a sequence attend to every other position in parallel, which made training on large data far more efficient9. The transformer became the dominant architecture for language and beyond.
Scaling and Foundation Models
The transformer made scale the central lever. Researchers found that performance improved predictably as models grew in parameters and training data — a relationship documented in scaling laws by OpenAI researchers in 2020. Large language models such as GPT-2 (2019) and GPT-3 (2020) showed that a single model trained on vast text could perform many tasks from a prompt, without task-specific training. The term foundation model was coined around 2021 for such large pretrained systems, adaptable to many downstream uses. The same period saw deep learning spread into other domains: graph neural networks, which operate on data structured as nodes and edges (molecules, social networks), were formalized in influential work by Thomas Kipf and Max Welling in 2016 and by others, and deep reinforcement learning produced systems that mastered Atari games and the board game Go.
The Present
Deep learning today is the default approach across much of AI. Convolutional networks remain central to vision; transformers dominate language and increasingly vision and other modalities; graph networks handle relational data; and generative models — GANs, diffusion models, and autoregressive language models — produce text, images, and audio. The field's open questions are real: how to make models reliable and interpretable, how to train them without enormous energy cost, how to ensure they generalize beyond their training data, and whether scaling alone will continue to deliver. The history above explains why the field believes it might: each time the approach seemed stuck, a new idea or a new resource — a training trick, a GPU, a dataset, an architecture — reopened the path.
References
- Deep learning (introduction, Overview, Before 1980, 2000s, Deep learning revolution), Wikipedia
- Perceptron (History), Wikipedia
- Machine learning (History), Wikipedia
- Backpropagation, Wikipedia
- Restricted Boltzmann machine, Wikipedia
- AlexNet, Wikipedia
- Generative adversarial network, Wikipedia
- Attention Is All You Need, Wikipedia
- Transformer (deep learning), Wikipedia
The frameworks of Deep Learning
Each framework is a way of working the field has used: when it held, and the question it answers.
- 1958-1969
Perceptron and Early Neural Networks
Can a machine learn to classify patterns from examples?
- 1982-1995
Connectionism
Can cognition emerge from many simple connected units?
About Connectionism
Connectionism is a framework within cognitive science and artificial intelligence that explains mental phenomena as emerging from networks of simple, interconnected units. Rather than viewing the mind as a device that manipulates abstract symbols according to explicit rules, connectionism holds that knowledge and behavior arise from the strengths of connections between these units. In the context of deep learning, connectionism is the intellectual ancestor of modern neural networks: the core idea is that a large number of simple processing elements, working in parallel, can learn to perform complex tasks by adjusting the weights of their interconnections based on experience.The central problem connectionism addresses is the rigidity of classical, rule-based artificial intelligence. Traditional “good old-fashioned AI” (GOFAI) assumed that intelligence could be captured by explicit logical rules and symbolic representations, which required a human expert to hand-code every rule and fact. This approach struggled with tasks that are easy for humans but hard to formalize, such as recognizing a face, understanding spoken language, or generalizing from noisy, incomplete data. Connectionism offers an alternative: instead of programming rules, you provide a learning algorithm and a large amount of data, and the network discovers its own internal representations. This shift from hand-crafted symbols to learned, distributed representations is the framework’s most distinctive organizing idea.Connectionism differs from nearby approaches in several key ways. Unlike symbolic AI, it does not rely on explicit, localist representations where each concept corresponds to a single symbol. Instead, representations are distributed across many units, and a single unit can participate in representing many different concepts. Unlike early perceptrons, which were limited to linearly separable problems, connectionist networks with hidden layers can learn non-linear functions. And unlike purely behaviorist stimulus-response models, connectionism posits internal states and representations, albeit ones that are learned rather than innate. The emphasis is on emergence: complex behavior is not designed directly but emerges from the collective activity of many simple units.To understand connectionism, it is helpful to learn a few essential concepts in a logical order. First, the artificial neuron (or unit) is the basic building block: it receives inputs, multiplies each by a weight, sums them, and applies an activation function to produce an output. Second, a network architecture organizes these units into layers—an input layer, one or more hidden layers, and an output layer. Third, a learning rule specifies how weights are adjusted. The most important is backpropagation, which computes the error at the output and propagates it backward through the network to update weights. Fourth, distributed representation means that a concept is encoded as a pattern of activity across many units, rather than in a single dedicated unit. Finally, graceful degradation refers to the ability of a connectionist system to tolerate damage or noise: because knowledge is spread across many connections, the system fails gradually rather than catastrophically.The intellectual history of connectionism is often described in waves. The first wave began in the 1940s with Warren McCulloch and Walter Pitts, who showed that networks of simplified neurons could compute logical functions. Frank Rosenblatt’s perceptron in 1958 was a major milestone, demonstrating a simple learning rule for single-layer networks. However, Marvin Minsky and Seymour Papert’s 1969 book Perceptrons exposed the limitations of single-layer networks, particularly their inability to solve problems like XOR, which led to a period of reduced funding and interest. The second wave arrived in the mid-1980s with the publication of Parallel Distributed Processing (1986) by David Rumelhart, James McClelland, and the PDP Research Group. This work reintroduced hidden layers and popularized the backpropagation algorithm, showing that multi-layer networks could overcome the earlier limitations. This period, roughly from the mid-1980s to the mid-1990s, is the core era of connectionism as a distinct framework, marked by intense interest in its implications for psychology and neuroscience. The third wave, beginning around 2012, is what we now call deep learning, where much larger networks, more data, and faster hardware led to dramatic breakthroughs in image recognition, speech processing, and language modeling.Major figures in connectionism include McCulloch and Pitts for their foundational neuron model, Rosenblatt for the perceptron, and Rumelhart and McClelland for the PDP movement and backpropagation. John Hopfield contributed the Hopfield network, a recurrent network used for associative memory. Geoffrey Hinton, though more associated with the deep learning revival, was also a key figure in the 1980s connectionist era, working on distributed representations and Boltzmann machines. The continuing significance of connectionism is profound: it provides the theoretical foundation for modern deep learning, and its principles—learning from data, distributed representations, and emergent behavior—remain central to contemporary AI. Moreover, connectionism continues to influence cognitive science, offering a biologically plausible alternative to symbolic models of the mind, and it informs debates about the nature of concepts, language, and reasoning.For a reader approaching connectionism, the practical prerequisites are modest. A basic familiarity with linear algebra (vectors, matrices, dot products) and calculus (derivatives, especially the chain rule) is helpful for understanding how networks learn, but it is not strictly necessary for grasping the conceptual ideas. Some exposure to introductory machine learning or neural networks will make the material easier, but the core ideas—units, weights, learning, distributed representation—can be understood without any specialist background. If you have no prior knowledge, you can still follow the narrative by focusing on the intuitive metaphors: think of a network as a team of many small workers who coordinate by adjusting how strongly they listen to one another, gradually improving their collective performance through trial and error. No deep mathematical training is required to appreciate why connectionism matters, but a willingness to think in terms of patterns and probabilities rather than rules and symbols will serve you well.The 7 concepts its tower teaches
- Artificial Neuron
- Network Architecture
- Learning Rule
- Backpropagation
- Distributed Representation
- Graceful Degradation
- Connectionist Model
- 1986-Present
Backpropagation and gradient-based training
How can a multi-layer network adjust its weights?
- 1986-2017
Recurrent neural network
How can a network remember sequences over time?
- 1989-Present
Convolutional neural network
How can a network learn to see images?
About Convolutional neural network
A convolutional neural network, or CNN, is a type of deep learning architecture designed to process data with a grid-like topology, most famously images. Its core organizing idea is that instead of treating every pixel as an independent input, the network learns a set of small, reusable filters that scan across the input to detect local patterns. This approach is a direct response to the limitations of fully connected networks, which are computationally wasteful and prone to overfitting when applied to high-dimensional data like photographs.The central problem CNNs address is the challenge of learning spatial hierarchies of features from raw pixel values. In a traditional fully connected network, an image of 100×100 pixels would require 10,000 weights per neuron in the first hidden layer, and the network would have no built-in assumption that nearby pixels are related. A CNN, by contrast, uses a small filter—say, 5×5 pixels—that slides across the entire image. This single filter is shared across all locations, meaning only 25 weights are needed for that layer. This weight sharing dramatically reduces the number of parameters and provides a form of regularization that helps prevent overfitting. More importantly, it builds in the inductive bias that a pattern learned in one part of the image is useful everywhere else.The essential concepts of a CNN can be learned in a logical sequence. First is the convolution operation itself, where a filter slides over the input and computes a dot product at each position, producing a feature map. Second is the activation function, typically ReLU (rectified linear unit), which introduces nonlinearity and helps mitigate the vanishing gradient problem that plagued earlier networks. Third is pooling, a downsampling operation that reduces the spatial size of feature maps, making the network more computationally efficient and providing a degree of translation invariance—though it is important to note that most CNNs are not truly invariant to translation because of this downsampling. Fourth is the fully connected classifier at the end of the network, which takes the high-level features learned by the convolutional layers and produces the final output, such as class probabilities. Finally, dropout is a regularization technique that randomly drops neurons during training to prevent co-adaptation and overfitting.The intellectual history of CNNs begins well before their dominant period. The foundational architecture was established by Yann LeCun and colleagues in the late 1980s and 1990s with LeNet, which was successfully applied to handwritten digit recognition for postal mail sorting. However, the framework remained a niche technique for years because training deep networks was slow and datasets were small. The pivotal moment came in 2012, when Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton won the ImageNet Large Scale Visual Recognition Challenge with AlexNet, a deep CNN that dramatically outperformed all traditional computer vision methods. This victory, made possible by the availability of large datasets and GPU computing, ignited the deep learning revolution and established CNNs as the de-facto standard for computer vision. Subsequent influential works in the following years, such as the deeper VGGNet and the ResNet architecture with its skip connections, refined the recipe and pushed performance further.CNNs differ from nearby approaches in several key ways. Compared to recurrent neural networks (RNNs), which process sequential data and maintain a hidden state over time, CNNs process spatial data in a feedforward manner and are better suited for images. RNNs were the dominant framework for sequence modeling before the rise of deep learning, but they struggled with long-range dependencies and were computationally expensive to train. CNNs, with their parallelizable convolution operations, were far more efficient on GPUs. Compared to the attention-based transformer architectures that emerged later, CNNs rely on local receptive fields and weight sharing, whereas transformers use global self-attention mechanisms that can relate distant elements directly. Transformers have since surpassed CNNs on many vision tasks, but CNNs remain widely used, especially in settings where computational efficiency and inductive bias for local structure are valuable.For a reader approaching this topic, the prerequisites are modest. A basic understanding of linear algebra (matrix multiplication) and calculus (derivatives) is helpful, as is familiarity with the general concept of gradient descent and backpropagation. No specialist knowledge of image processing is required, though intuition about how edges, textures, and shapes compose into objects will make the material more intuitive. The core ideas of convolution, pooling, and weight sharing can be grasped with high-school mathematics, and the practical implementation of CNNs is now accessible through modern deep learning libraries.The continuing significance of CNNs lies in their efficiency and their demonstrated effectiveness on structured spatial data. While transformers have become the dominant architecture for large-scale vision models, CNNs remain the backbone of countless deployed systems, from facial recognition and medical image analysis to autonomous driving and industrial inspection. Their architectural principles—local connectivity, weight sharing, and hierarchical feature learning—have also influenced the design of other neural network families. Understanding CNNs is essential not only because they represent a landmark achievement in deep learning, but because they embody the fundamental lesson that building the right inductive biases into a network is often more powerful than simply adding more data and computation.The 11 concepts its tower teaches
- Convolution
- Activation function
- Pooling
- Fully connected layer
- Dropout
- Feature hierarchy learning
- Translation invariance
- LeNet
- AlexNet
- VGGNet
- ResNet
- 2006-2012
Unsupervised Pretraining and Deep Belief Networks
How can deep networks be trained without labelled data?
- 2012-Present
- 2013-Present
- 2014-Present
Generative adversarial network
How can a network generate realistic new data?
- 2014-Present
- 2016-Present
- 2019-Present
Scaling Laws and Foundation Models
How does performance scale with model size and data?
- 2020-Present
How the ideas connect
Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.
Convolutional neural network influenced by Backpropagation and gradient-based training
LeCun and colleagues combined convolutional layers with backpropagation to train a handwritten zip-code recognizer by 1989. Backpropagation supplied the gradient-based training that made the convolutional architecture work end to end.
Carried by Yann LeCun.
Deep Learning at Scale influenced by Backpropagation and gradient-based training
The deep learning boom depended on backpropagation to train large networks end to end. AlexNet and the systems that followed were trained with gradient-based methods descended from the 1986 algorithm.
Carried by Geoffrey Hinton; AlexNet (2012).
Transformer reacted against Recurrent neural network
The transformer was introduced explicitly to overcome the sequential processing of recurrent networks, dispensing with recurrence in favour of attention. 'Attention Is All You Need' (2017) defined itself against the RNN-based sequence models that had dominated.
Carried by Ashish Vaswani; Attention Is All You Need (2017).
Deep Learning at Scale influenced by Convolutional neural network
AlexNet was a convolutional network trained on GPUs, and its 2012 ImageNet victory set off the deep learning boom. The convolutional architecture developed by LeCun and others was the direct basis of the scaled-up system.
Carried by Alex Krizhevsky; Ilya Sutskever; AlexNet (2012).
Scaling Laws and Foundation Models influenced by Transformer
Scaling laws and foundation models were built on the transformer, whose parallel attention made training on vast data efficient. GPT-2 and GPT-3 were transformer language models whose scaling behaviour was documented by OpenAI researchers.
Carried by OpenAI; GPT-3 (2020).
Backpropagation and gradient-based training influenced by Perceptron and Early Neural Networks
The perceptron's error-correcting weight update was the starting point that backpropagation generalized to multiple layers. Rumelhart, Hinton and Williams explicitly presented backpropagation as an extension of the perceptron learning rule to hidden units.
Carried by Frank Rosenblatt; David Rumelhart; Geoffrey Hinton.
Unsupervised Pretraining and Deep Belief Networks revived Connectionism
The deep belief network work of 2006 deliberately revived connectionist ambitions for deep layered networks after the statistical-learning era. Hinton and colleagues kept the connectionist faith that many simple units could learn representations.
Carried by Geoffrey Hinton; Yoshua Bengio.
Recurrent neural network influenced by Backpropagation and gradient-based training
Recurrent networks were trained by backpropagation through time, an extension of the gradient-based method to unfolded sequences. The LSTM of Hochreiter and Schmidhuber (1997) was designed specifically to overcome the vanishing-gradient problem that backpropagation through time suffered.
Carried by Sepp Hochreiter; Jürgen Schmidhuber.
Unsupervised Pretraining and Deep Belief Networks influenced by Backpropagation and gradient-based training
Unsupervised pretraining was proposed as a way to initialize deep networks so that gradient-based fine-tuning could succeed. Hinton, Osindero and Teh's 2006 deep belief network used pretrained weights as a starting point for backpropagation.
Carried by Geoffrey Hinton; Simon Osindero; Yee-Whye Teh.
Deep Reinforcement Learning influenced by Recurrent neural network
Deep reinforcement learning used recurrent networks to represent state and value in partially observed environments. The deep Q-network work of 2013 combined Q-learning with a deep network, and recurrent variants followed for sequential decision problems.
Carried by Volodymyr Mnih; DeepMind.
Deep Learning at Scale influenced by Unsupervised Pretraining and Deep Belief Networks
The pretraining era persuaded the field that deep networks were trainable and gave the approach its modern name, preparing the ground for the 2012 boom. Better initialization and activation functions developed in that period made end-to-end training work at scale.
Carried by Geoffrey Hinton; Yoshua Bengio.
Diffusion model reacted against Generative adversarial network
Diffusion models arose partly as an alternative to generative adversarial networks, avoiding their unstable adversarial training. They became a major generative approach in the years after 2020.
Carried by Jonathan Ho; Denoising Diffusion Probabilistic Models (2020).
Beyond the field
Where Deep Learning's ideas came from in other fields, and what they led to there: the strongest 12 of 63 links.
Perceptron and Early Neural Networks influenced Deep learning in Artificial Intelligence
The perceptron and early neural networks supplied the layered unit architecture and the idea of learning weights from data that deep learning revived and extended with backpropagation and depth.
Carried by Frank Rosenblatt; The Perceptron (1958).
Perceptron and Early Neural Networks influenced by Artificial neural network modeling in Computational Neuroscience
McCulloch and Pitts' 1943 formal neuron and Hebb's 1949 synaptic learning rule supplied the model of a threshold unit and of learning by modifying connection strengths that Rosenblatt built the Perceptron on.
Carried by Warren S. McCulloch; Walter Pitts; Donald O. Hebb.
Perceptron and Early Neural Networks revived by Deep Learning in Data Science
Deep learning revived and scaled the multilayer perceptron tradition of early neural networks, adding many layers, backpropagation and large-scale training after the perceptron's limits had stalled the field.
Carried by Frank Rosenblatt; Geoffrey Hinton.
Backpropagation and gradient-based training influenced by First-Order and Stochastic Gradient Methods in Mathematical Optimization
Backpropagation is an application of first-order gradient descent; the optimisation literature on steepest descent and stochastic approximation provided the algorithm that network trainers adopted.
Carried by Herbert Robbins; Sutton Monro.
Backpropagation and gradient-based training grew out of Perceptron and early neural network learning in Learning And Concept Formation
Backpropagation grew directly out of the perceptron tradition: the perceptron learning rule was generalised to multilayer networks by propagating error backwards through hidden units, overcoming the limitation Minsky and Papert had identified.
Carried by David E. Rumelhart; Geoffrey Hinton; Ronald J. Williams.
Backpropagation and gradient-based training influenced GPU-Accelerated Deep Learning Systems in ML Systems
GPU deep-learning systems accelerate backpropagation and gradient-based training, which remain the core algorithm. The frameworks exist to run that algorithm efficiently on parallel hardware.
Backpropagation and gradient-based training led to Deep learning in Artificial Intelligence
Backpropagation and gradient-based training, popularised in the 1980s, is the training method that deep learning scaled up; deep networks are trained by exactly this procedure.
Carried by David Rumelhart; Geoffrey Hinton; Learning representations by back-propagating errors (1986).
Backpropagation and gradient-based training influenced Function Approximation and Deep Reinforcement Learning in Reinforcement Learning
Backpropagation and gradient-based training is the optimisation engine of deep RL: policy gradients and value-network training are backpropagation applied to RL objectives. The 1986 backpropagation work is a direct ancestor of the gradient methods used in DQN and policy-gradient algorithms.
Carried by David Rumelhart; Geoffrey Hinton; Ronald Williams.
Backpropagation and gradient-based training influenced Graph-Based Deep Learning Frameworks in ML Systems
Graph-based deep learning frameworks train graph neural networks by backpropagation over message-passing computations. The gradient-based training machinery developed in the 1980s is the direct basis for how these frameworks compute and apply gradients.
Recurrent neural network influenced Neural sequence modeling in Natural Language Processing
Neural sequence modeling in NLP was built directly on recurrent neural networks, using LSTM and gated recurrent architectures developed in deep learning for sequence transduction and language modeling.
Carried by Sepp Hochreiter; Jürgen Schmidhuber.
Deep Learning at Scale influenced Machine Learning for Network Management in Networks
Deep learning at scale, especially the 2012 ImageNet breakthrough and subsequent sequence models, provided the architectures and training methods that machine learning for network management applies to traffic and telemetry data.
Carried by Alex Krizhevsky; Geoffrey Hinton.
Deep Learning at Scale influenced by GPU-Accelerated Deep Learning Systems in ML Systems
GPU-accelerated training, notably the two-GPU convolutional network of Krizhevsky, Sutskever and Hinton in 2012, made large-scale deep learning feasible and set off the scale era. The systems work supplied the compute methods the framework depends on.
Carried by AlexNet (2012); Alex Krizhevsky; Ilya Sutskever.

Deep Learning is a planet in Nousfarer
Its city has a tower for each of these 13 frameworks. Land, walk in, and climb one concept at a time.
Land on Deep LearningRelated fields
Fields its ideas are linked to
- Computer Vision 6 links
- Machine Learning 6 links
- ML Systems 6 links
- Artificial Intelligence 5 links
- Computational Neuroscience 5 links
- Natural Language Processing 4 links
- Learning And Concept Formation 4 links
- Statistical Learning 4 links
- Computational Cognitive Science 3 links
- Cognitive Science 2 links
- Data Science 2 links
- Medical Image Analysis 2 links