- Atlas
- Computer Science
- Machine Learning
Machine Learning
Machine learning is the branch of artificial intelligence concerned with programs that improve at a task by extracting patterns from data rather than by following rules written out in advance.
12 frameworks, from 1958 to today · Artificial Intelligence · Computer Science
- Connectionism and Neural Networks 1958-Present
- Instance-Based Learning 1960-2000
- Statistical Learning Theory 1970-Present
- Symbolic Concept Learning 1975-1995
- Bayesian and probabilistic graphical models 1985-2015
- Bayesian inference 1985-Present
- Graphical model 1985-Present
- Regularization and Model Selection 1985-Present
- Inductive Logic Programming 1988-2005
- Ensemble Learning 1990-Present
- Kernel method 1992-2015
- Deep Learning 2012-Present
Overview
Machine learning is the branch of artificial intelligence concerned with programs that improve at a task by extracting patterns from data rather than by following rules written out in advance12. A spam filter that learns from thousands of labelled emails, a system that predicts a house price from past sales, a network that recognizes a face after seeing many faces: these are machine-learning systems1. What is at stake is generalization — the ability to do well on cases never seen during training1. A program that memorizes its training data has learned nothing useful; the whole discipline is organized around the gap between fitting what you have seen and performing on what you have not1. That gap, and the methods invented to close it, give machine learning its history.
Learning machines before learning theory
Early artificial intelligence in the 1950s and 1960s was mostly symbolic: intelligence as manipulation of explicit symbols and rules. But a parallel strand asked whether a machine could be taught instead. In 1943 Warren McCulloch and Walter Pitts had described an idealized neuron — a unit that sums weighted inputs and fires if the sum crosses a threshold — and in 1949 Donald Hebb proposed that learning could consist of strengthening connections between neurons that fire together13. Frank Rosenblatt's perceptron (1958) turned this into a working device: a single layer of threshold units whose weights were adjusted automatically from labelled examples43. The perceptron learning rule had a genuine guarantee, the perceptron convergence theorem, showing that if the data could be separated by a straight boundary, the procedure would find one in finite steps.
The limitation was equally sharp. In 1969 Marvin Minsky and Seymour Papert showed that a single-layer perceptron cannot compute even simple non-linear functions such as exclusive-or, and that stacking layers did not obviously help because no one knew how to train the hidden layers. Their critique helped cool interest in neural networks for over a decade4. Meanwhile other traditions grew: in the late 1950s Arthur Samuel's checkers program improved its play by adjusting weights through self-play, and in the 1960s work on nearest-neighbour classification — assigning a new case the label of the most similar stored examples — established instance-based learning, which needs no explicit model at all, only a notion of similarity and a store of past cases.
Symbolic concept learning and the rise of induction
Through the 1970s and 1980s, the dominant machine-learning work was symbolic1. The problem was concept learning: given positive and negative examples of a category, find a description that covers the positives and excludes the negatives5. Tom Mitchell's candidate-elimination algorithm (version space, early 1980s) maintained the set of all hypotheses consistent with the data seen so far6. Ross Quinlan's ID3 (1979) and its successor C4.5 built decision trees by choosing, at each node, the attribute that best separated the classes — measured by information gain — producing rules a person could read. These systems made learning concrete and inspectable, and they introduced ideas that outlived the symbolic era: that learning is search through a hypothesis space, and that a preference for simpler hypotheses (Occam's razor) helps choose among consistent ones7.
Inductive logic programming, developed from the late 1980s (notably by Stephen Muggleton and colleagues), pushed the same idea into first-order logic: learning general logical rules from examples and background knowledge, so that a system could infer, say, a recursive family relation from facts about parents18. It was expressive and elegant but computationally costly and hard to scale.
Statistics enters: risk, regularization, and the bias–variance tradeoff
The decisive shift came from statistics. Instead of asking whether a hypothesis is logically consistent with the data, the statistical view asks how well it will perform on future data drawn from the same distribution1. This is empirical risk minimization: choose the model that minimizes average loss on the training sample, in the hope that sample performance tracks true performance19. The hope is not automatic. Overfitting — fitting noise in the training data — is the central failure mode, and the remedies are regularization: penalties on model complexity, early stopping, or restricting the hypothesis class110.
The bias–variance tradeoff names the tension in plain terms1. A very flexible model has low bias (it can represent the truth) but high variance (it changes wildly with the particular sample); a rigid model is the reverse. Neither extreme generalizes well; the art is finding the middle. Cross-validation — repeatedly holding out part of the data to estimate future performance — became the standard practical tool for that choice. Vladimir Vapnik and Alexey Chervonenkis's work from the 1960s and 1970s, later called statistical learning theory, gave this intuition a rigorous foundation through measures of a model class's capacity (the VC dimension) and bounds on generalization error1112. The field's vocabulary — training versus test error, generalization, capacity — dates from this convergence, crystallized in the 1980s and 1990s.
Connectionism revived: backpropagation and neural networks
The obstacle Minsky and Papert had identified was training multilayer networks. The key idea — backpropagation of error — computes, for each weight in a layered network, how much it contributed to the output error, by propagating the error backward through the layers using the chain rule of calculus13. The idea had multiple independent origins; its widespread adoption followed a 1986 paper by David Rumelhart, Geoffrey Hinton, and Ronald Williams, which showed that hidden layers could learn useful internal representations. This was connectionism: intelligence as distributed activity in networks of simple units, with knowledge stored in weights rather than rules4. The 1986 Parallel Distributed Processing volumes, edited by Rumelhart and James McClelland, made the case at book length.
The practical results were striking. Yann LeCun's convolutional networks, trained by backpropagation in the late 1980s and 1990s, read handwritten zip codes and later a large share of handwritten checks in the United States. Yet through the 1990s neural networks were often outperformed on small datasets and were hard to train at depth; their moment would come later.
Margins, kernels, and ensembles
In the 1990s the statistical strand produced its most influential practical method. The support vector machine, developed by Vapnik and colleagues (Corinna Cortes and Vapnik, 1995), does not just find any separating boundary but the one with the widest margin — the largest buffer between the boundary and the nearest examples of each class12. Theory suggested wide margins generalize well. Kernel methods extended this to non-linear problems without explicitly computing high-dimensional features: a kernel function measures similarity between examples, and the algorithm works entirely through those similarities1214. Kernels also powered kernel PCA and Gaussian processes, a Bayesian approach to regression and classification.
Ensemble learning took a different route to accuracy: combine many weak models15. Bagging (Leo Breiman, 1996) trains models on bootstrap resamples and averages them, reducing variance; boosting (Yoav Freund and Robert Schapire, from the early 1990s) trains models sequentially, each focusing on the errors of the last, and Freund and Schapire's AdaBoost (1997) became a benchmark-beating workhorse. Breiman's random forests (2001) combined bagging with random feature selection and were robust and easy to use. These methods, along with support vector machines, dominated applied machine learning around 2000.
Probabilistic models and the Bayesian turn
A separate tradition treated learning as probabilistic inference16. Bayesian networks, developed by Judea Pearl in the 1980s, represent dependencies among variables as a directed graph and allow efficient inference and learning from data117. Graphical models more broadly — including hidden Markov models, which powered speech recognition and bioinformatics through the 1990s — gave a principled language for uncertainty. The Bayesian view treats learning as updating a prior distribution over hypotheses in light of data, yielding a posterior; this supplies automatic regularization and a natural account of uncertainty. Expectation-maximization (Dempster, Laird, and Rubin, 1977) made it practical to fit models with hidden variables. These ideas coexisted with, and often competed against, the margin-based and neural approaches, each claiming better use of limited data or better-calibrated uncertainty.
Deep learning
The turning point came when the obstacles to deep networks — insufficient data, insufficient compute, poor initialization, and the vanishing-gradient problem (error signals shrinking as they propagate back through many layers) — were overcome one by one. In 2006 Hinton and colleagues showed that deep belief networks could be pretrained layer by layer, reviving interest. The decisive demonstration was in image recognition: in 2012, Alex Krizhevsky, Ilya Sutskever, and Hinton's deep convolutional network AlexNet won the ImageNet competition by a large margin, trained on graphics processors. Depth plus scale plus data proved far more effective than hand-engineered features.
From there the field moved quickly: better activation functions (ReLU), dropout regularization, batch normalization, and then the transformer architecture (Vaswani and colleagues, 2017), whose attention mechanism allowed models to weigh relationships across long sequences and train at enormous scale. Deep learning absorbed the earlier lessons — backpropagation and gradient descent from connectionism, regularization and cross-validation from statistics, convolutional structure from vision — and became the default approach across vision, speech, and language.
Where the field stands
Today machine learning is dominated by deep neural networks trained by gradient descent on large datasets, with the earlier traditions still present and often combined: ensembles of trees remain strong on tabular data; kernel methods and Gaussian processes persist where data are scarce and uncertainty matters; probabilistic graphical models underlie much of causal inference and structured prediction; statistical learning theory supplies the vocabulary of generalization. The open questions are the old ones in sharper form: why do heavily overparameterized networks generalize at all, when classical theory says they should not? How can models be made robust to distribution shift, to adversarial inputs, and to the biases in their training data? How much data and compute is enough, and what are the limits of scaling? The discipline that began by asking how a machine could learn a straight boundary now asks how to trust systems with billions of parameters — but the question at its center, how to do well on what you have never seen, has not changed1.
References
- Machine learning (introduction, Supervised learning, Theory, History, Artificial intelligence, Association rules, Statistics, Overfitting, Bayesian networks), Wikipedia
- Outline of machine learning, Wikipedia
- Perceptron (History, introduction), Wikipedia
- Connectionism, Wikipedia
- Concept learning, Wikipedia
- Version space learning (introduction, Historical background), Wikipedia
- Occam's razor, Wikipedia
- Inductive logic programming (introduction, History), Wikipedia
- Empirical risk minimization, Wikipedia
- Regularization (mathematics), Wikipedia
- Vapnik–Chervonenkis theory, Wikipedia
- Support vector machine (introduction, History), Wikipedia
- Backpropagation, Wikipedia
- Kernel method, Wikipedia
- Ensemble learning, Wikipedia
- Bayesian inference, Wikipedia
- Bayesian network (introduction, History), Wikipedia
The frameworks of Machine Learning
Each framework is a way of working the field has used: when it held, and the question it answers.
- 1958-Present
Connectionism and Neural Networks
How can networks of simple units learn?
- 1960-2000
- 1970-Present
Statistical Learning Theory
When will a model generalize to new data?
About Statistical Learning Theory
Statistical learning theory is the mathematical backbone of machine learning: a framework for asking, in precise terms, how a system that learns from a finite set of examples can be expected to perform on data it has never seen. Where much of machine learning is driven by engineering intuition — build a model, train it, hope it generalizes — statistical learning theory insists on formal answers. It treats learning not as a craft but as a statistical inference problem: given data, find a function that predicts well, and prove that it will keep predicting well.The problem the framework addresses is deceptively simple. A learner observes a training set of input-output pairs, drawn from some unknown probability distribution, and must select a hypothesis function that maps new inputs to correct outputs. The catch is that the learner never sees the true distribution — only a finite sample from it. Any function that fits the training data perfectly might fail completely on new data, a phenomenon known as overfitting. Statistical learning theory provides the vocabulary and the theorems for understanding when a function that works on the training set will also work in general, and for quantifying the price of complexity.The Core Organizing Idea: Empirical Risk MinimizationThe central construct of statistical learning theory is the distinction between two quantities. The first is the expected risk (or true risk): the average loss a hypothesis incurs over the entire unknown data distribution. The second is the empirical risk: the average loss over the training sample. The ideal learner would minimize expected risk, but since the distribution is unknown, it can only minimize empirical risk. This strategy — pick the function with the lowest error on the training data — is called empirical risk minimization (ERM).The entire framework hangs on one question: how close is the empirical risk to the expected risk? If the gap is small, then minimizing on the training set genuinely improves performance on future data. If the gap is large, the learner is fooling itself. Statistical learning theory's signature contribution is a family of results bounding this gap in terms of the complexity of the hypothesis space — the set of functions the learner is allowed to choose from.Essential Concepts in Learning OrderTo understand the framework, the concepts build on one another in a natural sequence.The learning setup. A learning problem consists of an input space, an output space, a loss function measuring prediction error, and a hypothesis space of candidate functions. The learner receives a training sample and must select a hypothesis. This formalization, standard across the field, is itself a contribution of statistical learning theory: it forces clarity about what is being learned and what counts as success.Expected risk and empirical risk. As described above, these two quantities define the gap that the theory seeks to control. The expected risk is the true measure of quality; the empirical risk is the only one the learner can compute.Overfitting and the bias-variance tradeoff. A hypothesis space that is too small cannot capture the true pattern (high bias); one that is too large can memorize the training noise (high variance). The theory explains this tradeoff quantitatively: larger hypothesis spaces make it easier to fit the training data but harder to guarantee that the fit means anything.Uniform convergence and the VC dimension. The key insight of Vapnik and Chervonenkis in the 1960s and 1970s was that the gap between empirical and expected risk can be bounded uniformly over the entire hypothesis space. If the space is not too complex, then with high probability, every hypothesis has empirical risk close to its expected risk — and therefore the one with the lowest empirical risk is genuinely good. The measure of complexity they introduced is the Vapnik-Chervonenkis (VC) dimension: the size of the largest set of points that the hypothesis space can shatter, or label in all possible ways. A finite VC dimension yields concrete bounds on the sample size needed to guarantee generalization.Structural risk minimization. Rather than fixing a single hypothesis space, one can nest a sequence of spaces of increasing complexity and choose the one that optimally balances empirical error against a complexity penalty. This is structural risk minimization (SRM), a principled alternative to the ad hoc model selection that dominates much of applied machine learning.Rademacher complexity. A later refinement, Rademacher complexity measures the capacity of a hypothesis space by its ability to fit random noise. It often yields tighter bounds than the VC dimension and extends naturally to more general settings, including regression and kernel methods.Regularization. In practice, the complexity penalties suggested by the theory take the form of regularization terms added to the training objective — penalizing large weights, for example. Statistical learning theory provides the justification for why such penalties work: they shrink the effective hypothesis space and thereby tighten the generalization bounds.How It Differs from Nearby ApproachesStatistical learning theory is often confused with the broader field of machine learning, but its stance is distinctive. Where Bayesian machine learning treats model parameters as random variables and reasons about posterior distributions, statistical learning theory is frequentist: it makes no prior assumptions about the data-generating process and instead asks what can be guaranteed for any distribution. Where deep learning is driven by large-scale empirical experimentation and architectural innovation, statistical learning theory demands proofs and worst-case guarantees. Where symbolic machine learning seeks interpretable rules and logical structures, statistical learning theory works with continuous functions and geometric notions of complexity.The framework is also distinct from computational learning theory, which focuses on the computational difficulty of learning; statistical learning theory focuses instead on the statistical difficulty — how much data is needed, and how well a learner can be expected to perform. The two are complementary, but they ask different questions.Intellectual HistoryThe roots of statistical learning theory lie in classical statistics, particularly in the theory of empirical processes and uniform convergence of averages. But the framework as such emerged in the Soviet Union in the 1960s, when Vladimir Vapnik and Alexey Chervonenkis began developing what became known as VC theory. Their work, published in Russian in the late 1960s and early 1970s, established the fundamental bounds connecting the capacity of a hypothesis space to the reliability of empirical risk minimization.The framework remained largely unknown in the West until the 1990s, when Vapnik, by then at AT&T Bell Laboratories, developed the support vector machine (SVM) as a practical embodiment of the theory. The SVM's success on real-world problems — handwriting recognition, text classification, bioinformatics — brought statistical learning theory to the center of machine learning. Vapnik's 1995 book The Nature of Statistical Learning Theory and his 1998 Statistical Learning Theory codified the framework and made it accessible to a generation of researchers. The theory also influenced the development of kernel methods, which embed data into high-dimensional feature spaces where linear hypotheses become powerful, while the theory's complexity measures keep the effective capacity under control.Continuing SignificanceStatistical learning theory remains the intellectual foundation of machine learning, even as the field has moved toward deep neural networks whose theoretical analysis is far more difficult. The framework's concepts — generalization bounds, capacity control, the bias-variance tradeoff — are the vocabulary in which researchers discuss why models succeed or fail. Modern developments such as the analysis of overparameterized networks, the double-descent phenomenon, and the theory of transfer learning all engage with the questions that Vapnik and Chervonenkis first posed.The framework also has a practical legacy. The principle that model complexity must be controlled, whether through structural risk minimization, regularization, or cross-validation, is now standard practice in every branch of machine learning. The SVM and kernel methods remain widely used in domains where interpretability and theoretical guarantees matter more than raw predictive power.PrerequisitesA working understanding of statistical learning theory requires a modest background in probability theory — random variables, expectation, concentration inequalities such as Hoeffding's inequality — and some familiarity with basic statistics, particularly the notion of estimation error. Linear algebra and basic calculus are helpful for following the derivations. No prior knowledge of machine learning is strictly necessary, though it helps to have seen a simple learning algorithm such as linear regression or the perceptron to ground the abstract concepts. For a reader with these foundations, the core ideas of the framework can be grasped without specialized mathematical training; the proofs, however, are another matter and reward a deeper background in empirical process theory and functional analysis.The 11 concepts its tower teaches
- Learning Setup
- Expected Risk
- Empirical Risk
- Empirical Risk Minimization
- Overfitting
- Bias-Variance Tradeoff
- Uniform Convergence
- VC Dimension
- Structural Risk Minimization
- Rademacher Complexity
- Regularization
- 1975-1995
- 1985-2015
Bayesian and probabilistic graphical models
How can graphs represent uncertainty among variables?
- 1985-Present
- 1985-Present
- 1985-Present
- 1988-2005
- 1990-Present
- 1992-2015
- 2012-Present
Deep Learning
How can deep networks learn useful representations?
About Deep Learning
Deep learning addresses a foundational challenge in machine learning: how can a model learn representations of data at multiple levels of abstraction, from raw input to high-level concepts, without relying on hand-engineered features? Unlike earlier frameworks that often required significant domain expertise to design input features, deep learning aims to discover these features automatically through a hierarchy of layers. This approach became dominant after 2006 when researchers found effective ways to train deep neural networks, but its roots extend to earlier connectionist ideas.The core curriculum for deep learning builds from the basic building block of the artificial neuron to the architecture of layered networks and the procedures for training them. A first course typically covers these concepts in a logical sequence:- Artificial Neuron and Activation Functions: The fundamental computational unit, inspired by biological neurons. Common activation functions like ReLU (Rectified Linear Unit) and sigmoid introduce non-linearity, enabling networks to learn complex patterns.
- Feedforward Neural Network: A basic architecture where information flows from the input layer, through one or more hidden layers, to the output layer without cycles. This structure forms the template for more specialized architectures.
- Backpropagation: The core algorithm for training feedforward networks. It efficiently computes the gradient of the loss function with respect to every weight by propagating errors backward from the output to the input, using the chain rule of calculus.
- Stochastic Gradient Descent and Loss Functions: Optimization procedures that iteratively update network weights to minimize a loss function (like cross-entropy for classification). Stochastic gradient descent uses a random subset of training data per update, making training feasible on large datasets.
- Convolutional Neural Network (CNN) and Pooling: A specialized architecture designed for grid-like data such as images. Convolutional layers learn local, spatially invariant features (e.g., edges, textures), and pooling layers down-sample representations, reducing dimensionality and providing translation invariance.
- Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM): Architectures for sequential data such as text or time series. RNNs maintain a hidden state that acts as a memory. LSTMs introduce gating mechanisms to learn long-range dependencies, mitigating the vanishing gradient problem.
- Regularization (dropout, weight decay, batch normalization): Techniques to combat overfitting. Dropout randomly turns off neurons during training, weight decay penalizes large weights, and batch normalization stabilizes and accelerates training by normalizing layer inputs.
Turning to intellectual history, the story of deep learning is one of repeated cycles of promise, setback, and eventual triumph, with the period from 2006 to the present marking its sustained dominance. The foundational idea of an artificial neuron dates to Warren McCulloch and Walter Pitts in 1943, and Frank Rosenblatt's perceptron in 1958 showed that simple neural networks could learn. However, Marvin Minsky and Seymour Papert's 1969 book Perceptrons exposed the perceptron's severe limitations (notably its inability to learn the XOR function), leading to an "AI winter" for connectionist research.A revival began in the 1980s with the Connectionism framework. Key figures like David Rumelhart, Geoffrey Hinton, and Ronald Williams popularized backpropagation in 1986, showing that multi-layer networks could overcome the XOR problem. Yet, training deep networks remained difficult due to vanishing gradients and limited data and compute. Shallow architectures (e.g., support vector machines from Kernel Methods) often outperformed neural networks for two decades.The decisive breakthrough came in 2006, marking the start of the current deep learning era. Geoffrey Hinton, Simon Osindero, and Yee-Whye Teh published "A Fast Learning Algorithm for Deep Belief Nets," showing that greedy layer-wise pretraining of a deep generative model could initialize weights effectively, enabling successful training of deep networks. This paper is widely credited with launching the modern deep learning wave.Subsequent rapid progress followed. In 2009, Hinton and his students demonstrated that deep neural networks trained with GPUs could dramatically improve speech recognition, a practical triumph that industry adopted. From 2010 onward, key figures and seminal works built the modern curriculum. Yann LeCun's work on convolutional networks (e.g., 1998's LeNet-5) gained widespread attention when Alex Kirzhevsky, Ilya Sutskever, and Hinton's 2012 work "ImageNet Classification with Deep Convolutional Networks" (AlexNet) won the ImageNet Large Scale Visual Recognition Challenge, drastically reducing error rates. This catalyzed the explosion of deep learning in computer vision. For sequence modeling, Hochreiter and Schmidhuber's 1997 LSTM paper became canonical, and later works like Sutskever et al.'s 2014 "Sequence to Sequence Learning with Neural Networks" enabled machine translation.Deep learning's success is also tied to large-scale resources. The Statistical Learning framework, which dominated from 1995, coexisted with deep learning, providing theoretical foundations (like bias-variance tradeoffs) that informed regularization practices. Deep learning also absorbed tools from Probabilistic Machine Learning (e.g., using dropout as approximate Bayesian inference). The framework's relationship with connectionism is direct: deep learning is connectionism after 2006, revitalized by deeper architectures, big data, and powerful hardware.Deep learning transformed machine learning by making feature engineering unnecessary for many tasks, enabling end-to-end learning from raw data. Its legacy is a reorientation of the field toward representation learning and scalable gradient-based optimization.The 14 concepts its tower teaches
- Activation Functions
- Artificial Neuron
- Backpropagation
- Batch Normalization
- Convolutional Neural Network
- Dropout
- Feedforward Neural Network
- Long Short-Term Memory
- Loss Functions
- Pooling
- Recurrent Neural Network
- Regularization
- Stochastic Gradient Descent
- Weight Decay
How the ideas connect
Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.
Deep Learning grew out of Connectionism and Neural Networks
Deep learning is the continuation of the connectionist programme: multilayer networks trained by backpropagation, scaled up with more data and compute. The 1986 backpropagation work and LeCun's convolutional networks are its direct ancestors.
Carried by David Rumelhart; Geoffrey Hinton; Yann LeCun.
Kernel method grew out of Statistical Learning Theory
The support vector machine grew directly out of Vapnik's statistical learning theory, using margin and capacity bounds to justify the widest-margin classifier. Cortes and Vapnik's 1995 paper is the canonical statement.
Carried by Vladimir Vapnik; Corinna Cortes; Support Vector Networks (1995).
Inductive Logic Programming grew out of Symbolic Concept Learning
Inductive logic programming extended symbolic concept learning from attribute-value hypotheses to first-order logic. Muggleton and colleagues took the search-through-hypothesis-space framing of concept learning and made the hypotheses logical clauses.
Carried by Stephen Muggleton.
Regularization and Model Selection influenced by Statistical Learning Theory
Statistical learning theory supplied the vocabulary of capacity, generalization error and the bias-variance tradeoff that regularization and model selection formalized. Cross-validation and complexity penalties were justified by VC-style bounds.
Carried by Vladimir Vapnik; Alexey Chervonenkis.
Bayesian and probabilistic graphical models rival of Connectionism and Neural Networks
Through the late 1980s and 1990s connectionist networks and probabilistic graphical models competed as accounts of learning under uncertainty. Each claimed better use of limited data or better-calibrated uncertainty.
Deep Learning influenced by Regularization and Model Selection
Deep learning adopted regularization techniques such as weight decay, dropout and early stopping, and model selection by validation, from the earlier statistical toolkit. These practices were carried into training deep networks.
Deep Learning influenced by Ensemble Learning
Ensemble ideas persisted into the deep-learning era, for example in dropout as implicit model averaging and in boosted or bagged deep networks. The combination of many weak learners informed later architectures.
Beyond the field
Where Machine Learning's ideas came from in other fields, and what they led to there: the strongest 12 of 58 links.
Connectionism and Neural Networks influenced Connectionism in Mental Representation
The connectionist revival of the 1980s drew directly on neural-network models in machine learning, especially backpropagation and the perceptron tradition. Rumelhart, Hinton and Williams's learning procedure became the engine of the PDP programme in cognitive science.
Carried by David Rumelhart; Geoffrey Hinton; Parallel Distributed Processing (1986).
Connectionism and Neural Networks influenced Connectionist and emergentist modelling in Neurolinguistics
Connectionist and emergentist modelling of language uses the neural-network methods of connectionism: distributed units, weighted connections and learning by error correction. The language models are applications of that machinery.
Statistical Learning Theory influenced Machine learning in Philosophy Of Artificial Intelligence
Statistical learning theory, especially VC theory and the bias-variance framework, supplied the generalization concepts that modern machine learning uses to justify and diagnose models. Vapnik's work is a direct ancestor of the field's theoretical apparatus.
Carried by Vladimir Vapnik.
Statistical Learning Theory influenced Kernel Methods and Support Vector Machines in Statistical Learning
The SVM framework was derived from statistical learning theory, particularly the VC theory of generalization developed from the 1970s onward. Vapnik's theoretical work on uniform convergence and capacity control directly produced the SVM.
Carried by Vladimir Vapnik.
Bayesian and probabilistic graphical models led to Probabilistic programming and automatic differentiation in Statistical Computing
Probabilistic programming grew directly out of Bayesian and probabilistic graphical models, generalizing graphical model specification into a programming language. Probabilistic programs compile to graphical model structures and use inference methods developed for them.
Carried by Bishop (2006).
Bayesian and probabilistic graphical models influenced Statistical Relational Learning in Probabilistic Ai
Statistical relational learning extends probabilistic graphical models to relational and first-order domains, as in relational Markov networks and Markov logic networks. The graphical-model framework supplies the probabilistic semantics that SRL lifts to relational structures.
Carried by Pedro Domingos; Lise Getoor; Ben Taskar.
Bayesian inference led to Approximate Inference in Probabilistic Ai
Approximate inference arose as the set of techniques for making Bayesian inference tractable when exact posteriors are unavailable, via variational Bayes, MCMC and loopy belief propagation. It is a branch of Bayesian inference rather than a rival.
Carried by Michael I. Jordan; Martin Wainwright; Radford Neal.
Bayesian inference influenced Bayesian Deep Learning in Probabilistic Ai
Bayesian deep learning applies Bayesian inference methods, especially variational inference and Markov chain Monte Carlo, to the weights and predictions of neural networks. The variational autoencoder and Bayes-by-backpropagation work of the mid-2010s took the machinery of variational Bayes and applied it to deep networks.
Carried by Variational Autoencoders (2013); Bayes by Backprop (2015).
Bayesian inference influenced by Bayesian probability in Probability Theory
Modern Bayesian inference in machine learning uses the probability calculus of Bayesian probability: prior distributions updated by likelihood to posteriors, as formulated by Bayes and Laplace. The whole computational framework is an application of that probability theory.
Carried by Thomas Bayes; Pierre-Simon Laplace.
Deep Learning led to Diffusion model in Deep Learning
Diffusion models are a branch of deep learning: the denoising networks are trained by backpropagation on large datasets, and the field grew directly out of deep learning practice. The score-based and DDPM formulations were built with the same architectures and optimisation methods as other deep networks.
Carried by Jonathan Ho; Denoising Diffusion Probabilistic Models (2020).
Deep Learning led to Scaling Laws and Foundation Models in Deep Learning
Scaling laws describe how the performance of deep networks improves with data, parameters and compute, and foundation models are large deep networks. The framework is a direct outgrowth of machine-learning deep learning.
Deep Learning influenced Deep learning as cognitive modelling in Computational Cognitive Science
Deep learning as cognitive modelling takes its networks, training algorithms and representational findings directly from machine-learning deep learning, especially the backpropagation-trained convolutional networks of the 2010s.
Carried by AlexNet (2012); Geoffrey Hinton; Yann LeCun.

Machine Learning is a planet in Nousfarer
Its city has a tower for each of these 12 frameworks. Land, walk in, and climb one concept at a time.
Land on Machine LearningRelated fields
Fields its ideas are linked to
- Deep Learning 6 links
- Statistical Learning 6 links
- Probabilistic Ai 5 links
- Statistical Inference 5 links
- Artificial Intelligence 2 links
- Computer Vision 2 links
- Computational Cognitive Science 2 links
- ML Systems 2 links
- Probability Theory 2 links
- Networks 2 links
- Models And Representation 2 links
- Philosophy Of Artificial Intelligence 2 links
More in Artificial Intelligence
- Ai Ethics And Fairness
- Ai Safety
- Artificial Intelligence
- Computer Vision
- Deep Learning
- Evolutionary Computation
- Knowledge Representation Reasoning
- Multiagent Systems
- Natural Language Processing
- Planning And Search
- Probabilistic Ai
- Reinforcement Learning
- Speech Recognition And Synthesis
See where Machine Learning sits on the Map of Human Knowledge