Machine Learning

Machine learning is the branch of artificial intelligence concerned with programs that improve at a task by extracting patterns from data rather than by following rules written out in advance.

12 frameworks, from 1958 to today · Artificial Intelligence · Computer Science

still a working view ended · each leads to its entry below

Overview

Machine learning is the branch of artificial intelligence concerned with programs that improve at a task by extracting patterns from data rather than by following rules written out in advance12. A spam filter that learns from thousands of labelled emails, a system that predicts a house price from past sales, a network that recognizes a face after seeing many faces: these are machine-learning systems1. What is at stake is generalization — the ability to do well on cases never seen during training1. A program that memorizes its training data has learned nothing useful; the whole discipline is organized around the gap between fitting what you have seen and performing on what you have not1. That gap, and the methods invented to close it, give machine learning its history.

Learning machines before learning theory

Early artificial intelligence in the 1950s and 1960s was mostly symbolic: intelligence as manipulation of explicit symbols and rules. But a parallel strand asked whether a machine could be taught instead. In 1943 Warren McCulloch and Walter Pitts had described an idealized neuron — a unit that sums weighted inputs and fires if the sum crosses a threshold — and in 1949 Donald Hebb proposed that learning could consist of strengthening connections between neurons that fire together13. Frank Rosenblatt's perceptron (1958) turned this into a working device: a single layer of threshold units whose weights were adjusted automatically from labelled examples43. The perceptron learning rule had a genuine guarantee, the perceptron convergence theorem, showing that if the data could be separated by a straight boundary, the procedure would find one in finite steps.

The limitation was equally sharp. In 1969 Marvin Minsky and Seymour Papert showed that a single-layer perceptron cannot compute even simple non-linear functions such as exclusive-or, and that stacking layers did not obviously help because no one knew how to train the hidden layers. Their critique helped cool interest in neural networks for over a decade4. Meanwhile other traditions grew: in the late 1950s Arthur Samuel's checkers program improved its play by adjusting weights through self-play, and in the 1960s work on nearest-neighbour classification — assigning a new case the label of the most similar stored examples — established instance-based learning, which needs no explicit model at all, only a notion of similarity and a store of past cases.

Symbolic concept learning and the rise of induction

Through the 1970s and 1980s, the dominant machine-learning work was symbolic1. The problem was concept learning: given positive and negative examples of a category, find a description that covers the positives and excludes the negatives5. Tom Mitchell's candidate-elimination algorithm (version space, early 1980s) maintained the set of all hypotheses consistent with the data seen so far6. Ross Quinlan's ID3 (1979) and its successor C4.5 built decision trees by choosing, at each node, the attribute that best separated the classes — measured by information gain — producing rules a person could read. These systems made learning concrete and inspectable, and they introduced ideas that outlived the symbolic era: that learning is search through a hypothesis space, and that a preference for simpler hypotheses (Occam's razor) helps choose among consistent ones7.

Inductive logic programming, developed from the late 1980s (notably by Stephen Muggleton and colleagues), pushed the same idea into first-order logic: learning general logical rules from examples and background knowledge, so that a system could infer, say, a recursive family relation from facts about parents18. It was expressive and elegant but computationally costly and hard to scale.

Statistics enters: risk, regularization, and the bias–variance tradeoff

The decisive shift came from statistics. Instead of asking whether a hypothesis is logically consistent with the data, the statistical view asks how well it will perform on future data drawn from the same distribution1. This is empirical risk minimization: choose the model that minimizes average loss on the training sample, in the hope that sample performance tracks true performance19. The hope is not automatic. Overfitting — fitting noise in the training data — is the central failure mode, and the remedies are regularization: penalties on model complexity, early stopping, or restricting the hypothesis class110.

The bias–variance tradeoff names the tension in plain terms1. A very flexible model has low bias (it can represent the truth) but high variance (it changes wildly with the particular sample); a rigid model is the reverse. Neither extreme generalizes well; the art is finding the middle. Cross-validation — repeatedly holding out part of the data to estimate future performance — became the standard practical tool for that choice. Vladimir Vapnik and Alexey Chervonenkis's work from the 1960s and 1970s, later called statistical learning theory, gave this intuition a rigorous foundation through measures of a model class's capacity (the VC dimension) and bounds on generalization error1112. The field's vocabulary — training versus test error, generalization, capacity — dates from this convergence, crystallized in the 1980s and 1990s.

Connectionism revived: backpropagation and neural networks

The obstacle Minsky and Papert had identified was training multilayer networks. The key idea — backpropagation of error — computes, for each weight in a layered network, how much it contributed to the output error, by propagating the error backward through the layers using the chain rule of calculus13. The idea had multiple independent origins; its widespread adoption followed a 1986 paper by David Rumelhart, Geoffrey Hinton, and Ronald Williams, which showed that hidden layers could learn useful internal representations. This was connectionism: intelligence as distributed activity in networks of simple units, with knowledge stored in weights rather than rules4. The 1986 Parallel Distributed Processing volumes, edited by Rumelhart and James McClelland, made the case at book length.

The practical results were striking. Yann LeCun's convolutional networks, trained by backpropagation in the late 1980s and 1990s, read handwritten zip codes and later a large share of handwritten checks in the United States. Yet through the 1990s neural networks were often outperformed on small datasets and were hard to train at depth; their moment would come later.

Margins, kernels, and ensembles

In the 1990s the statistical strand produced its most influential practical method. The support vector machine, developed by Vapnik and colleagues (Corinna Cortes and Vapnik, 1995), does not just find any separating boundary but the one with the widest margin — the largest buffer between the boundary and the nearest examples of each class12. Theory suggested wide margins generalize well. Kernel methods extended this to non-linear problems without explicitly computing high-dimensional features: a kernel function measures similarity between examples, and the algorithm works entirely through those similarities1214. Kernels also powered kernel PCA and Gaussian processes, a Bayesian approach to regression and classification.

Ensemble learning took a different route to accuracy: combine many weak models15. Bagging (Leo Breiman, 1996) trains models on bootstrap resamples and averages them, reducing variance; boosting (Yoav Freund and Robert Schapire, from the early 1990s) trains models sequentially, each focusing on the errors of the last, and Freund and Schapire's AdaBoost (1997) became a benchmark-beating workhorse. Breiman's random forests (2001) combined bagging with random feature selection and were robust and easy to use. These methods, along with support vector machines, dominated applied machine learning around 2000.

Probabilistic models and the Bayesian turn

A separate tradition treated learning as probabilistic inference16. Bayesian networks, developed by Judea Pearl in the 1980s, represent dependencies among variables as a directed graph and allow efficient inference and learning from data117. Graphical models more broadly — including hidden Markov models, which powered speech recognition and bioinformatics through the 1990s — gave a principled language for uncertainty. The Bayesian view treats learning as updating a prior distribution over hypotheses in light of data, yielding a posterior; this supplies automatic regularization and a natural account of uncertainty. Expectation-maximization (Dempster, Laird, and Rubin, 1977) made it practical to fit models with hidden variables. These ideas coexisted with, and often competed against, the margin-based and neural approaches, each claiming better use of limited data or better-calibrated uncertainty.

Deep learning

The turning point came when the obstacles to deep networks — insufficient data, insufficient compute, poor initialization, and the vanishing-gradient problem (error signals shrinking as they propagate back through many layers) — were overcome one by one. In 2006 Hinton and colleagues showed that deep belief networks could be pretrained layer by layer, reviving interest. The decisive demonstration was in image recognition: in 2012, Alex Krizhevsky, Ilya Sutskever, and Hinton's deep convolutional network AlexNet won the ImageNet competition by a large margin, trained on graphics processors. Depth plus scale plus data proved far more effective than hand-engineered features.

From there the field moved quickly: better activation functions (ReLU), dropout regularization, batch normalization, and then the transformer architecture (Vaswani and colleagues, 2017), whose attention mechanism allowed models to weigh relationships across long sequences and train at enormous scale. Deep learning absorbed the earlier lessons — backpropagation and gradient descent from connectionism, regularization and cross-validation from statistics, convolutional structure from vision — and became the default approach across vision, speech, and language.

Where the field stands

Today machine learning is dominated by deep neural networks trained by gradient descent on large datasets, with the earlier traditions still present and often combined: ensembles of trees remain strong on tabular data; kernel methods and Gaussian processes persist where data are scarce and uncertainty matters; probabilistic graphical models underlie much of causal inference and structured prediction; statistical learning theory supplies the vocabulary of generalization. The open questions are the old ones in sharper form: why do heavily overparameterized networks generalize at all, when classical theory says they should not? How can models be made robust to distribution shift, to adversarial inputs, and to the biases in their training data? How much data and compute is enough, and what are the limits of scaling? The discipline that began by asking how a machine could learn a straight boundary now asks how to trust systems with billions of parameters — but the question at its center, how to do well on what you have never seen, has not changed1.

References

  1. Machine learning (introduction, Supervised learning, Theory, History, Artificial intelligence, Association rules, Statistics, Overfitting, Bayesian networks), Wikipedia
  2. Outline of machine learning, Wikipedia
  3. Perceptron (History, introduction), Wikipedia
  4. Connectionism, Wikipedia
  5. Concept learning, Wikipedia
  6. Version space learning (introduction, Historical background), Wikipedia
  7. Occam's razor, Wikipedia
  8. Inductive logic programming (introduction, History), Wikipedia
  9. Empirical risk minimization, Wikipedia
  10. Regularization (mathematics), Wikipedia
  11. Vapnik–Chervonenkis theory, Wikipedia
  12. Support vector machine (introduction, History), Wikipedia
  13. Backpropagation, Wikipedia
  14. Kernel method, Wikipedia
  15. Ensemble learning, Wikipedia
  16. Bayesian inference, Wikipedia
  17. Bayesian network (introduction, History), Wikipedia

The frameworks of Machine Learning

Each framework is a way of working the field has used: when it held, and the question it answers.

  1. 1958-Present
  2. 1960-2000

    Instance-Based Learning

    How can past examples classify new cases?

    Fly to its tower in Nousfarer

  3. 1970-Present

    Statistical Learning Theory

    When will a model generalize to new data?

    About Statistical Learning Theory
    Statistical learning theory is the mathematical backbone of machine learning: a framework for asking, in precise terms, how a system that learns from a finite set of examples can be expected to perform on data it has never seen. Where much of machine learning is driven by engineering intuition — build a model, train it, hope it generalizes — statistical learning theory insists on formal answers. It treats learning not as a craft but as a statistical inference problem: given data, find a function that predicts well, and prove that it will keep predicting well.
    The problem the framework addresses is deceptively simple. A learner observes a training set of input-output pairs, drawn from some unknown probability distribution, and must select a hypothesis function that maps new inputs to correct outputs. The catch is that the learner never sees the true distribution — only a finite sample from it. Any function that fits the training data perfectly might fail completely on new data, a phenomenon known as overfitting. Statistical learning theory provides the vocabulary and the theorems for understanding when a function that works on the training set will also work in general, and for quantifying the price of complexity.

    The Core Organizing Idea: Empirical Risk Minimization

    The central construct of statistical learning theory is the distinction between two quantities. The first is the expected risk (or true risk): the average loss a hypothesis incurs over the entire unknown data distribution. The second is the empirical risk: the average loss over the training sample. The ideal learner would minimize expected risk, but since the distribution is unknown, it can only minimize empirical risk. This strategy — pick the function with the lowest error on the training data — is called empirical risk minimization (ERM).
    The entire framework hangs on one question: how close is the empirical risk to the expected risk? If the gap is small, then minimizing on the training set genuinely improves performance on future data. If the gap is large, the learner is fooling itself. Statistical learning theory's signature contribution is a family of results bounding this gap in terms of the complexity of the hypothesis space — the set of functions the learner is allowed to choose from.

    Essential Concepts in Learning Order

    To understand the framework, the concepts build on one another in a natural sequence.
    The learning setup. A learning problem consists of an input space, an output space, a loss function measuring prediction error, and a hypothesis space of candidate functions. The learner receives a training sample and must select a hypothesis. This formalization, standard across the field, is itself a contribution of statistical learning theory: it forces clarity about what is being learned and what counts as success.
    Expected risk and empirical risk. As described above, these two quantities define the gap that the theory seeks to control. The expected risk is the true measure of quality; the empirical risk is the only one the learner can compute.
    Overfitting and the bias-variance tradeoff. A hypothesis space that is too small cannot capture the true pattern (high bias); one that is too large can memorize the training noise (high variance). The theory explains this tradeoff quantitatively: larger hypothesis spaces make it easier to fit the training data but harder to guarantee that the fit means anything.
    Uniform convergence and the VC dimension. The key insight of Vapnik and Chervonenkis in the 1960s and 1970s was that the gap between empirical and expected risk can be bounded uniformly over the entire hypothesis space. If the space is not too complex, then with high probability, every hypothesis has empirical risk close to its expected risk — and therefore the one with the lowest empirical risk is genuinely good. The measure of complexity they introduced is the Vapnik-Chervonenkis (VC) dimension: the size of the largest set of points that the hypothesis space can shatter, or label in all possible ways. A finite VC dimension yields concrete bounds on the sample size needed to guarantee generalization.
    Structural risk minimization. Rather than fixing a single hypothesis space, one can nest a sequence of spaces of increasing complexity and choose the one that optimally balances empirical error against a complexity penalty. This is structural risk minimization (SRM), a principled alternative to the ad hoc model selection that dominates much of applied machine learning.
    Rademacher complexity. A later refinement, Rademacher complexity measures the capacity of a hypothesis space by its ability to fit random noise. It often yields tighter bounds than the VC dimension and extends naturally to more general settings, including regression and kernel methods.
    Regularization. In practice, the complexity penalties suggested by the theory take the form of regularization terms added to the training objective — penalizing large weights, for example. Statistical learning theory provides the justification for why such penalties work: they shrink the effective hypothesis space and thereby tighten the generalization bounds.

    How It Differs from Nearby Approaches

    Statistical learning theory is often confused with the broader field of machine learning, but its stance is distinctive. Where Bayesian machine learning treats model parameters as random variables and reasons about posterior distributions, statistical learning theory is frequentist: it makes no prior assumptions about the data-generating process and instead asks what can be guaranteed for any distribution. Where deep learning is driven by large-scale empirical experimentation and architectural innovation, statistical learning theory demands proofs and worst-case guarantees. Where symbolic machine learning seeks interpretable rules and logical structures, statistical learning theory works with continuous functions and geometric notions of complexity.
    The framework is also distinct from computational learning theory, which focuses on the computational difficulty of learning; statistical learning theory focuses instead on the statistical difficulty — how much data is needed, and how well a learner can be expected to perform. The two are complementary, but they ask different questions.

    Intellectual History

    The roots of statistical learning theory lie in classical statistics, particularly in the theory of empirical processes and uniform convergence of averages. But the framework as such emerged in the Soviet Union in the 1960s, when Vladimir Vapnik and Alexey Chervonenkis began developing what became known as VC theory. Their work, published in Russian in the late 1960s and early 1970s, established the fundamental bounds connecting the capacity of a hypothesis space to the reliability of empirical risk minimization.
    The framework remained largely unknown in the West until the 1990s, when Vapnik, by then at AT&T Bell Laboratories, developed the support vector machine (SVM) as a practical embodiment of the theory. The SVM's success on real-world problems — handwriting recognition, text classification, bioinformatics — brought statistical learning theory to the center of machine learning. Vapnik's 1995 book The Nature of Statistical Learning Theory and his 1998 Statistical Learning Theory codified the framework and made it accessible to a generation of researchers. The theory also influenced the development of kernel methods, which embed data into high-dimensional feature spaces where linear hypotheses become powerful, while the theory's complexity measures keep the effective capacity under control.

    Continuing Significance

    Statistical learning theory remains the intellectual foundation of machine learning, even as the field has moved toward deep neural networks whose theoretical analysis is far more difficult. The framework's concepts — generalization bounds, capacity control, the bias-variance tradeoff — are the vocabulary in which researchers discuss why models succeed or fail. Modern developments such as the analysis of overparameterized networks, the double-descent phenomenon, and the theory of transfer learning all engage with the questions that Vapnik and Chervonenkis first posed.
    The framework also has a practical legacy. The principle that model complexity must be controlled, whether through structural risk minimization, regularization, or cross-validation, is now standard practice in every branch of machine learning. The SVM and kernel methods remain widely used in domains where interpretability and theoretical guarantees matter more than raw predictive power.

    Prerequisites

    A working understanding of statistical learning theory requires a modest background in probability theory — random variables, expectation, concentration inequalities such as Hoeffding's inequality — and some familiarity with basic statistics, particularly the notion of estimation error. Linear algebra and basic calculus are helpful for following the derivations. No prior knowledge of machine learning is strictly necessary, though it helps to have seen a simple learning algorithm such as linear regression or the perceptron to ground the abstract concepts. For a reader with these foundations, the core ideas of the framework can be grasped without specialized mathematical training; the proofs, however, are another matter and reward a deeper background in empirical process theory and functional analysis.
    The 11 concepts its tower teaches
    • Learning Setup
    • Expected Risk
    • Empirical Risk
    • Empirical Risk Minimization
    • Overfitting
    • Bias-Variance Tradeoff
    • Uniform Convergence
    • VC Dimension
    • Structural Risk Minimization
    • Rademacher Complexity
    • Regularization

    Fly to its tower in Nousfarer

  4. 1975-1995

    Symbolic Concept Learning

    How can examples yield general concepts?

    Fly to its tower in Nousfarer

  5. 1985-2015

    Bayesian and probabilistic graphical models

    How can graphs represent uncertainty among variables?

    Fly to its tower in Nousfarer

  6. 1985-Present

    Bayesian inference

    How should beliefs change with evidence?

    Fly to its tower in Nousfarer

  7. 1985-Present

    Graphical model

    How do variables depend on one another?

    Fly to its tower in Nousfarer

  8. 1985-Present
  9. 1988-2005

    Inductive Logic Programming

    How can logic be learned from examples?

    Fly to its tower in Nousfarer

  10. 1990-Present

    Ensemble Learning

    Can many weak models make a strong one?

    Fly to its tower in Nousfarer

  11. 1992-2015

    Kernel method

    How can similarity solve non-linear problems?

    Fly to its tower in Nousfarer

  12. 2012-Present

    Deep Learning

    How can deep networks learn useful representations?

    About Deep Learning
    Deep learning addresses a foundational challenge in machine learning: how can a model learn representations of data at multiple levels of abstraction, from raw input to high-level concepts, without relying on hand-engineered features? Unlike earlier frameworks that often required significant domain expertise to design input features, deep learning aims to discover these features automatically through a hierarchy of layers. This approach became dominant after 2006 when researchers found effective ways to train deep neural networks, but its roots extend to earlier connectionist ideas.
    The core curriculum for deep learning builds from the basic building block of the artificial neuron to the architecture of layered networks and the procedures for training them. A first course typically covers these concepts in a logical sequence:
    • Artificial Neuron and Activation Functions: The fundamental computational unit, inspired by biological neurons. Common activation functions like ReLU (Rectified Linear Unit) and sigmoid introduce non-linearity, enabling networks to learn complex patterns.
    • Feedforward Neural Network: A basic architecture where information flows from the input layer, through one or more hidden layers, to the output layer without cycles. This structure forms the template for more specialized architectures.
    • Backpropagation: The core algorithm for training feedforward networks. It efficiently computes the gradient of the loss function with respect to every weight by propagating errors backward from the output to the input, using the chain rule of calculus.
    • Stochastic Gradient Descent and Loss Functions: Optimization procedures that iteratively update network weights to minimize a loss function (like cross-entropy for classification). Stochastic gradient descent uses a random subset of training data per update, making training feasible on large datasets.
    • Convolutional Neural Network (CNN) and Pooling: A specialized architecture designed for grid-like data such as images. Convolutional layers learn local, spatially invariant features (e.g., edges, textures), and pooling layers down-sample representations, reducing dimensionality and providing translation invariance.
    • Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM): Architectures for sequential data such as text or time series. RNNs maintain a hidden state that acts as a memory. LSTMs introduce gating mechanisms to learn long-range dependencies, mitigating the vanishing gradient problem.
    • Regularization (dropout, weight decay, batch normalization): Techniques to combat overfitting. Dropout randomly turns off neurons during training, weight decay penalizes large weights, and batch normalization stabilizes and accelerates training by normalizing layer inputs.
    Turning to intellectual history, the story of deep learning is one of repeated cycles of promise, setback, and eventual triumph, with the period from 2006 to the present marking its sustained dominance. The foundational idea of an artificial neuron dates to Warren McCulloch and Walter Pitts in 1943, and Frank Rosenblatt's perceptron in 1958 showed that simple neural networks could learn. However, Marvin Minsky and Seymour Papert's 1969 book Perceptrons exposed the perceptron's severe limitations (notably its inability to learn the XOR function), leading to an "AI winter" for connectionist research.
    A revival began in the 1980s with the Connectionism framework. Key figures like David Rumelhart, Geoffrey Hinton, and Ronald Williams popularized backpropagation in 1986, showing that multi-layer networks could overcome the XOR problem. Yet, training deep networks remained difficult due to vanishing gradients and limited data and compute. Shallow architectures (e.g., support vector machines from Kernel Methods) often outperformed neural networks for two decades.
    The decisive breakthrough came in 2006, marking the start of the current deep learning era. Geoffrey Hinton, Simon Osindero, and Yee-Whye Teh published "A Fast Learning Algorithm for Deep Belief Nets," showing that greedy layer-wise pretraining of a deep generative model could initialize weights effectively, enabling successful training of deep networks. This paper is widely credited with launching the modern deep learning wave.
    Subsequent rapid progress followed. In 2009, Hinton and his students demonstrated that deep neural networks trained with GPUs could dramatically improve speech recognition, a practical triumph that industry adopted. From 2010 onward, key figures and seminal works built the modern curriculum. Yann LeCun's work on convolutional networks (e.g., 1998's LeNet-5) gained widespread attention when Alex Kirzhevsky, Ilya Sutskever, and Hinton's 2012 work "ImageNet Classification with Deep Convolutional Networks" (AlexNet) won the ImageNet Large Scale Visual Recognition Challenge, drastically reducing error rates. This catalyzed the explosion of deep learning in computer vision. For sequence modeling, Hochreiter and Schmidhuber's 1997 LSTM paper became canonical, and later works like Sutskever et al.'s 2014 "Sequence to Sequence Learning with Neural Networks" enabled machine translation.
    Deep learning's success is also tied to large-scale resources. The Statistical Learning framework, which dominated from 1995, coexisted with deep learning, providing theoretical foundations (like bias-variance tradeoffs) that informed regularization practices. Deep learning also absorbed tools from Probabilistic Machine Learning (e.g., using dropout as approximate Bayesian inference). The framework's relationship with connectionism is direct: deep learning is connectionism after 2006, revitalized by deeper architectures, big data, and powerful hardware.
    Deep learning transformed machine learning by making feature engineering unnecessary for many tasks, enabling end-to-end learning from raw data. Its legacy is a reorientation of the field toward representation learning and scalable gradient-based optimization.
    The 14 concepts its tower teaches
    • Activation Functions
    • Artificial Neuron
    • Backpropagation
    • Batch Normalization
    • Convolutional Neural Network
    • Dropout
    • Feedforward Neural Network
    • Long Short-Term Memory
    • Loss Functions
    • Pooling
    • Recurrent Neural Network
    • Regularization
    • Stochastic Gradient Descent
    • Weight Decay

    Fly to its tower in Nousfarer

How the ideas connect

Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.

Connectionism and Neural NetworksInstance-Based LearningStatistical Learning TheorySymbolic Concept LearningBayesian and probabilistic graphical …Bayesian inferenceGraphical modelRegularization and Model SelectionInductive Logic ProgrammingEnsemble LearningKernel methodDeep Learning

Beyond the field

Where Machine Learning's ideas came from in other fields, and what they led to there: the strongest 12 of 58 links.

A city in Nousfarer: a red-carpeted stair climbing terraces to a crowned tower on an acropolis, more towers round it, forest and snowy mountains beyond.
A city in Nousfarer, the game played in the browser.

Machine Learning is a planet in Nousfarer

Its city has a tower for each of these 12 frameworks. Land, walk in, and climb one concept at a time.

Land on Machine Learning

Related fields

Fields its ideas are linked to

More in Artificial Intelligence

See where Machine Learning sits on the Map of Human Knowledge