Loading field map
Reinforcement learning (RL) is a subfield of artificial intelligence concerned with how an agent ought to take actions in an environment so as to maximize a cumulative reward signal. Unlike supervised learning, which learns from a fixed set of labeled examples, or unsupervised learning, which seeks structure in unlabeled data, RL is defined by the interaction between a decision-maker and its surroundings. The agent is not told which action is correct; it must discover that through trial and error, balancing the immediate consequences of its choices against their longer-term effects.
At its heart, RL formalizes the problem of learning from interaction. The standard mathematical framework is the Markov decision process (MDP). An MDP is defined by a set of states, a set of actions, a transition function that gives the probability of moving from one state to another given an action, and a reward function that gives the immediate numerical payoff for each transition. The agent's goal is to learn a policy—a mapping from states to actions—that maximizes the total accumulated reward over time, typically with future rewards discounted by a factor between 0 and 1.
This formulation captures a wide range of problems. Playing chess, controlling a robot arm, managing a power grid, and recommending content to a user can all be cast as MDPs. The central difficulty is that the agent does not know the transition function or the reward function in advance. It must explore its environment to gather information, while simultaneously exploiting what it already knows to obtain reward. This is the exploration-exploitation dilemma, a fundamental tension that distinguishes RL from other forms of learning.
A second core challenge is the credit assignment problem. When an agent receives a reward, it may be the result of an action taken many steps earlier. The agent must determine which of its past decisions was responsible for the current outcome. This is particularly difficult when rewards are sparse, such as winning a game only after a long sequence of moves.
The intellectual roots of reinforcement learning lie in two distinct traditions: the study of animal learning in psychology and the theory of optimal control in mathematics and engineering.
In psychology, the idea of learning through reward and punishment—operant conditioning—was developed in the early twentieth century. Researchers such as Edward Thorndike and B. F. Skinner observed that animals repeat behaviors that lead to satisfying outcomes and avoid those that lead to unpleasant ones. This principle, sometimes called the law of effect, provided a conceptual vocabulary for describing how an agent might learn from the consequences of its actions.
In parallel, mathematicians and engineers developed dynamic programming and optimal control theory. Richard Bellman's work in the 1950s introduced the Bellman equation, which expresses the value of a state as the sum of the immediate reward and the discounted value of the best possible next state. This equation is the backbone of modern RL. Dynamic programming provides a way to compute an optimal policy when the transition function is known, but it requires a complete model of the environment and is computationally intractable for large problems.
The modern field of reinforcement learning emerged in the 1980s when researchers began combining these ideas with computational methods. The key insight was to use experience—samples of transitions and rewards—to estimate the value function without needing a model of the environment. This approach, called temporal-difference learning, was developed by Richard Sutton and Andrew Barto. It updates the value of a state based on the difference between the observed reward plus the value of the next state and the current estimate. This simple update rule allowed agents to learn directly from experience, without knowing the transition probabilities.
The 1990s saw the development of Q-learning, an off-policy algorithm that learns the value of taking a particular action in a particular state. Q-learning was a major advance because it could learn the optimal policy even when the agent was behaving suboptimally, which made it useful for exploration. However, these early algorithms were limited to problems with small, discrete state spaces. They could not handle the high-dimensional inputs of real-world problems like images or speech.
The field of RL is organized around several distinct research programmes, each addressing a different aspect of the core problem. These approaches are not mutually exclusive; they are often combined in practice.
The most fundamental division in RL is between model-based and model-free methods. A model of the environment is a representation of the transition function and the reward function. Model-based methods learn this model from experience and then use it to plan. They can simulate future trajectories and choose actions by looking ahead. This makes them sample-efficient, meaning they require fewer interactions with the real environment to learn a good policy. However, the model itself may be inaccurate, and errors in the model can lead to poor decisions.
Model-free methods skip the model entirely. They directly learn a policy or a value function from experience. They are simpler to implement and can be more robust to model errors, but they typically require many more samples to learn. The two main families of model-free methods are value-based and policy-based.
Value-based methods estimate the value function—the expected total reward from a given state or state-action pair. The agent then chooses actions that lead to states with the highest value. Q-learning is the canonical value-based method. Policy-based methods, by contrast, directly parameterize the policy and adjust its parameters to increase the probability of actions that lead to high reward. They are often better suited for problems with continuous action spaces, where it is not feasible to evaluate every possible action.
The most transformative development in RL was the integration of deep neural networks as function approximators. Traditional RL methods stored value estimates in tables, which limited them to small state spaces. Deep neural networks can represent complex functions over high-dimensional inputs, such as images or raw sensor data.
The breakthrough came in 2013, when researchers at DeepMind demonstrated a deep Q-network (DQN) that could play Atari video games at a superhuman level using only the raw pixels as input. The key innovations were the use of a deep neural network to approximate the Q-function and the introduction of experience replay, a technique that stores past transitions and samples from them randomly to break the correlation between consecutive samples. This stabilized the training process and made deep RL practical.
Since then, deep RL has been extended to many domains. The actor-critic architecture combines value-based and policy-based methods: an actor learns the policy, while a critic learns the value function and provides feedback to the actor. This approach has been used to achieve superhuman performance in games like Go and Dota 2, and to control robots in simulated environments.
Policy gradient methods are a family of algorithms that directly optimize the policy by computing the gradient of the expected reward with respect to the policy parameters. They are particularly well-suited for problems with continuous action spaces, where value-based methods struggle. The REINFORCE algorithm, introduced in the 1990s, is the simplest policy gradient method. It uses Monte Carlo sampling to estimate the gradient, which makes it unbiased but high-variance.
Modern policy gradient methods, such as proximal policy optimization (PPO) and trust region policy optimization (TRPO), address the variance problem by constraining the size of the policy updates. They ensure that the new policy does not deviate too far from the old one, which prevents catastrophic performance collapse. These methods have become the default choice for many RL applications because they are relatively stable and easy to tune.
A separate research programme addresses the problem of long-horizon tasks by decomposing them into a hierarchy of subtasks. Instead of learning a single policy that maps states to actions, hierarchical RL learns a high-level policy that selects among sub-policies, each of which is responsible for a specific subtask. This approach is motivated by the observation that many complex tasks have a natural hierarchical structure, and learning a flat policy for them is inefficient.
The options framework, introduced by Sutton and colleagues, formalizes this idea. An option is a temporally extended action that includes a policy, a termination condition, and an initiation set. The agent learns to select among options, and each option is itself a policy. Hierarchical RL has been applied to tasks such as navigation and manipulation, but it remains a challenging research area because the sub-task decomposition is often not known in advance.
The current state of RL is characterized by a combination of impressive successes and significant limitations. On the success side, RL has achieved superhuman performance in several games, including Go, chess, and Dota 2. It has been used to optimize data center cooling, to design better chip layouts, and to control robotic arms in industrial settings. These successes have attracted substantial investment from industry and have made RL one of the most visible subfields of AI.
However, the field is also grappling with several fundamental challenges. The sample efficiency problem remains acute: most RL algorithms require millions of interactions with the environment to learn a good policy. This is acceptable in simulated environments, but it is often impractical in the real world, where interactions are slow and costly. Model-based methods are seen as a promising direction for addressing this issue, but they have not yet achieved the same level of performance as model-free methods in complex domains.
The reward specification problem is another major concern. In many real-world applications, it is difficult to define a reward function that accurately captures the desired behavior. A robot that is rewarded for reaching a goal may find a way to cheat, such as by knocking the goal over. This has led to research on inverse RL, where the agent infers the reward function from demonstrations, and on safe RL, where the agent is constrained to avoid certain states or actions.
The exploration problem remains open. The exploration-exploitation trade-off is fundamental, but current exploration strategies are often inefficient. Random exploration, where the agent takes random actions with a certain probability, is simple but does not scale to large state spaces. More sophisticated methods, such as curiosity-driven exploration, which rewards the agent for visiting novel states, have shown promise but are not yet fully understood.
Finally, the theoretical foundations of RL are still being developed. While the convergence properties of tabular methods are well understood, the behavior of deep RL algorithms is not. There is no guarantee that a deep Q-network will converge to the optimal policy, and the field relies heavily on empirical evaluation. This has led to a reproducibility crisis, where small changes in implementation details can lead to large differences in performance.
Despite these limitations, RL remains a vibrant and rapidly evolving field. The combination of a clear formal framework, a rich set of algorithms, and a wide range of applications makes it one of the most active areas of AI research. The field is not a single monolithic approach but a collection of methods that address different aspects of the sequential decision-making problem, and the most successful systems often combine ideas from multiple approaches.