Robot Perception And Slam

Robot perception is the part of robotics that answers the question "what is around me, and where am I?" A robot that cannot answer it is blind: it can repeat a motion it was taught, but it cannot cope with a room that has been rearranged, a corridor it has never seen, or a map that is slightly wrong.

9 frameworks, from 1985 to today · Robotics · Engineering

still a working view ended · each leads to its entry below

Overview

Robot perception is the part of robotics that answers the question "what is around me, and where am I?" A robot that cannot answer it is blind: it can repeat a motion it was taught, but it cannot cope with a room that has been rearranged, a corridor it has never seen, or a map that is slightly wrong. SLAM—Simultaneous Localization and Mapping—is the hardest version of that question1. A robot enters an unknown building with no map and no external positioning system1. It must build a map of the place while simultaneously working out where it is within the map it is still building1. Each half of the problem depends on the other: to place a landmark on the map you need to know where you were when you saw it, and to know where you are you need the map. The history of this subfield is the history of learning to solve that circular problem reliably, and then of learning to do it with cameras, with cheap inertial sensors, and increasingly with learned models rather than hand-written geometry.

Dead Reckoning and the Limits of Odometry

The earliest mobile robots navigated by odometry: counting wheel rotations to estimate how far and how far around the robot has travelled. This is dead reckoning, and its flaw is that errors accumulate. A small miscalibration in wheel radius, a little slip on a polished floor, and the estimated position drifts steadily away from the true one, with nothing in the system to notice the discrepancy. The classic remedy from the 1980s and 1990s was to add beacons or landmarks at known positions and triangulate against them—but that presumes someone has already surveyed the environment and installed the infrastructure. The research question that drove the field was how to escape that presumption.

Probabilistic Mapping and the Occupancy Grid

The first great shift was to stop treating the robot's knowledge as a single best guess and treat it as a probability distribution. In the mid-1980s, Alberto Elfes and Hans Moravec independently developed occupancy grid mapping: divide the floor plan into a fine grid of cells, and for each cell store the probability that it is occupied. A sonar or laser range reading does not say "there is a wall here"; it says "I saw something at this distance along this bearing," which weakly marks cells along the ray as free and the cell at the end as occupied. Bayes' rule lets each new reading update every cell's probability, so evidence accumulates and noise averages out. The grid became the standard map representation for indoor robots for two decades, and it is still used today. Its weakness is memory and computation—a large building at fine resolution is millions of cells—and it assumes the robot knows where it is when it takes each reading, which is exactly what SLAM must not assume.

The Kalman Filter Enters Robotics

The decisive conceptual move came from estimation theory. The Kalman filter, published by Rudolf Kálmán in 1960, is a recursive recipe for estimating the state of a system from noisy measurements: keep a best estimate and an uncertainty (a covariance), predict forward using a motion model, then correct using a sensor reading, weighting each by how much you trust it23. It is optimal for linear systems with Gaussian noise. Robots are neither, but the extended Kalman filter (EKF) linearizes the models around the current estimate and applies the same machinery2.

In 1986, Randall Smith and Peter Cheeseman published a paper on estimating spatial relationships from uncertain observations that framed mapping as an estimation problem1. Then in the late 1980s and 1990s, a series of researchers—among them John Leonard and Hugh Durrant-Whyte, and later Sebastian Thrun, Dieter Fox, and Wolfram Burgard—built EKF-SLAM: put the robot's pose and every landmark's position into one big state vector, and let the filter's covariance capture the correlations between them. This was the key insight. Because the same uncertain robot pose was used to observe many landmarks, the landmarks' errors are correlated, and the filter tracks those correlations. When the robot later re-observes a landmark, the correction propagates through the correlations to sharpen the whole map. EKF-SLAM worked, and it made the circularity of SLAM tractable for the first time. Its costs were real: the covariance matrix grows with the square of the number of landmarks, and linearization errors can make the filter overconfident and diverge.

Particle Filters and Monte Carlo Localization

A parallel line of work attacked the simpler problem of localization given a known map. Monte Carlo localization, introduced by Fox, Burgard, and Thrun in the late 1990s, represents the robot's belief as a cloud of weighted samples—particles—each a hypothetical pose. The robot moves; each particle is moved according to the noisy motion model. The robot senses; each particle is reweighted by how well the reading matches what a robot at that pose would have seen. Particles that fit badly die out and are resampled around the survivors. The cloud concentrates on the true pose. This approach handles non-Gaussian, multi-modal beliefs—the "kidnapped robot" case, where the robot is picked up and put elsewhere and must relocalize from scratch—far better than a Kalman filter can. Particle filters became the standard for localization and were later extended to full SLAM (FastSLAM, by Michael Montemerlo and Thrun in the early 2000s, factored the problem into a particle filter over robot trajectories and per-landmark Kalman filters).

Graph SLAM and the Return of Least Squares

The next turning point was a change of representation. Instead of maintaining a filter online, graph SLAM records the robot's trajectory as a graph: nodes are poses, and edges are constraints—"odometry says I moved this far," "I observed landmark L from here," "I revisited a place I have been before." Solving SLAM then means finding the trajectory and map that best satisfy all constraints at once, a large least-squares optimization. This idea has roots in the 1990s (work by Feng Lu and Evangelos Milios, and by John Folkesson and Henrik Christensen), but it became practical in the 2000s and 2010s as researchers—notably Frank Dellaert and Michael Kaess with factor graphs and incremental smoothing, and Giorgio Grisetti, Rainer Kümmerle, Cyrill Stachniss, and Burgard with the graph optimization library g2o—developed sparse solvers that could handle thousands of poses. Graph SLAM's advantage over filtering is that it can revise the whole past when new information arrives, rather than committing to old decisions. It is the dominant formulation today.

Loop Closure and Place Recognition

Graph SLAM only works if the graph has the right edges, and the critical ones are loop closures: recognizing that the robot has returned to a place it visited before1. Without loop closure, drift accumulates forever; with a false loop closure, the map is torn apart. Place recognition—matching a current sensor view against previously recorded ones—became a research problem in its own right. Early systems matched laser range scans; later work matched camera images. A landmark result was FAB-MAP, by Mark Cummins and Paul Newman in 2008, which used a probabilistic model over visual words to decide whether two images came from the same place, with explicit handling of the possibility that they did not. The problem is hard because the same place looks different at different times of day, from different angles, and with people moving through it. Robust loop closure, including rejecting false matches, is still an active concern.

Visual SLAM

Cameras are cheap, passive, information-rich, and everywhere—so making SLAM work from images alone was an obvious goal and a hard one. A single camera gives bearing to features but not depth, so the map's scale is unobservable until the camera moves. Visual SLAM matured through the 2000s and 2010s. Andrew Davison's MonoSLAM (2003) demonstrated real-time monocular EKF-SLAM. Georg Klein and David Murray's PTAM (2007) split tracking (fast, every frame) from mapping (slower, background), a structure that most later systems adopted. Raúl Mur-Artal, José María Martínez Montiel, and Juan Domingo Tardós's ORB-SLAM (2015) and its successors became a widely used reference: feature-based, with parallel threads for tracking, local mapping, and loop closing, and the ability to relocalize after tracking loss. A competing strand, direct methods, skips feature extraction and aligns image intensities directly—LSD-SLAM by Jakob Engel and colleagues (2014) is a prominent example. Visual SLAM made dense, camera-only mapping practical and pushed the field toward the sensors robots actually carry.

Visual-Inertial Fusion

Cameras are accurate but can be confused by motion blur, textureless walls, or fast rotation; inertial measurement units (IMUs—gyroscopes and accelerometers) are cheap and fast but drift badly. Fusing them is complementary: the IMU carries the estimate through brief visual outages, and the camera bounds the IMU's drift. Visual-inertial odometry became a major research thrust in the 2010s, with influential systems including MSCKF (Anastasios Mourikis and Stergios Roumeliotis, 2007) and, later, open frameworks such as VINS-Mono (Tong Qin, Peiliang Li, and Shaojie Shen, 2018). The combination is now standard on drones, phones, and mobile robots, and it is what makes robust SLAM possible on small, low-cost platforms.

Learning Enters Perception

The most recent shift is the use of machine learning, especially deep neural networks, inside perception pipelines4. Instead of hand-designing features and geometric models, researchers train networks on large datasets to detect objects, estimate depth from a single image, segment scenes, and recognize places. Learning-based robot perception has changed what robots can see: semantic segmentation tells a robot not just that something is there but that it is a chair, a person, or a door; learned depth estimation gives monocular cameras a sense of distance; learned place recognition handles appearance change better than hand-crafted descriptors. Learned components are increasingly combined with classical geometric SLAM—networks propose, geometry verifies—and end-to-end learned odometry and mapping systems are an active frontier. The open questions are the familiar ones in new form: how to know when a learned model is wrong, how to generalize beyond the training environments, and how to certify that a perception system is safe enough to trust.

Where the Field Stands

Today the main approaches descend directly from this history. Graph-based optimization with robust loop closure is the backbone of most modern SLAM systems; visual-inertial fusion is the standard sensor configuration for mobile platforms; occupancy grids and their three-dimensional descendants remain the default map for navigation; particle filters persist in localization; and learned models are being folded into every stage of the pipeline. The questions still open are the ones the field has always faced in new clothes: robustness in environments that change, in weather and lighting that degrade sensors, and in situations no training set anticipated; knowing when the estimate is unreliable; and scaling to large, long-lived maps that many robots share. The circular problem that defined the field—where am I, given a map I am still making—has been solved well enough to put robots in warehouses, on roads, and in homes. It has not been solved so well that the field is finished.

References

  1. Simultaneous localization and mapping (introduction, History, Loop closure), Wikipedia
  2. Extended Kalman filter (History, introduction), Wikipedia
  3. Kalman filter (introduction, History), Wikipedia
  4. Deep learning, Wikipedia

The frameworks of Robot Perception And Slam

Each framework is a way of working the field has used: when it held, and the question it answers.

  1. 1985-Present

    Occupancy Grid Mapping

    How should a robot represent which cells are occupied?

    Fly to its tower in Nousfarer

  2. 1985-Present

    Probabilistic Spatial Uncertainty Estimation

    How uncertain is the robot about where things are?

    Fly to its tower in Nousfarer

  3. 1986-2010

    Kalman Filter and EKF-SLAM

    How can pose and landmarks be estimated together?

    Fly to its tower in Nousfarer

  4. 1999-Present

    Monte Carlo localization

    How can a robot find itself on a known map?

    Fly to its tower in Nousfarer

  5. 2003-Present

    Visual SLAM

    How can a robot map and locate itself from cameras?

    Fly to its tower in Nousfarer

  6. 2005-Present

    Graph SLAM and Factor Graph Optimization

    How can all past constraints be satisfied at once?

    Fly to its tower in Nousfarer

  7. 2007-Present

    Visual-Inertial Fusion

    How can cameras and inertial sensors be combined reliably?

    Fly to its tower in Nousfarer

  8. 2008-Present

    Place Recognition and Loop Closure

    How can a robot recognize a place it has seen before?

    Fly to its tower in Nousfarer

  9. 2012-Present

    Learning-Based Robot Perception

    Can robots learn to perceive without hand-written geometry?

    Fly to its tower in Nousfarer

How the ideas connect

Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.

Occupancy Grid MappingProbabilistic Spatial Uncertainty Est…Kalman Filter and EKF-SLAMMonte Carlo localizationVisual SLAMGraph SLAM and Factor Graph Optimizat…Visual-Inertial FusionPlace Recognition and Loop ClosureLearning-Based Robot Perception

Beyond the field

Where Robot Perception And Slam's ideas came from in other fields, and what they led to there: the strongest 12 of 27 links.

A city in Nousfarer: a red-carpeted stair climbing terraces to a crowned tower on an acropolis, more towers round it, forest and snowy mountains beyond.
A city in Nousfarer, the game played in the browser.

Robot Perception And Slam is a planet in Nousfarer

Its city has a tower for each of these 9 frameworks. Land, walk in, and climb one concept at a time.

Land on Robot Perception And Slam

Related fields

Fields its ideas are linked to

More in Robotics

See where Robot Perception And Slam sits on the Map of Human Knowledge