- Atlas
- Engineering
- Robot Perception And Slam
Robot Perception And Slam
Robot perception is the part of robotics that answers the question "what is around me, and where am I?" A robot that cannot answer it is blind: it can repeat a motion it was taught, but it cannot cope with a room that has been rearranged, a corridor it has never seen, or a map that is slightly wrong.
9 frameworks, from 1985 to today · Robotics · Engineering
- Occupancy Grid Mapping 1985-Present
- Probabilistic Spatial Uncertainty Estimation 1985-Present
- Kalman Filter and EKF-SLAM 1986-2010
- Monte Carlo localization 1999-Present
- Visual SLAM 2003-Present
- Graph SLAM and Factor Graph Optimization 2005-Present
- Visual-Inertial Fusion 2007-Present
- Place Recognition and Loop Closure 2008-Present
- Learning-Based Robot Perception 2012-Present
Overview
Robot perception is the part of robotics that answers the question "what is around me, and where am I?" A robot that cannot answer it is blind: it can repeat a motion it was taught, but it cannot cope with a room that has been rearranged, a corridor it has never seen, or a map that is slightly wrong. SLAM—Simultaneous Localization and Mapping—is the hardest version of that question1. A robot enters an unknown building with no map and no external positioning system1. It must build a map of the place while simultaneously working out where it is within the map it is still building1. Each half of the problem depends on the other: to place a landmark on the map you need to know where you were when you saw it, and to know where you are you need the map. The history of this subfield is the history of learning to solve that circular problem reliably, and then of learning to do it with cameras, with cheap inertial sensors, and increasingly with learned models rather than hand-written geometry.
Dead Reckoning and the Limits of Odometry
The earliest mobile robots navigated by odometry: counting wheel rotations to estimate how far and how far around the robot has travelled. This is dead reckoning, and its flaw is that errors accumulate. A small miscalibration in wheel radius, a little slip on a polished floor, and the estimated position drifts steadily away from the true one, with nothing in the system to notice the discrepancy. The classic remedy from the 1980s and 1990s was to add beacons or landmarks at known positions and triangulate against them—but that presumes someone has already surveyed the environment and installed the infrastructure. The research question that drove the field was how to escape that presumption.
Probabilistic Mapping and the Occupancy Grid
The first great shift was to stop treating the robot's knowledge as a single best guess and treat it as a probability distribution. In the mid-1980s, Alberto Elfes and Hans Moravec independently developed occupancy grid mapping: divide the floor plan into a fine grid of cells, and for each cell store the probability that it is occupied. A sonar or laser range reading does not say "there is a wall here"; it says "I saw something at this distance along this bearing," which weakly marks cells along the ray as free and the cell at the end as occupied. Bayes' rule lets each new reading update every cell's probability, so evidence accumulates and noise averages out. The grid became the standard map representation for indoor robots for two decades, and it is still used today. Its weakness is memory and computation—a large building at fine resolution is millions of cells—and it assumes the robot knows where it is when it takes each reading, which is exactly what SLAM must not assume.
The Kalman Filter Enters Robotics
The decisive conceptual move came from estimation theory. The Kalman filter, published by Rudolf Kálmán in 1960, is a recursive recipe for estimating the state of a system from noisy measurements: keep a best estimate and an uncertainty (a covariance), predict forward using a motion model, then correct using a sensor reading, weighting each by how much you trust it23. It is optimal for linear systems with Gaussian noise. Robots are neither, but the extended Kalman filter (EKF) linearizes the models around the current estimate and applies the same machinery2.
In 1986, Randall Smith and Peter Cheeseman published a paper on estimating spatial relationships from uncertain observations that framed mapping as an estimation problem1. Then in the late 1980s and 1990s, a series of researchers—among them John Leonard and Hugh Durrant-Whyte, and later Sebastian Thrun, Dieter Fox, and Wolfram Burgard—built EKF-SLAM: put the robot's pose and every landmark's position into one big state vector, and let the filter's covariance capture the correlations between them. This was the key insight. Because the same uncertain robot pose was used to observe many landmarks, the landmarks' errors are correlated, and the filter tracks those correlations. When the robot later re-observes a landmark, the correction propagates through the correlations to sharpen the whole map. EKF-SLAM worked, and it made the circularity of SLAM tractable for the first time. Its costs were real: the covariance matrix grows with the square of the number of landmarks, and linearization errors can make the filter overconfident and diverge.
Particle Filters and Monte Carlo Localization
A parallel line of work attacked the simpler problem of localization given a known map. Monte Carlo localization, introduced by Fox, Burgard, and Thrun in the late 1990s, represents the robot's belief as a cloud of weighted samples—particles—each a hypothetical pose. The robot moves; each particle is moved according to the noisy motion model. The robot senses; each particle is reweighted by how well the reading matches what a robot at that pose would have seen. Particles that fit badly die out and are resampled around the survivors. The cloud concentrates on the true pose. This approach handles non-Gaussian, multi-modal beliefs—the "kidnapped robot" case, where the robot is picked up and put elsewhere and must relocalize from scratch—far better than a Kalman filter can. Particle filters became the standard for localization and were later extended to full SLAM (FastSLAM, by Michael Montemerlo and Thrun in the early 2000s, factored the problem into a particle filter over robot trajectories and per-landmark Kalman filters).
Graph SLAM and the Return of Least Squares
The next turning point was a change of representation. Instead of maintaining a filter online, graph SLAM records the robot's trajectory as a graph: nodes are poses, and edges are constraints—"odometry says I moved this far," "I observed landmark L from here," "I revisited a place I have been before." Solving SLAM then means finding the trajectory and map that best satisfy all constraints at once, a large least-squares optimization. This idea has roots in the 1990s (work by Feng Lu and Evangelos Milios, and by John Folkesson and Henrik Christensen), but it became practical in the 2000s and 2010s as researchers—notably Frank Dellaert and Michael Kaess with factor graphs and incremental smoothing, and Giorgio Grisetti, Rainer Kümmerle, Cyrill Stachniss, and Burgard with the graph optimization library g2o—developed sparse solvers that could handle thousands of poses. Graph SLAM's advantage over filtering is that it can revise the whole past when new information arrives, rather than committing to old decisions. It is the dominant formulation today.
Loop Closure and Place Recognition
Graph SLAM only works if the graph has the right edges, and the critical ones are loop closures: recognizing that the robot has returned to a place it visited before1. Without loop closure, drift accumulates forever; with a false loop closure, the map is torn apart. Place recognition—matching a current sensor view against previously recorded ones—became a research problem in its own right. Early systems matched laser range scans; later work matched camera images. A landmark result was FAB-MAP, by Mark Cummins and Paul Newman in 2008, which used a probabilistic model over visual words to decide whether two images came from the same place, with explicit handling of the possibility that they did not. The problem is hard because the same place looks different at different times of day, from different angles, and with people moving through it. Robust loop closure, including rejecting false matches, is still an active concern.
Visual SLAM
Cameras are cheap, passive, information-rich, and everywhere—so making SLAM work from images alone was an obvious goal and a hard one. A single camera gives bearing to features but not depth, so the map's scale is unobservable until the camera moves. Visual SLAM matured through the 2000s and 2010s. Andrew Davison's MonoSLAM (2003) demonstrated real-time monocular EKF-SLAM. Georg Klein and David Murray's PTAM (2007) split tracking (fast, every frame) from mapping (slower, background), a structure that most later systems adopted. Raúl Mur-Artal, José María Martínez Montiel, and Juan Domingo Tardós's ORB-SLAM (2015) and its successors became a widely used reference: feature-based, with parallel threads for tracking, local mapping, and loop closing, and the ability to relocalize after tracking loss. A competing strand, direct methods, skips feature extraction and aligns image intensities directly—LSD-SLAM by Jakob Engel and colleagues (2014) is a prominent example. Visual SLAM made dense, camera-only mapping practical and pushed the field toward the sensors robots actually carry.
Visual-Inertial Fusion
Cameras are accurate but can be confused by motion blur, textureless walls, or fast rotation; inertial measurement units (IMUs—gyroscopes and accelerometers) are cheap and fast but drift badly. Fusing them is complementary: the IMU carries the estimate through brief visual outages, and the camera bounds the IMU's drift. Visual-inertial odometry became a major research thrust in the 2010s, with influential systems including MSCKF (Anastasios Mourikis and Stergios Roumeliotis, 2007) and, later, open frameworks such as VINS-Mono (Tong Qin, Peiliang Li, and Shaojie Shen, 2018). The combination is now standard on drones, phones, and mobile robots, and it is what makes robust SLAM possible on small, low-cost platforms.
Learning Enters Perception
The most recent shift is the use of machine learning, especially deep neural networks, inside perception pipelines4. Instead of hand-designing features and geometric models, researchers train networks on large datasets to detect objects, estimate depth from a single image, segment scenes, and recognize places. Learning-based robot perception has changed what robots can see: semantic segmentation tells a robot not just that something is there but that it is a chair, a person, or a door; learned depth estimation gives monocular cameras a sense of distance; learned place recognition handles appearance change better than hand-crafted descriptors. Learned components are increasingly combined with classical geometric SLAM—networks propose, geometry verifies—and end-to-end learned odometry and mapping systems are an active frontier. The open questions are the familiar ones in new form: how to know when a learned model is wrong, how to generalize beyond the training environments, and how to certify that a perception system is safe enough to trust.
Where the Field Stands
Today the main approaches descend directly from this history. Graph-based optimization with robust loop closure is the backbone of most modern SLAM systems; visual-inertial fusion is the standard sensor configuration for mobile platforms; occupancy grids and their three-dimensional descendants remain the default map for navigation; particle filters persist in localization; and learned models are being folded into every stage of the pipeline. The questions still open are the ones the field has always faced in new clothes: robustness in environments that change, in weather and lighting that degrade sensors, and in situations no training set anticipated; knowing when the estimate is unreliable; and scaling to large, long-lived maps that many robots share. The circular problem that defined the field—where am I, given a map I am still making—has been solved well enough to put robots in warehouses, on roads, and in homes. It has not been solved so well that the field is finished.
References
- Simultaneous localization and mapping (introduction, History, Loop closure), Wikipedia
- Extended Kalman filter (History, introduction), Wikipedia
- Kalman filter (introduction, History), Wikipedia
- Deep learning, Wikipedia
The frameworks of Robot Perception And Slam
Each framework is a way of working the field has used: when it held, and the question it answers.
- 1985-Present
Occupancy Grid Mapping
How should a robot represent which cells are occupied?
- 1985-Present
Probabilistic Spatial Uncertainty Estimation
How uncertain is the robot about where things are?
- 1986-2010
Kalman Filter and EKF-SLAM
How can pose and landmarks be estimated together?
- 1999-Present
- 2003-Present
- 2005-Present
Graph SLAM and Factor Graph Optimization
How can all past constraints be satisfied at once?
- 2007-Present
Visual-Inertial Fusion
How can cameras and inertial sensors be combined reliably?
- 2008-Present
Place Recognition and Loop Closure
How can a robot recognize a place it has seen before?
- 2012-Present
Learning-Based Robot Perception
Can robots learn to perceive without hand-written geometry?
How the ideas connect
Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.
Graph SLAM and Factor Graph Optimization replaced Kalman Filter and EKF-SLAM
Graph SLAM replaced the online EKF formulation by recording pose constraints and solving a global least-squares problem, allowing the whole trajectory to be revised when new information arrived. Sparse solvers and factor graphs made this the dominant formulation over filtering.
Carried by Frank Dellaert; Michael Kaess; Giorgio Grisetti.
Monte Carlo localization reacted against Kalman Filter and EKF-SLAM
Monte Carlo localization was introduced as a particle-filter alternative to Kalman-filter localization, handling non-Gaussian and multi-modal beliefs such as the kidnapped-robot case that EKF methods could not. It defined itself against the Gaussian single-hypothesis assumption of the Kalman approach.
Carried by Dieter Fox; Wolfram Burgard; Sebastian Thrun.
Visual SLAM grew out of Kalman Filter and EKF-SLAM
MonoSLAM demonstrated real-time monocular SLAM using an extended Kalman filter over camera pose and feature positions, directly applying the EKF-SLAM formulation to a single camera. Visual SLAM thus grew out of the EKF-SLAM framework before later systems moved to graph optimization.
Carried by Andrew Davison; MonoSLAM (2003).
Visual-Inertial Fusion grew out of Visual SLAM
Visual-inertial fusion grew out of visual SLAM by adding IMU measurements to overcome the fragility of cameras under motion blur and fast rotation. The camera-based pose estimation of visual SLAM became one half of the fused estimator.
Place Recognition and Loop Closure influenced by Visual SLAM
Visual SLAM systems needed to recognize previously visited places from camera images to close loops, and FAB-MAP addressed this by matching visual words probabilistically. Place recognition thus developed as a component serving camera-based mapping.
Carried by Mark Cummins; Paul Newman.
Visual SLAM influenced by Graph SLAM and Factor Graph Optimization
PTAM split tracking from mapping and used bundle adjustment over keyframes, bringing least-squares optimization into visual SLAM. Later systems such as ORB-SLAM adopted graph-based optimization for local mapping and loop closing, carrying the graph SLAM approach into camera-based systems.
Carried by Georg Klein; David Murray; PTAM (2007).
Place Recognition and Loop Closure influenced by Graph SLAM and Factor Graph Optimization
Graph SLAM made loop closures the critical edges that correct accumulated drift, turning place recognition into a necessary component. FAB-MAP and later place-recognition systems were built to supply reliable loop-closure constraints for graph-based SLAM.
Carried by Mark Cummins; Paul Newman; FAB-MAP (2008).
Visual-Inertial Fusion influenced by Kalman Filter and EKF-SLAM
Visual-inertial fusion adopted the Kalman-filter and EKF machinery from EKF-SLAM to combine camera and IMU measurements in a recursive estimator. The filtering framework for fusing uncertain sensors passed into the visual-inertial systems.
Learning-Based Robot Perception influenced by Visual SLAM
Learning-based robot perception took the feature detection, matching, and place-recognition tasks defined by visual SLAM and replaced hand-crafted pipelines with learned models. The problems and benchmarks of visual SLAM shaped what learned perception was trained to do.
Learning-Based Robot Perception influenced by Place Recognition and Loop Closure
Place recognition's difficulty with appearance change and false matches motivated learned visual representations for matching places. Learning-based perception adopted loop-closure and place-recognition as a target task.
Beyond the field
Where Robot Perception And Slam's ideas came from in other fields, and what they led to there: the strongest 12 of 27 links.
Probabilistic Spatial Uncertainty Estimation influenced by Stochastic estimation and Kalman filtering in Control Theory
The Kalman filter and the probabilistic state estimation formalism of control theory supplied the mathematical apparatus for representing and propagating spatial uncertainty in robotics. Early work on uncertain spatial relationships in robotics took this estimation framework and applied it to robot pose and landmark uncertainty.
Carried by Randall Smith; Peter Cheeseman.
Probabilistic Spatial Uncertainty Estimation influenced by Stochastic estimation and Kalman filtering in Control Systems
Probabilistic robot localization and mapping adopted the Kalman filter and its covariance propagation directly from stochastic estimation and control theory. Smith and Cheeseman's 1986 work on representing spatial relationships applied this estimation machinery to robot maps, making the filter the backbone of early probabilistic spatial uncertainty estimation.
Carried by Randall Smith; Peter Cheeseman.
Kalman Filter and EKF-SLAM influenced by Kalman filter in Signal Processing
EKF-SLAM is the Kalman filter applied to the joint robot-landmark state, so the filter's linearization and covariance update are the core machinery of the framework. Smith, Self and Cheeseman's 1987-1990 work on estimating spatial relationships established this application directly on Kalman filtering.
Carried by Randall Smith; Matthew Self; Peter Cheeseman.
Kalman Filter and EKF-SLAM influenced by Stochastic estimation and Kalman filtering in Control Theory
The Kalman filter formalism from control theory, including the extended Kalman filter for nonlinear systems, was carried into SLAM as its estimation engine. Early SLAM formulations applied this control-theoretic filter to the joint estimation of robot pose and landmark positions.
Carried by Randall Smith; Peter Cheeseman.
Kalman Filter and EKF-SLAM influenced by Stochastic estimation and Kalman filtering in Control Systems
EKF-SLAM took the extended Kalman filter from stochastic estimation and control systems and applied it to the simultaneous localization and mapping problem. The state augmentation and covariance propagation used in SLAM are the standard estimation-theory techniques.
Carried by Randall Smith; Peter Cheeseman.
Monte Carlo localization influenced by Monte Carlo method in Computational Physics
Monte Carlo localization samples particle sets to represent the belief over robot pose, directly applying the Monte Carlo method of drawing random samples to estimate quantities. The particle filter used in MCL is a sequential Monte Carlo method.
Carried by Dieter Fox; Wolfram Burgard; Sebastian Thrun.
Visual SLAM influenced Room-scale and inside-out tracking in Virtual Reality
Inside-out tracking in consumer VR headsets was built directly on visual SLAM methods developed in robotics, using camera-based mapping and localisation to track the headset in a room. The SLAM algorithms and their engineering practice passed into VR tracking systems.
Visual-Inertial Fusion influenced Room-scale and inside-out tracking in Virtual Reality
Visual-inertial fusion, combining cameras with inertial measurement units, is the core technique of inside-out head and controller tracking in room-scale VR. The robotics and SLAM literature on visual-inertial odometry supplied the estimators used in these systems.
Visual-Inertial Fusion influenced by Inertial navigation system in Guidance Navigation Control
Visual-inertial fusion takes the inertial measurement unit, its error models and strapdown mechanisation directly from inertial navigation. The IMU preintegration and bias modelling used in VIO descend from inertial navigation practice.
Visual-Inertial Fusion influenced by Sensor Fusion and Kalman Filtering in Guidance Navigation Control
Visual-inertial fusion is built on the extended Kalman filter and related sensor-fusion estimators developed in guidance and navigation. The MSCKF and later visual-inertial estimators apply the same recursive state-estimation machinery to combine camera and IMU measurements.
Carried by Rudolph E. Kalman; Anastasios Mourikis; Stergios Roumeliotis.
Learning-Based Robot Perception grew out of Robot learning in Robotics
Learning-based robot perception is a branch of robot learning, applying its learning algorithms to perceptual tasks such as object detection and pose estimation. The methods and benchmarks grew directly out of the robot-learning community.
Probabilistic Spatial Uncertainty Estimation influenced Probabilistic Manipulation under Uncertainty in Robot Manipulation
Probabilistic spatial uncertainty estimation supplied the belief-state and covariance representations that manipulation under uncertainty uses to reason about object pose and contact uncertainty.

Robot Perception And Slam is a planet in Nousfarer
Its city has a tower for each of these 9 frameworks. Land, walk in, and climb one concept at a time.
Land on Robot Perception And SlamRelated fields
Fields its ideas are linked to
- Virtual Reality 3 links
- Robot Learning 3 links
- Guidance Navigation Control 2 links
- Control Theory 2 links
- Control Systems 2 links
- Robotics 2 links
- Computer Vision 1 link
- Randomized Algorithms 1 link
- Perception 1 link
- Signal Processing 1 link
- Operations Research 1 link
- Systems Simulation 1 link
More in Robotics
- Human Robot Interaction
- Mobile Robotics
- Multi Robot Systems
- Robot Control
- Robot Kinematics Dynamics
- Robot Learning
- Robot Manipulation
- Robot Motion Planning
- Robotics
See where Robot Perception And Slam sits on the Map of Human Knowledge