UrbanGround: From Local Perception to Spatial Agency in a Real-Scale City

Noosaga Placements

• Computer Visionsubfield90%
  The paper focuses on evaluating MLLM agents' visual perception and spatial reasoning in a photorealistic urban environment, with tasks centered on visual recognition and scene understanding.
  Multimodal large language models (MLLMs) can interpret a street viewtest whether an agent can ground a local scene well enough to answer spatial questions after active observation
• Supervised Deep Learningframework80%
  The benchmark evaluates MLLM agents that are trained with supervised learning on large-scale multimodal data, and the tasks involve supervised visual recognition.
  Visual Recognition (VR)multimodal large language models
• Artificial Intelligencesubfield70%
  The work evaluates general AI agents (MLLMs) on embodied navigation and planning tasks, contributing to the broader study of AI agency in complex environments.
  how far current MLLM agents can turn local urban perception into reliable actionsupport broader study of how far current MLLM agents can explore reliably
• Learning-Based Roboticsframework70%
  The agents are learning-based MLLMs that use perception and reasoning to act in the environment, fitting the learning-based robotics paradigm.
  MLLM agentsclosed-loop interaction from a first-person view
• Planning And Searchsubfield60%
  The benchmark includes tasks requiring route planning, scheduling, and replanning under constraints, which are core to planning and search.
  Multi-stop Route PlanningTime-window SchedulingDynamic Road-closure Replanning
• Self-Supervised Learningframework50%
  MLLMs often rely on self-supervised pretraining, and the benchmark tests their ability to generalize to novel urban environments without task-specific fine-tuning.
  current MLLM agentsexplore reliably in complex, open-ended urban environments
• Markov Decision Processesframework40%
  Navigation tasks can be modeled as Markov decision processes, and the benchmark implicitly evaluates agents' ability to make sequential decisions under uncertainty.
  closed-loop interactiongoal-directed behavior
• Multiagent Systemssubfield30%
  The paper evaluates single-agent navigation, but the presence of pedestrians and dynamic obstacles touches on multi-agent interaction, though not the central focus.
  Navigation among Pedestrians

Abstract

Multimodal large language models (MLLMs) can interpret a street view, but urban agency depends on whether such local evidence remains useful after the agent starts to move. In this paper, we investigate how far current MLLM agents can turn local urban perception into reliable action in a complicated real-scale city. We propose UrbanGround, the first sandbox to make this question testable in a physically constrained replica of Hong Kong built from territory-wide 3D geospatial data. UrbanGround supports closed-loop interaction from a first-person view and provides an interactive map for navigation. Agents can directly enter the 3D city and explore from a first-person view. Our analysis follows the growth of the spatial problem through three research questions. We first test whether an agent can ground a local scene well enough to answer spatial questions after active observation. Then we ask whether that grounding supports navigation as destinations become farther away and less explicit. Finally, we examine whether the resulting behavior survives changes in route availability and pedestrian motion. Contemporary MLLM agents usually show useful atomic abilities in visual recognition and short-range spatial reasoning, while orientation and pedestrian-aware movement remain unreliable. Their central failure emerges over extended exploration, where local abilities do not compose into sustained goal-directed behavior and errors accumulate without effective correction. We hope UrbanGround will support broader study of how far current MLLM agents can explore reliably in complex, open-ended urban environments.

Paper Context