MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agents

Noosaga Placements

• Computer Visionsubfield90%
  The paper focuses on evaluating agents that process visual inputs (images, UI screenshots) and extract information from them to perform tool calls. The primary arXiv category is cs.CV, and the core challenge is visual grounding and perception.
  We introduce MM-ToolSandBox, a benchmark and evaluation framework for visually grounded tool-calling agents.Our failure analysis further reveals that visual precision, not only planning, is a primary bottleneck for capable models: 53% of failures stem from incorrect information extraction from images despite otherwise correct task workflows.primary_category: cs.CV
• Vision-Language Modelsframework80%
  The paper evaluates vision-language models (VLMs) on visual tool-calling tasks. The benchmark is designed to test the visual grounding and reasoning capabilities of multimodal models.
  Evaluating 12 state-of-the-art models, from 4B open-weight to frontier proprietary systemsThe framework provides a stateful execution environment spanning 500+ tools across 16 application domains, supporting multi-image, multi-turn tasks where agents must ground progressively arriving visual inputs into executable tool calls
• Artificial Intelligencesubfield85%
  The paper evaluates AI agents that combine perception, reasoning, planning, and tool use in a multi-turn conversational setting. It addresses core AI challenges of agentic behavior, planning, and tool invocation.
  Evaluating 12 state-of-the-art models, from 4B open-weight to frontier proprietary systems, shows that current models still lack robust visual tool-calling capability.A planning-to-precision crossover emerges with scale: smaller models fail at deciding what to do, while larger models fail at perceiving what they see.Solving visual tool-calling tasks requires more than visual recognition. The agent must extract task-relevant information from images, map visual evidence to the correct tool or code action, execute that action against a stateful environment, and continue the interaction when the user's intent evolves.
• Large Language Modelsframework70%
  The paper evaluates large language models (LLMs) as agents that use tools and interact with users. The benchmark tests LLM-based agents on tool-augmented tasks with visual inputs.
  Evaluating LLM-based agents on tool-augmented tasks has become an active area of researchAll models share a single fixed harness... comparing models under a common, well-specified harness is what makes performance differences attributable to the model rather than to the surrounding system.
• Natural Language Processingsubfield70%
  The benchmark involves multi-turn conversational interaction between a user simulator and an agent, requiring natural language understanding and generation. The agent must handle goal revisions, error corrections, and state mutations in dialogue.
  supporting multi-image, multi-turn tasks where agents must ground progressively arriving visual inputs into executable tool calls while handling realistic conversational phenomena (goal revisions, error corrections, state mutations).The front-end interaction between the agent and user is a natural user-assistant dialogue, while the backend interaction with the environment is mediated through typed tool calls.
• Deep Learningframework60%
  The paper evaluates deep learning models (including GPT-5.4, Claude 4.5, Gemini) on a benchmark. The analysis of model scaling and reasoning budget relates to deep learning.
  Evaluating 12 state-of-the-art models, from 4B open-weight to frontier proprietary systemsGPT-5.4 shows a clear gain when native thinking is enabled
• Deep Learningsubfield60%
  The paper evaluates deep learning models (LLMs, VLMs) including GPT-5.4, Claude 4.5, Gemini, Qwen, etc. The analysis of model scaling and reasoning budget relates to deep learning architectures.
  Evaluating 12 state-of-the-art models, from 4B open-weight to frontier proprietary systemsGPT-5.4 shows a clear gain when native thinking is enabled: Agent Success Rate improves from 29.1% without thinking to 36.1% with medium thinking and 41.5% with high thinking.Scaling model size improves task completion significantly.

Abstract

We introduce MM-ToolSandBox, a benchmark and evaluation framework for visually grounded tool-calling agents. The framework provides a stateful execution environment spanning 500+ tools across 16 application domains, supporting multi-image, multi-turn tasks where agents must ground progressively arriving visual inputs into executable tool calls while handling realistic conversational phenomena (goal revisions, error corrections, state mutations). An automated scenario generation pipeline produces diverse, visually grounded scenarios through information-flow-guided planning and multi-stage quality filtering, yielding 258 human-verified nominal scenarios and 50 variants targeting interactive UI applications. Evaluating 12 state-of-the-art models, from 4B open-weight to frontier proprietary systems, shows that current models still lack robust visual tool-calling capability: even the best model achieves below 50% success rate. Our failure analysis further reveals that visual precision, not only planning, is a primary bottleneck for capable models: 53% of failures stem from incorrect information extraction from images despite otherwise correct task workflows. A planning-to-precision crossover emerges with scale: smaller models fail at deciding what to do, while larger models fail at perceiving what they see, suggesting fundamentally different research directions for improving models at different capability levels. The framework and the benchmark are publicly available at https://github.com/apple/ml-mmtoolsandbox

Paper Context