BTS-AgentBench: A Deterministic, Replayable Pipeline from Read-Only Telemetry Logs to Agent Benchmarks
Paper Guide Brief
Reading Brief
This paper introduces BTS-AgentBench, a benchmark construction method that compiles read-only building telemetry logs into multi-turn LLM agent evaluation episodes. The pipeline normalizes raw telemetry into a tool store, builds static executable tasks, and lifts them into typed operator-facing episodes with deterministic scoring. The released 532-row benchmark is validated via exact replay and a controller-aware acceptance rule, and portability is demonstrated on the XAI4HEAT corpus.
Central Claim
Presents a deterministic, replayable pipeline for converting raw telemetry logs into multi-turn agent benchmark episodes, with a released 532-row benchmark (BTS-AgentBench) and a portability study on XAI4HEAT.
Contribution
Presents a deterministic, replayable pipeline for converting raw telemetry logs into multi-turn agent benchmark episodes, with a released 532-row benchmark (BTS-AgentBench) and a portability study on XAI4HEAT.
Why It Matters
The work is novel in treating industrial telemetry logs as reusable substrates for deterministic benchmark construction, separating source computation, interaction contracts, and surface realization, and using a controller-aware acceptance rule to harden the release.
Prerequisites
multi-turn agent evaluation, telemetry analysis, tool use, quality-aware reporting, timestamp policy
Atlas Placement
Natural Language Processing (subfield)
Read If
You care about multi-turn agent evaluation, telemetry analysis, tool use.
Skip If
You only care about BTS-AgentBench, ITBench.
Noosaga Placements
- The work involves natural language rendering of tasks and evaluation of language model outputs.Deterministic renderers lexicalize the typed fields as bounded operator turns.Natural-language answers are normalized against declared action expressions.
Abstract
Industrial sites contain large volumes of read-only telemetry, but few benchmarks specify how to compile these records into executable multi-turn agent tasks. We present a telemetry-to-episode construction method instantiated as BTS-AgentBench. The pipeline normalizes BTS metadata and raw histories into a read-only tool store, compiles static tasks with tool-derived gold answers and evidence, and lifts retained tasks into typed, bounded operator-facing episodes. The 532-row release adds clarification, goal revision, timestamp policy, quality-gated reporting, and evidence attribution while preserving the source computation and split. Coded contract preflight reports zero findings, and the construction-exclusion controller completes 0/532 rows. Two independent raw-to-episode builds match all 11 logical tool-store exports and reproduce the released 356/87/89 train/dev/test artifact exactly. Applying the shared construction path to XAI4HEAT produces 204 episodes; on its 41-row held-out test split, the controller completes 0 rows and the retained GPT-5.5 execution completes all 41. Code, artifacts, and replay reports are available at https://github.com/kjy7567/BTS-AgentBench.
Paper Context
Classified from the full extracted paper text (62,857 characters). The Paper Guide brief above is the user-facing synthesis; raw context is kept out of the page.
Full-paper context sent 62,857 of 62,857 extracted characters to classification.