WorldDirector: Building Controllable World Simulators with Persistent Dynamic Memory

Noosaga Placements

• Diffusion Modelsframework95%
  The core video generation model is based on diffusion models. The paper explicitly states it uses a diffusion transformer (DiT) and follows the flow matching framework, which is a type of diffusion/flow-based generative model.
  These embeddings are subsequently injected into each Diffusion Transformer (DiT) block via an adaptive normalization layer.We follow the flow matching framework [31, 17] to perform post-training using the mean squared error (MSE) loss.
• Computer Visionsubfield95%
  The paper presents a video generation and world simulation framework, which is a core computer vision task. It focuses on video synthesis, object tracking, appearance consistency, and camera control, all within the computer vision domain.
  We present WorldDirector, a highly controllable video world model framework designed for persistent dynamic object memory and unrestricted viewpoint exploration.arXiv primary category: cs.CV
• Deep Learningsubfield90%
  The method heavily relies on deep learning components: a diffusion transformer (DiT) as the backbone, an LLM for planning, and a 3D VAE for encoding. The training uses flow matching and the model is built on a pre-trained foundation model (LingBot-World-Base).
  We build WorldDirector on the pre-trained LingBot-World-Base model [38].We follow the flow matching framework [31, 17] to perform post-training using the mean squared error (MSE) loss.These embeddings are subsequently injected into each Diffusion Transformer (DiT) block via an adaptive normalization layer.
• Vision Transformers and Foundation Modelsframework90%
  The model is built on a pre-trained foundation model (LingBot-World-Base) and uses a Diffusion Transformer (DiT) architecture, which falls under Vision Transformers and Foundation Models. The paper also uses a 3D VAE and other transformer-based components.
  We build WorldDirector on the pre-trained LingBot-World-Base model [38].These embeddings are subsequently injected into each Diffusion Transformer (DiT) block via an adaptive normalization layer.
• Large Language Modelsframework85%
  The framework uses an LLM (Gemini) as a central orchestrator for planning 3D trajectories and camera movements. This is a direct use of a Large Language Model for a planning task within the video generation pipeline.
  By leveraging an LLM to coordinate 3D trajectories with camera movements...We employ Gemini [37] as the core semantic engine for world planning.
• Natural Language Processingsubfield70%
  The framework uses an LLM (Gemini) as a central orchestrator to translate user instructions into 3D trajectories and camera movements. It also uses Qwen2.5-VL-72B for generating fine-grained textual captions of entity dynamics.
  By leveraging an LLM to coordinate 3D trajectories with camera movements...We employ Gemini [37] as the core semantic engine for world planning.We leverage SAM [26] and DepthAnything v2 [47] to roughly estimate the 3D bounding boxes... This structured information is then fed into the LLM, prompting it to analytically plan the corresponding 3D trajectories...
• Representation Learningframework70%
  The paper introduces an Appearance Binding mechanism that injects RGB dynamic object features from context as visual anchors, which is a form of representation learning for maintaining identity consistency. The model also learns to represent dynamic object states.
  To prevent identity distortion when a hidden entity re-enters the frame, we propose an Appearance Binding mechanism that injects RGB dynamic object features from context as visual anchors.
• Planning And Searchsubfield60%
  The LLM performs trajectory planning for dynamic objects and camera movements, which is a form of motion planning. The paper explicitly uses the term 'World Planning via LLM' and describes generating 3D bounding box trajectories and camera paths.
  World Planning via LLM. We first estimate the 3D bounding boxes of target dynamic objects... which then forecasts continuous 3D box trajectories—comprising both spatial coordinates and orientations—alongside our designed camera path.By leveraging an LLM to coordinate 3D trajectories with camera movements...

Abstract

We present WorldDirector, a highly controllable video world model framework designed for persistent dynamic object memory and unrestricted viewpoint exploration. Unlike existing world models that entangle physical dynamics with pixel rendering and rely on continuous visual observation to sustain motion, our framework explicitly decouples semantic motion orchestration from visual generation. By leveraging an LLM to coordinate 3D trajectories with camera movements and subsequently employing these orchestrated trajectories as control signals for video generation, our approach ensures strict physical logic and appearance stability, successfully preserving the exact visual identities of dynamic entities even when they re-enter the scene after prolonged periods out of view. Experimental results demonstrate that our method supports the synthesis of complex and extended events with unprecedented controllability and persistent dynamic object memory. Project Page: https://worlddirector.github.io/

Paper Context