Latent Memory Palace: Reasoning for Control as Autoregressive Variational Inference

Noosaga Placements

• Reinforcement Learningsubfield95%
  The paper derives a latent-space reinforcement learning technique (PPO-style clipped surrogate) to optimize the variational lower bound, and the resulting policy is evaluated on control tasks. The core optimization is framed as an RL problem over latent token trajectories.
  We derive a latent-space reinforcement learning technique to tractably optimize its variational lower bound.To tractably optimize through the variable-length autoregressive sampling process, we convert the variational lower bound into a reinforcement learning (RL) objective over latent trajectories and apply RL techniques.Optimization with a PPO-style clipped surrogate.
• Reinforcement Learningframework95%
  The paper uses reinforcement learning (specifically a PPO-style clipped surrogate) to optimize the variational lower bound over latent token trajectories. This is a core part of the method.
  We derive a latent-space reinforcement learning technique to tractably optimize its variational lower bound.Optimization with a PPO-style clipped surrogate.We collect on-policy latent rollouts into a buffer and optimize the objective using a trust-region style surrogate.
• Robot Learningsubfield95%
  The method is applied to robot learning tasks (behavior cloning from demonstrations) and evaluated on real-world and simulated robot manipulation benchmarks. The paper explicitly addresses robotic control policies.
  We ask: can iterative, adaptive computation benefit sequential decision making problems such as robotics?We evaluate language-conditioned multitask policies in zero-shot and finetuned settings.LMP-π yields control policies capable of iterative latent reasoning, with consistent performance gains across a range of robot learning tasks in both simulation and the real world.
• Probabilistic Machine Learningframework90%
  The paper formulates reasoning as variational inference, a core probabilistic machine learning technique. It uses latent variable models, evidence lower bounds, and KL divergence.
  Our method, Latent Memory Palace (LMP), formulates reasoning as variational inference with an autoregressive latent distribution.We derive a latent-space reinforcement learning technique to tractably optimize its variational lower bound.Policy learning then amounts to maximum likelihood estimation of p(a|o). Since this involves an intractable marginalization over z, we approximate the true posterior p(z|o,a) with a variational distribution qθ(z|o,a) and maximize the evidence lower bound (ELBO).
• Deep Learningsubfield80%
  The method uses deep neural networks (transformers with cross-attention) and is inspired by reasoning in large language models. The latent space is learned end-to-end using deep learning techniques.
  We parameterize the prior pθ(z|o) and the posterior qθ(z|o,a) using a shared causal transformer with cross-attention conditioning.The action decoder is a bidirectional transformer conditioned on the observation embeddings and the sampled latent tokens.
• Actor-Critic Methodsframework70%
  The paper uses a PPO-style clipped surrogate, which is an actor-critic method. The latent 'policy' (posterior) is optimized using this method.
  Optimization with a PPO-style clipped surrogate.We collect on-policy latent rollouts into a buffer and optimize the objective using a trust-region style surrogate [27].
• Probabilistic Aisubfield70%
  The paper formulates reasoning as variational inference, a core probabilistic AI technique. It uses probabilistic latent variable models and evidence lower bounds.
  Our method, Latent Memory Palace (LMP), formulates reasoning as variational inference with an autoregressive latent distribution.We derive a latent-space reinforcement learning technique to tractably optimize its variational lower bound.
• Statistical Learningframework60%
  The paper is situated within the statistical learning framework of variational inference and maximum likelihood estimation. The method is a form of statistical learning for control policies.
  We consider the imitation learning setting, where the goal is to learn a control policy πθ: O → A from a dataset of expert demonstrations.Policy learning then amounts to maximum likelihood estimation of p(a|o).

Abstract

Human decision-making is highly flexible -- some actions are taken immediately; others require longer deliberation. Language models have exhibited a similar capacity for adaptive "reasoning." However, transferring this capability to continuous control policies has been challenging, as directly reasoning in language space may lack the granularity for spatial understanding and precise motions. In this work, we show that reasoning for control policies can emerge by organizing information in an autoregressive latent space reminiscent of a memory palace, where retrieval is iterative and adaptive. Our method, Latent Memory Palace (LMP), formulates reasoning as variational inference with an autoregressive latent distribution. We derive a latent-space reinforcement learning technique to tractably optimize its variational lower bound. The resulting policy, LMP-$π$, achieves strong empirical performance in simulation and real-world domains while exhibiting interpretable, adaptive allocation of test-time compute. We further show that the same framework yields a variable-length action tokenizer, LMP-$\texttt{tok}$, which significantly improves the performance of downstream autoregressive policies. Together, these results present a new perspective on latent reasoning for control through the lens of variational inference.

Paper Context