Consolidating RLVR Capabilities Across Domains: A Deep Dive into Fusion Paradigms

Noosaga Placements

• Natural Language Processingsubfield90%
  The paper focuses on large language models and their post-training via RLVR, which is a core NLP topic.
  Reinforcement learning with verifiable rewards (RLVR) improves specific capabilities of large language modelsarXiv:2608.27409v1 [cs.CL]
• Deep Reinforcement Learningframework90%
  The paper uses GRPO, a deep reinforcement learning algorithm, for RLVR training.
  GRPORollout batch size 128, Mini batch size 128
• Reinforcement Learningsubfield90%
  The paper studies RLVR, a reinforcement learning paradigm, and compares RL-based fusion methods.
  Reinforcement learning with verifiable rewards (RLVR)Mix RL depends on domain mixture proportionsGRPO
• Reinforcement Learningframework80%
  The paper studies reinforcement learning with verifiable rewards, which falls under the RL framework.
  Reinforcement learning with verifiable rewards (RLVR)Mix RL depends on domain mixture proportions
• Machine Learningsubfield80%
  The paper compares model merging and distillation techniques, which are machine learning methods for combining models.
  Merge combines expert task vectorsMix RL pools their datasetsmulti-teacher on-policy distillation (MOPD)
• Ensemble Methodsframework70%
  The paper compares model merging methods, which are ensemble-like techniques for combining multiple models.
  Merge combines expert task vectorsTask Arithmetic (TA)TIES-Merging
• Deep Learningsubfield60%
  The paper uses deep learning models (LLMs) and techniques like LoRA, which are deep learning methods.
  LoRA-based RLQwen3-4B-Instruct-2507Qwen3-8B (non-thinking)
• Statistical Learning Theoryframework50%
  The paper evaluates generalization and held-out performance, which relates to statistical learning theory.
  held-out capabilitiesgeneralization

Abstract

Reinforcement learning with verifiable rewards (RLVR) improves specific capabilities of large language models, but covering multiple capabilities often involves training separate domain experts and subsequently consolidating them. We organize three fusion paradigms by the artefacts they reuse: Merge combines expert task vectors, Mix RL pools their datasets, and multi-teacher on-policy distillation (MOPD) uses both. Because they have largely been studied in isolation, how they compare and how to choose among them remain unclear. We compare all three using shared experts and data across model scales and a multi-domain benchmark suite. Although their average performance differs by at most 1.4 points, the gap reaches 8.6 points on a single benchmark, with domain-level variation tracking cross-domain relations visible in task-vector geometry. Training dynamics expose distinct constraints: Mix RL depends on domain mixture proportions, MOPD remains bounded by its teachers, and Merge compresses all expert updates into one. All three improve single-sample accuracy without measurable gains in solution coverage or losses in held-out capabilities. These results yield a practical guideline: use Merge when experts already exist and cheap fusion is paramount; Mix RL when training a unified model without experts, with domain proportions adjusted for cross-domain transfer; and MOPD when preserving domain-specific gains matters more than surpassing teachers or minimizing end-to-end cost.

Paper Context