Co-LMLM: Continuous-Query Limited Memory Language Models
Paper Guide Brief
Reading Brief
The paper proposes Co-LMLM, a limited memory language model that uses continuous vector queries to retrieve factual knowledge from a key-value store, replacing prior relational KB approaches. It introduces a scalable annotation pipeline for arbitrary text and demonstrates improved perplexity and factual precision over baselines.
Central Claim
A novel continuous-query limited memory language model (Co-LMLM) that externalizes factual knowledge into a dense retrieval index with continuous keys and textual values, trained jointly with a contrastive retrieval objective and a scalable annotation pipeline.
Contribution
A novel continuous-query limited memory language model (Co-LMLM) that externalizes factual knowledge into a dense retrieval index with continuous keys and textual values, trained jointly with a contrastive retrieval objective and a scalable annotation pipeline.
Why It Matters
By replacing relational queries with continuous vector queries emitted from the model's hidden state, Co-LMLM enables more flexible, lower-cost retrieval and scales to arbitrary text corpora beyond Wikipedia, achieving strong factual precision and perplexity at modest model sizes.
Prerequisites
limited memory language model, continuous query, dense retrieval, contrastive learning, InfoNCE loss
Atlas Placement
Natural Language Processing (subfield)
Read If
You care about limited memory language model, continuous query, dense retrieval.
Skip If
You only care about TriviaQA, PopQA.
Noosaga Placements
- The paper focuses on language model pretraining, knowledge retrieval, and factual precision, all core to NLP. The primary arXiv category is cs.CL.Limited memory language models (LMLMs) externalize factual knowledge during pretraining to a knowledge base (KB), rather than memorizing it in their weights.We propose continuous-query LMLM (CO-LMLM), where the KB pairs continuous keys with textual knowledge values.Across pretraining on Wikipedia and FineWeb-Edu and at multiple model scales, CO-LMLM outperforms prior LMLMs and vanilla LLMs in both perplexity and factual precision.
- Large Language Modelsframework90%The paper is situated within the Large Language Models framework, as it proposes a new variant of LLM that externalizes knowledge, comparing against standard LLMs and prior LMLMs.Recently, there has been increasing interest in large language models (LLMs) that are trained to externalize knowledge.CO-LMLM outperforms prior LMLMs and vanilla LLMs in both perplexity and factual precision.
- Transformer Architectureframework85%The model is a decoder-only Transformer, which is the core architecture used for both language modeling and retrieval.The CO-LMLM model is a decoder-only Transformer, which functions as both a language model and a dense retriever.
- The method uses a decoder-only Transformer, contrastive learning, and joint training of language modeling and retrieval, which are deep learning techniques.The CO-LMLM model is a decoder-only Transformer, which functions as both a language model and a dense retriever.We jointly minimize next-token prediction (NTP) and bidirectional contrastive losses.We use the InfoNCE loss [van den Oord et al., 2018], compute on the current batch.
- Pretrain-Finetune Frameworkframework70%The paper extends the pretrain-finetune framework by introducing a joint training objective that includes a contrastive retrieval loss alongside next-token prediction.We jointly minimize next-token prediction (NTP) and bidirectional contrastive losses.We optimize the joint loss L = LNTP + λLCL, with λ a hyperparameter set to 0.25 by default.
- The paper deals with externalizing factual knowledge into a KB and retrieving it, which touches on knowledge representation, though the focus is on the retrieval mechanism rather than formal reasoning.Limited memory language models (LMLMs) externalize factual knowledge during pretraining to a knowledge base (KB), rather than memorizing it in their weights.The KB is interpretable and easily editable, allowing for unlearning with no utility tradeoff, and enabling easy attribution of knowledge used to the source material.
Abstract
Limited memory language models (LMLMs) externalize factual knowledge during pretraining to a knowledge base (KB), rather than memorizing it in their weights. During generation, the model then fetches knowledge from the KB as needed. This recently introduced paradigm provides multiple advantages, including knowledge control capabilities that remain beyond conventional LLMs. We propose continuous-query LMLM (CO-LMLM), where the KB pairs continuous keys with textual knowledge values, a significant departure from prior reliance on relational KB and queries. CO-LMLM generates flexible vector queries at minimal cost, while still integrating human-readable and attributable retrieved knowledge into its generation. We pair this design with an annotation pipeline that tags free-form factual spans in arbitrary text, removing prior work's restriction to Wikipedia. Across pretraining on Wikipedia and FineWeb-Edu and at multiple model scales, CO-LMLM outperforms prior LMLMs and vanilla LLMs in both perplexity and factual precision. At 360M scale, this includes lower perplexity than models pretrained on 40x more data, and SimpleQA-verified performance that is in line with gpt-4o-mini and higher than Claude Sonnet 4.5.
Paper Context
Classified from the full extracted paper text (106,167 characters). The Paper Guide brief above is the user-facing synthesis; raw context is kept out of the page.
Full-paper context sent 106,167 of 106,167 extracted characters to classification.