The Illusion of Equivalency: Statistical Characterization of Quantization Effects in LLMs

Noosaga Placements

• Natural Language Processingsubfield90%
  The paper focuses on evaluating behavioral changes in large language models (LLMs) under quantization, using NLP benchmarks (HellaSwag, Winogrande, ARC) and language modeling perplexity (WikiText-2, C4). The core object of study is LLM behavior, which falls under natural language processing.
  Post-training quantization is widely used to deploy large language models in resource-constrained settingsWe evaluate quantized models using both language modeling corpora and downstream benchmark tasksWe use zero-shot benchmarks including HellaSwag, Winogrande, and ARC
• Large Language Modelsframework90%
  The paper studies post-training quantization effects on large language models (LLMs), which are the central object of the Large Language Models framework. The analysis is situated within the context of LLM deployment and behavioral evaluation.
  Post-training quantization is widely used to deploy large language models in resource-constrained settingsWe quantize the following four models: Llama-3.2-3B, Vicuna-7B-v1.5, Mistral-7B-v0.1, and Llama-3.1-8BOur core object of study is the joint effect of Tc on internal attention weights and on decision behavior over D
• Deep Learningsubfield80%
  The paper analyzes attention weight distortions in transformer-based LLMs, which are deep learning models. It studies structural changes in attention projection matrices (Q, K, V, O) under quantization, a core deep learning concern.
  We analyze quantization as a structural operator on attention weightsquery and key projections are consistently more sensitive than value and output projectionsA decoder-only transformer with N blocks and hidden width d
• Transformer Architectureframework80%
  The paper analyzes attention weight distortions in transformer-based LLMs, specifically focusing on query, key, value, and output projections. The Transformer Architecture framework is directly relevant as the paper studies structural changes in transformer attention mechanisms under quantization.
  We analyze quantization as a structural operator on attention weightsFor block index i in [N], let the self-attention projection matrices be WiQ, WiK, WiV, WiOquery and key projections are consistently more sensitive than value and output projections
• Ai Safetysubfield60%
  The paper exposes an 'illusion of equivalence' between base and quantized models, highlighting that behavioral consistency is not preserved even when task performance appears intact. This has implications for reliability and safety of deployed quantized models.
  These findings expose an illusion of equivalence between base and quantized modelsquantized variants do not reliably reproduce base-model behavior, even when accuracy or perplexity appears preservedmotivate behavioral evaluation beyond conventional performance metrics
• Robustness and Assuranceframework50%
  The paper's findings about behavioral divergence under quantization relate to robustness of LLMs, which is a concern within the Robustness and Assurance framework. However, the paper does not directly propose robustness methods or assurance techniques.
  behavioral divergence emerges under moderate quantization even when task performance appears preservedquantized variants do not reliably reproduce base-model behavior

Abstract

Post-training quantization is widely used to deploy large language models in resource-constrained settings, yet its evaluation relies almost exclusively on accuracy and perplexity. We show that these metrics fail to capture behavioral changes induced by quantization. We introduce correctness agreement, a decision-level metric that measures overlap in correct predictions between a base model and its quantized variants, independent of absolute accuracy. Across multiple models and quantization schemes from 8-bit to 2-bit, we find that behavioral divergence emerges under moderate quantization even when task performance appears preserved. To explain this effect, we analyze quantization as a structural operator on attention weights and quantify layer-wise distortions using statistical and distributional measures. Our results reveal non-linear breakpoints at low bit-widths and show that query and key projections are consistently more sensitive than value and output projections. These findings expose an illusion of equivalence between base and quantized models and motivate behavioral evaluation beyond conventional performance metrics.

Paper Context