- Atlas
- Engineering
- Information Theory
Information Theory
Information theory is the mathematical study of information: how much of it a message carries, how much a noisy channel can deliver, and how far a message can be compressed without losing what matters.
9 frameworks, from 1948 to today · Electrical Engineering · Engineering
- Shannon Theory of Communication 1948-Present
- Algebraic Coding Theory 1950-Present
- Rate-Distortion Theory 1959-Present
- Network Information Theory 1961-Present
- Universal Source Coding 1977-Present
- Minimum Description Length 1978-Present
- Iterative and Sparse-Graph Coding 1993-Present
- Quantum Shannon Theory 1993-Present
- Finite-Blocklength Information Theory 2010-Present
Overview
Information theory is the mathematical study of information: how much of it a message carries, how much a noisy channel can deliver, and how far a message can be compressed without losing what matters1. Its central insight is that information is not a vague quality but a measurable quantity, and that measurement sets hard limits on what any communication system can do12. Those limits are what make the field an engineering discipline as much as a mathematical one: they tell designers what is possible before a single circuit is built. The history below follows the field from its founding in the 1940s through the coding revolutions that turned its theorems into working technology.
Shannon and the Birth of a Science of Communication
Before the 1940s, communication engineering was a craft of signal-to-noise ratios and bandwidth budgets. Engineers knew that a stronger signal or a wider channel helped, but had no general theory of how much information a channel could carry3. Nor was there a rigorous notion of what "information" in a message even was. Norbert Wiener's work on cybernetics in the 1940s touched related questions, and earlier work by Harry Nyquist and Ralph Hartley at Bell Labs had sketched capacity ideas, but the field lacked a unifying framework.
Claude Shannon supplied it in "A Mathematical Theory of Communication," published in the Bell System Technical Journal in 194814. Shannon's move was to treat a message source as a random process and to define its entropy—a measure, in bits, of the average uncertainty of its output21. A source that emits a fair coin flip has one bit of entropy per symbol; a source that almost always emits the same symbol has almost none1. Entropy gave the field its first precise answer to "how much information is in this?"
Shannon then proved two theorems that became the field's twin pillars1. The source coding theorem says a source can be compressed to its entropy and no further: the entropy is the irreducible description length25. The channel coding theorem says that every noisy channel has a capacity—a maximum rate, in bits per channel use, at which information can be sent with arbitrarily small error probability, provided the encoder is allowed to use long enough blocks167. Below capacity, reliable communication is possible; above it, errors cannot be driven to zero1. This was startling: noise did not merely degrade transmission, it set a threshold, and below that threshold noise could be defeated entirely.
Shannon's 1948 paper also introduced the noisy-channel coding theorem's companion ideas—joint source-channel separation, the rate-distortion function for lossy compression, and the notion of a channel's capacity as a supremum over input distributions. The paper was dense and its proofs non-constructive: it showed that good codes exist without saying how to build them1. That gap defined the next half-century of research.
Rate-Distortion and the Theory of Lossy Compression
Shannon's 1948 paper already contained the seed of lossy compression: if a source need not be reproduced exactly, how few bits suffice? The answer is the rate-distortion function R(D), the minimum rate at which a source can be encoded so that the average distortion stays below D. Shannon developed this in a 1959 paper, "Coding Theorems for a Discrete Source with a Fidelity Criterion," building on earlier joint work with his colleague Warren Weaver's framing of communication levels. The function formalized a trade-off every engineer recognizes: you can have fewer bits or better fidelity, and the curve tells you exactly where the frontier lies.
Rate-distortion theory mattered because it separated the question of what to send from how to send it. It also made clear that lossy compression is not a failure of exactness but a deliberate, quantifiable choice. The theory's practical influence grew slowly, as it required models of source statistics and distortion measures that were hard to specify for real signals like speech and images. Its descendants—transform coding, perceptual distortion measures, and the rate-distortion analyses behind modern video codecs—became central to digital media.
Algebraic Coding Theory: Building Codes That Work
Shannon proved good codes exist; the engineering problem was to construct them1. The first answers came from algebra. Richard Hamming, working at Bell Labs, published his error-correcting codes in 1950: by adding parity bits in a structured pattern, a receiver could detect and correct single-bit errors. Hamming codes were simple and efficient for their purpose, and they established the template of adding redundancy in a designed pattern rather than merely repeating data8.
The 1950s and 1960s brought a wave of algebraic constructions. Reed–Solomon codes, introduced by Irving Reed and Gustave Solomon in 1960, worked over finite fields and could correct bursts of errors; they later became ubiquitous in storage media, QR codes, and deep-space communication. BCH codes, developed independently by Raj Chandra Bose and D. K. Ray-Chaudhuri and by Alexis Hocquenghem in the late 1950s, generalized Hamming's idea to multiple-error correction. Convolutional codes, described by Peter Elias in 1955, encoded a continuous stream rather than fixed blocks and were decoded by the Viterbi algorithm, published by Andrew Viterbi in 1967, which made them practical for satellite and cellular links.
These codes were powerful but, for decades, fell well short of Shannon capacity on noisy channels1. The gap was a standing embarrassment: theory promised reliable communication near capacity, but practical codes needed far more redundancy. Algebraic coding theory also developed its own deep questions—bounds on code size, decoding complexity, and the structure of codes over finite fields—that became a mathematical field in its own right.
Network Information Theory: Many Senders, Many Receivers
Shannon's theory addressed a single source and a single channel1. Real systems involve networks: multiple users sharing a medium, relays, interference, and feedback. Network information theory grew from the 1960s onward to ask what the fundamental limits are when many parties communicate at once. Key results include the multiple-access channel (several senders to one receiver), the broadcast channel (one sender to several receivers), and the relay channel1910. The field's landmark synthesis is Thomas Cover and Joy Thomas's textbook Elements of Information Theory (1991), which organized these results for a generation of students.
Network information theory is famously harder than the point-to-point case: many problems remain open or are solved only for special cases. Its practical importance grew with wireless networks, where interference and cooperation among users are the central design constraints. The theory's capacity regions—the sets of simultaneously achievable rates—give network designers targets and impossibility results, even when the exact regions are unknown.
Universal Source Coding and the Minimum Description Length Principle
Shannon's source coding theorem assumed the source statistics were known. In practice they rarely are. Universal source coding asks how to compress data when the source model is unknown, learning it from the data itself. The Lempel–Ziv algorithms, published by Abraham Lempel and Jacob Ziv in 1977 and 1978, achieved this by building a dictionary of repeated substrings on the fly; they became the basis of the GIF, ZIP, and PNG formats and remain among the most widely used compression methods ever devised.
A parallel development came from statistics and machine learning. The minimum description length (MDL) principle, developed by Jorma Rissanen from the late 1970s, reframed statistical model selection as a compression problem: the best model is the one that yields the shortest description of the data plus the model itself. MDL connected information theory to inductive inference and influenced later work on Kolmogorov complexity and Bayesian model selection. Universal coding and MDL together made clear that compression and learning are two views of the same problem—finding regularity in data.
Iterative and Sparse-Graph Coding: Approaching Capacity
For decades after Shannon, the capacity gap persisted1. The turning point came from codes defined on graphs. Low-density parity-check (LDPC) codes, invented by Robert Gallager in his 1960 MIT doctoral thesis, used sparse parity-check matrices and iterative decoding; they were largely forgotten for thirty years because the decoding was computationally daunting. Turbo codes, introduced by Claude Berrou, Alain Glavieux, and Punya Thitimajshima in 1993, combined two convolutional codes with an interleaver and iterative decoding, and came within a fraction of a decibel of Shannon capacity—a result that stunned the coding community.
Turbo codes revived interest in LDPC codes, which were rediscovered in the late 1990s and shown to perform comparably or better11. The unifying idea was iterative decoding: rather than solving the decoding problem exactly, pass probabilistic messages back and forth between constraints until the estimate converges1112. This approach, formalized as belief propagation on factor graphs, connected coding to statistical physics and to graphical models in machine learning. LDPC and turbo codes now underpin Wi-Fi, 5G, and deep-space links, and the capacity gap that had stood for fifty years was largely closed.
Quantum Shannon Theory
In the 1990s, information theory extended to quantum systems. Quantum Shannon theory asks what happens to Shannon's limits when the channel carries quantum states rather than classical bits13. Key results include the no-cloning theorem (an unknown quantum state cannot be copied), quantum teleportation (using entanglement and classical bits to transmit a quantum state), and quantum channel capacities that can exceed their classical counterparts. The field, developed by researchers including Charles Bennett, Peter Shor, and many others, underlies quantum cryptography and quantum computing's communication protocols. It remains an active frontier, with many capacity questions unresolved.
Finite-Blocklength Information Theory
Shannon's coding theorem is asymptotic: it promises arbitrarily small error only as block length grows without bound5. Real systems have short blocks—a packet, a control message, a sensor reading. Finite-blocklength information theory, developed notably by Yury Polyanskiy, H. Vincent Poor, and Sergio Verdú in a 2010 paper, characterizes the best achievable rate and error probability for finite block lengths, introducing the normal approximation that adds a correction term to capacity depending on block length and error tolerance. This work matters for ultra-reliable low-latency communication in 5G and for industrial control, where waiting for long blocks is not an option.
The Field Today
Information theory today is both a mature mathematical discipline and a live engineering toolkit1. Its core—entropy, capacity, rate-distortion—remains the language in which communication limits are stated114. Its descendants are everywhere: LDPC and polar codes in 5G, rate-distortion analysis in video compression, MDL and universal coding in machine learning, network information theory in wireless design, and quantum Shannon theory in the emerging quantum internet. Open questions persist: exact capacity regions for general networks, finite-blocklength limits for multi-user systems, the information-theoretic foundations of learning and privacy, and the quantum capacities of general channels. The field's founding bet—that information can be measured, and that measurement constrains what any system can do—has held up for more than seventy years and shows no sign of weakening1.
References
- Information theory (introduction, Quantities of information, Historical background, The classic work, Entropy of an information source, Overview, Channel capacity, Coding theory), Wikipedia
- Entropy (information theory), Wikipedia
- Shannon–Hartley theorem (Historical development), Wikipedia
- A Mathematical Theory of Communication, Wikipedia
- Shannon's source coding theorem, Wikipedia
- Noisy-channel coding theorem, Wikipedia
- Channel capacity, Wikipedia
- Hamming code, Wikipedia
- Interference channel, Wikipedia
- Relay channel, Wikipedia
- Low-density parity-check code (introduction, History), Wikipedia
- Belief propagation, Wikipedia
- Quantum information science, Wikipedia
- Rate–distortion theory, Wikipedia
The frameworks of Information Theory
Each framework is a way of working the field has used: when it held, and the question it answers.
- 1948-Present
Shannon Theory of Communication
How much information can a channel carry?
- 1950-Present
- 1959-Present
Rate-Distortion Theory
How few bits can represent data within a distortion limit?
- 1961-Present
Network Information Theory
What are the limits when many users communicate?
- 1977-Present
Universal Source Coding
How can data be compressed without knowing its statistics?
- 1978-Present
- 1993-Present
Iterative and Sparse-Graph Coding
How can codes approach the capacity limit?
- 1993-Present
Quantum Shannon Theory
What are the limits of transmitting quantum information?
- 2010-Present
Finite-Blocklength Information Theory
What are the limits for finite block lengths?
How the ideas connect
Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.
Algebraic Coding Theory grew out of Shannon Theory of Communication
Shannon's 1948 coding theorem proved that good codes exist without constructing them; Hamming's 1950 error-correcting codes and the algebraic constructions that followed were direct attempts to build codes meeting Shannon's promise.
Carried by Claude Shannon; Richard Hamming.
Rate-Distortion Theory grew out of Shannon Theory of Communication
Shannon's 1948 paper introduced the rate-distortion function for lossy compression; he developed it fully in his 1959 paper, making rate-distortion theory a direct extension of his original framework.
Carried by Claude Shannon; Coding Theorems for a Discrete Source with a Fidelity Criterion (1959).
Network Information Theory grew out of Shannon Theory of Communication
Network information theory extends Shannon's point-to-point capacity theorems to multi-user channels, asking for capacity regions when several senders and receivers share a medium.
Carried by Claude Shannon; Thomas M. Cover.
Universal Source Coding grew out of Shannon Theory of Communication
Universal source coding takes up Shannon's source coding theorem but drops the assumption that source statistics are known, seeking codes that adapt to unknown sources.
Carried by Jacob Ziv; Abraham Lempel.
Iterative and Sparse-Graph Coding grew out of Shannon Theory of Communication
Turbo codes and the revived LDPC codes were designed to approach Shannon capacity, the limit established in Shannon's channel coding theorem.
Carried by Claude Berrou; Robert Gallager.
Quantum Shannon Theory grew out of Shannon Theory of Communication
Quantum Shannon theory generalizes Shannon's classical notions of entropy, channel capacity, and coding theorems to quantum states and channels.
Carried by Benjamin Schumacher; Charles H. Bennett.
Minimum Description Length grew out of Shannon Theory of Communication
The minimum description length principle reframes statistical model selection using Shannon's notion of code length, treating the best model as the one giving the shortest description of data plus model.
Carried by Jorma Rissanen.
Finite-Blocklength Information Theory grew out of Shannon Theory of Communication
Finite-blocklength information theory refines Shannon's asymptotic coding theorems by deriving bounds on achievable rates and error probabilities for finite block lengths.
Carried by Yury Polyanskiy; H. Vincent Poor.
Iterative and Sparse-Graph Coding influenced by Algebraic Coding Theory
Turbo codes combined convolutional codes, an algebraic coding construction, with iterative decoding, and the revival of LDPC codes built on Gallager's sparse-matrix algebraic approach.
Carried by Claude Berrou.
Beyond the field
Where Information Theory's ideas came from in other fields, and what they led to there: the strongest 12 of 54 links.
Shannon Theory of Communication led to Information-theoretic security in Applied Cryptography
Information-theoretic security is a branch of Shannon's theory of communication, applying its channel and entropy concepts to the secrecy problem.
Carried by Claude Shannon.
Shannon Theory of Communication influenced Entropy in thermodynamics and information theory in Thermodynamics
Shannon's theory of communication introduced the entropy measure of information that the thermodynamic analogy is built on. The comparison of thermodynamic and information entropy takes its information-theoretic side from Shannon.
Carried by Claude Shannon.
Shannon Theory of Communication led to Information-Theoretic Cryptography in Cryptography
Shannon's theory of communication supplied the entropy, redundancy and equivocation measures that information-theoretic cryptography uses to define perfect secrecy. The cryptographic framework is a direct application and branch of that theory.
Carried by Claude Shannon.
Algebraic Coding Theory grew out of Information theory in Electrical Engineering
Algebraic coding theory is a branch of information theory, arising from Shannon's channel-coding theorem and its promise that reliable transmission is possible below capacity.
Carried by Claude Shannon.
Algebraic Coding Theory grew out of Shannon information theory in Coding Theory
Algebraic coding theory grew directly out of Shannon's 1948 coding theorems, which posed the existence of efficient codes and set the bounds that algebraic constructions like Hamming and Golay codes were built to approach.
Carried by Claude Shannon; Richard Hamming; Marcel Golay.
Algebraic Coding Theory grew out of Mathematical Theory of Communication in Communication
Algebraic coding theory grew out of Shannon's 'A Mathematical Theory of Communication', whose channel-coding theorem defined the problem the algebraic constructions address.
Carried by Claude Shannon; A Mathematical Theory of Communication (1948).
Algebraic Coding Theory influenced Distributed storage coding in Coding Theory
Regenerating codes and locally repairable codes for distributed storage were built from algebraic coding constructions, notably Reed-Solomon and MDS codes adapted to the repair-bandwidth tradeoff. Dimakis and colleagues formulated regenerating codes directly on this algebraic base.
Carried by Alexandros Dimakis; Kannan Ramchandran.
Rate-Distortion Theory grew out of Information theory in Electrical Engineering
Rate-distortion theory was developed by Shannon in 1959 as an extension of his information theory to lossy source coding, generalising the entropy-based source-coding theorem.
Carried by Claude Shannon.
Rate-Distortion Theory grew out of Shannon information theory in Coding Theory
Rate-distortion theory is a direct outgrowth of Shannon's information theory, formulated by Shannon himself as its lossy-coding extension.
Carried by Claude Shannon.
Rate-Distortion Theory grew out of Mathematical Theory of Communication in Communication
Rate-distortion theory extends the source-coding part of Shannon's 'A Mathematical Theory of Communication' to the case of allowed distortion.
Carried by Claude Shannon; A Mathematical Theory of Communication (1948).
Network Information Theory grew out of Information theory in Electrical Engineering
Network information theory extends Shannon's information theory from point-to-point channels to multi-terminal networks, beginning with Shannon's two-way channel paper of 1961.
Carried by Claude Shannon.
Network Information Theory grew out of Shannon information theory in Coding Theory
Network information theory is a direct extension of Shannon information theory, taking its entropy and capacity tools to multi-user settings.
Carried by Claude Shannon.

Information Theory is a planet in Nousfarer
Its city has a tower for each of these 9 frameworks. Land, walk in, and climb one concept at a time.
Land on Information TheoryRelated fields
Fields its ideas are linked to
- Coding Theory 16 links
- Communication Systems 8 links
- Communication 7 links
- Signal Processing 7 links
- Electrical Engineering 6 links
- Interpersonal Communication 2 links
- Quantum Information 2 links
- Thermodynamics 2 links
- Design Theory 1 link
- Extremal Combinatorics 1 link
- Applied Cryptography 1 link
- Cryptography 1 link
More in Electrical Engineering
- Circuit Analysis
- Communication Systems
- Control Systems
- Digital Systems
- Electrical Engineering
- Electromagnetics
- Photonics
- Power Electronics
- Power Systems
- Rf And Microwave Engineering
- Semiconductor Electronics
- Signal Processing
- Signals And Systems
- Vlsi And Integrated Circuit Design
See where Information Theory sits on the Map of Human Knowledge