Natural Language Processing

Natural language processing (NLP) is the branch of artificial intelligence concerned with getting computers to handle human language: to translate it, answer questions about it, summarize it, extract facts from it, generate it, and converse in it.

7 frameworks, from 1954 to today · Artificial Intelligence · Computer Science

still a working view ended · each leads to its entry below

Overview

Natural language processing (NLP) is the branch of artificial intelligence concerned with getting computers to handle human language: to translate it, answer questions about it, summarize it, extract facts from it, generate it, and converse in it1. What is at stake is unusually direct. Language is the medium in which most human knowledge is stored and most human business is done, and it is also the clearest everyday display of intelligence we have. A machine that reads a contract, answers a customer, or writes a coherent paragraph is doing something that, until recently, only people could do. The history of NLP is therefore a history of successive attempts to say what language is, computationally — as rules, as statistics, as learned representations — and of what each attempt could and could not do.

Mechanical translation and the rule-writing impulse

NLP begins as a translation problem. In the years after the Second World War, researchers in the United States and Britain proposed that a computer could translate Russian scientific text into English by carrying a dictionary and a set of grammatical rules. The Georgetown–IBM experiment of 1954, which translated a small set of Russian sentences into English using a hand-built dictionary and six grammar rules, was widely publicized as a demonstration that the problem was tractable. Funding followed.

The dominant idea of this era was symbolic and rule-based: language was a system of formal regularities, and a sufficiently thorough grammar plus a dictionary would let a machine parse a sentence into its structure and rewrite it in another language1. Early systems leaned on large bilingual dictionaries and hand-coded syntactic rules. The approach ran into the problem that has dogged rule-writing ever since: language is far more variable than any rule set anticipates, and every new sentence seems to require a new exception. A 1960 report by the U.S. Automatic Language Processing Advisory Committee concluded that machine translation had not delivered and that the money was better spent on basic linguistics; the field's funding collapsed, and machine translation went quiet for more than a decade.

Parsing, grammars, and the symbolic program

Through the 1960s and 1970s NLP regrouped around understanding rather than translation. Researchers built systems that answered questions about small, constrained worlds — a table of blocks, a database of lunar rock samples — by parsing the question into a logical form and querying a knowledge base. The intellectual bet was that meaning could be represented as formal structure and that grammar could be written precisely enough to recover it.

Two developments gave this program its shape. One was the formal grammar tradition: Noam Chomsky's work from the late 1950s argued that natural language could not be captured by finite-state devices and required richer grammatical structure, and computational linguists built parsers around context-free and then more expressive grammars. The other was the rise of logic-based semantics, in which a sentence's meaning is a formula and inference over formulas is what understanding consists of. By the 1980s, unification-based grammar formalisms and feature structures let linguists write large, precise grammars that computers could execute, and the field produced substantial hand-built grammar and lexicon resources.

The limits were structural. Hand-built grammars were expensive, brittle outside their designed domain, and hard to extend; ambiguity multiplied combinatorially; and the knowledge a system needed to disambiguate ordinary sentences — that a sentence about a restaurant is probably about eating — was not in the grammar at all. Symbolic NLP could be precise about a narrow world and nearly helpless outside it.

The statistical turn

The decisive change came in the late 1980s and 1990s, and it came from data1. The idea was to stop writing rules and instead estimate probabilities from large collections of text — corpora — so that the system learned which analyses were likely rather than which were licensed1. A part-of-speech tagger, for instance, could decide that "book" in "book a flight" is a verb by counting how often that word appears as a verb in that context across millions of words, rather than by consulting a rule.

Several things made this possible at once. Machine-readable text became abundant; statistical methods from speech recognition, where hidden Markov models had already displaced rule-based approaches, were imported into language work; and evaluation became quantitative, with shared benchmark tasks and scored results that let methods be compared. The probabilistic parser, the statistical machine-translation system trained on parallel text, and the maximum-entropy classifier for text categorization all date from this period. The field's self-image changed: NLP became an empirical science of models fitted to data, and the older symbolic tradition was pushed to the margins, though it never disappeared — hybrid systems combining hand-built resources with statistics remained common, and linguistically informed features often improved statistical models.

Statistical NLP's own ceiling was the feature engineering it required1. Practitioners had to decide by hand which properties of the text mattered — the previous two words, the word's suffix, its capitalization — and encode them. Progress depended on human ingenuity in designing those features, and the models could not see anything the designer had not thought to show them.

Neural sequence models

The next turn replaced hand-designed features with learned representations. Neural network methods for language had been tried since the 1980s and 1990s — a neural probabilistic language model that learned word representations and predicted the next word was published by Yoshua Bengio and colleagues in 2003 — but they became central in the 2010s as computation and data grew.

The key idea is the distributed representation: a word is a vector of numbers, learned from context, so that words used similarly end up near each other in the vector space. These word embeddings, popularized by the word2vec methods of 2013, gave every downstream task a shared, dense input and removed much feature engineering2. Recurrent neural networks, which process a sequence one element at a time while carrying a memory of what came before, became the standard architecture for translation, tagging, and language modeling; the addition of gating mechanisms (long short-term memory, introduced in 1997) let them handle longer dependencies. By the mid-2010s, neural machine translation — a single network trained end to end to map a source sentence to a target sentence — had overtaken the statistical phrase-based systems that had dominated translation for two decades.

Sequence models still had a weakness: they read text strictly left to right, one step at a time, which made them slow to train and forgetful over long passages. That constraint set up the next break.

Attention and the transformer

In 2014–2015, attention mechanisms were introduced into neural translation: instead of compressing a whole source sentence into one fixed vector, the decoder learned to look back at the relevant parts of the input at each step3. The 2017 paper "Attention Is All You Need" by Vaswani and colleagues went further and removed recurrence entirely. The transformer architecture processes all positions in a sequence in parallel and uses self-attention — each token weighing every other token — to build context4. This made training far more parallelizable and let models scale to sizes that recurrent networks could not reach4.

The transformer made possible a new way of building NLP systems: pre-train one large model on vast unlabelled text, then adapt it to specific tasks5. BERT (2018) pre-trained a transformer to predict masked words from their context and then fine-tuned it, setting new records across a wide range of understanding benchmarks6. GPT models took the other route, pre-training a transformer to predict the next word and then generating text3. The practical consequence was that a single pre-trained model, adapted with modest task-specific training, replaced the bespoke architecture built for each task — and the field's center of gravity moved from task-specific systems to general-purpose language models.

Alignment, instruction tuning, and the assistant era

A model trained only to predict the next word is not automatically a useful assistant: it will continue text in whatever style the training data suggests, and it has no notion of following an instruction or declining a harmful request. The response, developed from around 2020–2022, was alignment: further training that shapes the model's behavior toward what users want. Instruction tuning trains a model on many examples of instructions paired with good responses, so that it learns to treat a request as a request3. Reinforcement learning from human feedback (RLHF) trains a model to produce outputs that human raters prefer, using a reward model learned from those preferences73. The result was the conversational assistant that could be asked to do a task in plain language — the form in which large language models reached the public from late 20225.

Retrieval, tools, and multimodality

Two limitations of a pure language model became the frontier. First, a model's knowledge is frozen at its training data and it cannot reliably cite or verify facts; retrieval-augmented generation addresses this by having the model search an external document collection and condition its answer on what it finds8. Second, a model that only emits text cannot act; tool use gives it the ability to call a calculator, a search engine, a database, or an API, so that the language model becomes the reasoning layer over other systems. In parallel, multimodal language modeling extended the same architecture beyond text — models that take images, audio, or video as input alongside language, and increasingly produce them, so that "language model" now often means a model of language grounded in other modalities9.

Where the field stands

Today the dominant approach is pre-trained transformer language models, adapted by prompting, fine-tuning, or alignment, and increasingly combined with retrieval and external tools. The lineage is visible at every step: the symbolic tradition's insistence that structure and meaning matter survives in work on parsing, formal semantics, and neuro-symbolic hybrids; the statistical turn's commitment to evaluation on data survives in every benchmark; the neural turn's learned representations are the substrate of everything current. The open questions are the old ones in new dress. How does a system trained to predict text come to represent meaning and reference? How can its outputs be made reliable and its errors detected? How much can scale alone deliver, and where does grounding in the world, or explicit structure, remain necessary? The field has changed its methods repeatedly, but it has not stopped asking what it would take for a machine to actually understand.

References

  1. Natural language processing (introduction, Approaches: Symbolic, statistical, neural networks, Statistical NLP (1990s–present), Statistical approach, Neural networks), Wikipedia
  2. Word2Vec (introduction, History), Wikipedia
  3. Large language model (History, introduction, Fine-tuning), Wikipedia
  4. Transformer (deep learning), Wikipedia
  5. Generative pre-trained transformer (Background, introduction), Wikipedia
  6. BERT (language model), Wikipedia
  7. Reinforcement learning from human feedback, Wikipedia
  8. Retrieval-augmented generation, Wikipedia
  9. Multimodal learning, Wikipedia

The frameworks of Natural Language Processing

Each framework is a way of working the field has used: when it held, and the question it answers.

  1. 1954-1995

    Rule-based and symbolic NLP

    Can grammar rules let a machine parse and translate language?

    Fly to its tower in Nousfarer

  2. 1988-2015

    Statistical NLP

    Can probabilities estimated from text corpora resolve linguistic ambiguity?

    About Statistical NLP
    Statistical Natural Language Processing (NLP) emerged in the 1990s as a paradigm shift from hand-crafted rules to probabilistic models learned from large text corpora. Its central premise is that language behavior can be captured by estimating probability distributions over words, sequences, and structures. This framework addressed the scalability limitations of symbolic and rule-based systems by grounding linguistic knowledge in observed frequencies, enabling systems that could handle ambiguity, variability, and open-domain text.
    Core Curriculum The following concepts form the teaching spine of Statistical NLP, organized from foundational to advanced:
    1. Language Modeling and Evaluation:
    • N-gram Language Models estimate the probability of a word given the previous n−1n-1 words, forming the backbone of many tasks.
    • Smoothing Techniques (e.g., Add-k, Good-Turing, Kneser-Ney) adjust maximum likelihood estimates to avoid zero probabilities for unseen events.
    • Perplexity as Evaluation Metric measures how well a model predicts a test set; lower perplexity indicates better fit.
    1. Maximum Likelihood Estimation (MLE) is the principle of choosing parameters that maximize the probability of observed training data, applied throughout statistical models.
    1. Sequence Labeling with Hidden Markov Models:
    • Hidden Markov Models for POS Tagging represent sequences of hidden states (e.g., parts of speech) with transition and emission probabilities.
    • The Viterbi Algorithm efficiently finds the most likely state sequence, while the Baum-Welch Algorithm (an instance of EM) learns HMM parameters from unlabeled data.
    1. Probabilistic Parsing:
    • Probabilistic Context-Free Grammars (PCFGs) extend context-free grammars with rule probabilities, enabling stochastic parsing to disambiguate syntactic structures.
    1. Statistical Machine Translation:
    • The Noisy Channel Model frames translation as a decoding problem: find the most likely target sentence given a source sentence, using a language model and a translation model.
    • IBM Translation Models (Model 1–5) progressively increase alignment complexity, from word-to-word to fertility-based models.
    • Statistical Alignment (e.g., HMM Alignment for MT) estimates word correspondences between parallel sentences.
    1. Distributional and Clustering Methods:
    • n-gram and Skip-gram Models (Distributional) learn word representations from co-occurrence statistics, predicting either surrounding words (skip-gram) or contiguous sequences.
    • Brown Clustering (Hard Clustering) groups words into classes based on hierarchical mutual information, producing a hard partition of the vocabulary.
    1. Maximum Entropy Models (Log-Linear Models) combine multiple overlapping features by maximizing entropy subject to constraints, offering a flexible framework for classification and sequence labeling.
    1. Evaluation with Precision, Recall, F-measure (Early Statistical Evaluation) adapted these metrics from information retrieval to assess tasks like tagging, parsing, and translation.
    Intellectual History Statistical NLP crystallized in the late 1980s and early 1990s, as researchers in speech recognition and machine translation turned to probabilistic methods. The Noisy Channel Model originated with Claude Shannon's information theory (1948) and was adapted for speech by Jelinek et al. in the 1970s, but its application to NLP exploded with the IBM Candide project. Peter F. Brown, John Cocke, Stephen Della Pietra, Vincent Della Pietra, and Robert Mercer published "A Statistical Approach to Machine Translation" (1990), introducing IBM Models 1-5 and framing MT as a decoding problem. Frederick Jelinek's work on statistical language models, including n-grams and smoothing, led to the canonical textbook "Statistical Methods for Speech Recognition" (1997) and influenced NLP widely.
    Hidden Markov Models were adopted from speech recognition for tasks like part-of-speech tagging, with early taggers by Kenneth Church (1988) and Eric Brill (1992, though Brill's tagger used transformation-based learning). The Baum-Welch algorithm had been developed in the 1960s, but its use for unsupervised HMM training in NLP became standard. Probabilistic Context-Free Grammars were formalized by John Lafferty and others (1992), and parsing with PCFGs was advanced by Eugene Charniak (1993) and Michael Collins (1997).
    Smoothing techniques evolved from add-k to Good-Turing (introduced by Good in 1953, applied to NLP by Church and Gale in 1991) to Kneser-Ney smoothing (1995), which became a standard. Maximum Entropy models were introduced to NLP by Adam Berger, Stephen Della Pietra, and Vincent Della Pietra (1996) in "A Maximum Entropy Approach to Natural Language Processing". Brown clustering emerged from Peter Brown et al. (1992) in the con\text{} of class-based n-grams.
    Statistical NLP coexisted with Corpus-Based NLP, which provided the data, and Probabilistic Graphical Models, which offered general inference tools. It peaked in the 2000s, then was gradually supplemented by neural frameworks after 2010. Key textbooks include Christopher Manning and Hinrich Schütze's "Foundations of Statistical Natural Language Processing" (1999), which codified the curriculum. By 2015, neural methods had largely absorbed statistical approaches, but the statistical toolkit—especially language models, HMMs, and evaluation metrics—remains foundational.
    The 15 concepts its tower teaches
    • Baum-Welch Algorithm
    • Brown Clustering
    • Evaluation with Precision, Recall, F-measure
    • Hidden Markov Models for POS Tagging
    • IBM Translation Models
    • Maximum Entropy Models
    • Maximum Likelihood Estimation
    • n-gram and Skip-gram Models (Distributional)
    • N-gram Language Models
    • Noisy Channel Model
    • Perplexity as Evaluation Metric
    • Probabilistic Context-Free Grammars
    • Smoothing Techniques
    • Statistical Alignment
    • Viterbi Algorithm

    Fly to its tower in Nousfarer

  3. 2013-2018

    Neural sequence modeling

    Can learned word vectors and recurrent networks replace hand-built features?

    Fly to its tower in Nousfarer

  4. 2017-Present

    Transformer and pre-training paradigm

    Can one pre-trained model serve every language task?

    Fly to its tower in Nousfarer

  5. 2020-Present

    Alignment and instruction following

    How do we make a language model follow instructions and behave well?

    Fly to its tower in Nousfarer

  6. 2020-Present
  7. 2021-Present

    Multimodal language modeling

    Can one model learn language together with images and sound?

    Fly to its tower in Nousfarer

How the ideas connect

Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.

Rule-based and symbolic NLPStatistical NLPNeural sequence modelingTransformer and pre-training paradigmAlignment and instruction followingRetrieval-augmented and tool-using la…Multimodal language modeling

Beyond the field

Where Natural Language Processing's ideas came from in other fields, and what they led to there: the strongest 12 of 27 links.

A city in Nousfarer: a red-carpeted stair climbing terraces to a crowned tower on an acropolis, more towers round it, forest and snowy mountains beyond.
A city in Nousfarer, the game played in the browser.

Natural Language Processing is a planet in Nousfarer

Its city has a tower for each of these 7 frameworks. Land, walk in, and climb one concept at a time.

Land on Natural Language Processing

Related fields

Fields its ideas are linked to

More in Artificial Intelligence

See where Natural Language Processing sits on the Map of Human Knowledge