- Atlas
- Computer Science
- Natural Language Processing
Natural Language Processing
Natural language processing (NLP) is the branch of artificial intelligence concerned with getting computers to handle human language: to translate it, answer questions about it, summarize it, extract facts from it, generate it, and converse in it.
7 frameworks, from 1954 to today · Artificial Intelligence · Computer Science
Overview
Natural language processing (NLP) is the branch of artificial intelligence concerned with getting computers to handle human language: to translate it, answer questions about it, summarize it, extract facts from it, generate it, and converse in it1. What is at stake is unusually direct. Language is the medium in which most human knowledge is stored and most human business is done, and it is also the clearest everyday display of intelligence we have. A machine that reads a contract, answers a customer, or writes a coherent paragraph is doing something that, until recently, only people could do. The history of NLP is therefore a history of successive attempts to say what language is, computationally — as rules, as statistics, as learned representations — and of what each attempt could and could not do.
Mechanical translation and the rule-writing impulse
NLP begins as a translation problem. In the years after the Second World War, researchers in the United States and Britain proposed that a computer could translate Russian scientific text into English by carrying a dictionary and a set of grammatical rules. The Georgetown–IBM experiment of 1954, which translated a small set of Russian sentences into English using a hand-built dictionary and six grammar rules, was widely publicized as a demonstration that the problem was tractable. Funding followed.
The dominant idea of this era was symbolic and rule-based: language was a system of formal regularities, and a sufficiently thorough grammar plus a dictionary would let a machine parse a sentence into its structure and rewrite it in another language1. Early systems leaned on large bilingual dictionaries and hand-coded syntactic rules. The approach ran into the problem that has dogged rule-writing ever since: language is far more variable than any rule set anticipates, and every new sentence seems to require a new exception. A 1960 report by the U.S. Automatic Language Processing Advisory Committee concluded that machine translation had not delivered and that the money was better spent on basic linguistics; the field's funding collapsed, and machine translation went quiet for more than a decade.
Parsing, grammars, and the symbolic program
Through the 1960s and 1970s NLP regrouped around understanding rather than translation. Researchers built systems that answered questions about small, constrained worlds — a table of blocks, a database of lunar rock samples — by parsing the question into a logical form and querying a knowledge base. The intellectual bet was that meaning could be represented as formal structure and that grammar could be written precisely enough to recover it.
Two developments gave this program its shape. One was the formal grammar tradition: Noam Chomsky's work from the late 1950s argued that natural language could not be captured by finite-state devices and required richer grammatical structure, and computational linguists built parsers around context-free and then more expressive grammars. The other was the rise of logic-based semantics, in which a sentence's meaning is a formula and inference over formulas is what understanding consists of. By the 1980s, unification-based grammar formalisms and feature structures let linguists write large, precise grammars that computers could execute, and the field produced substantial hand-built grammar and lexicon resources.
The limits were structural. Hand-built grammars were expensive, brittle outside their designed domain, and hard to extend; ambiguity multiplied combinatorially; and the knowledge a system needed to disambiguate ordinary sentences — that a sentence about a restaurant is probably about eating — was not in the grammar at all. Symbolic NLP could be precise about a narrow world and nearly helpless outside it.
The statistical turn
The decisive change came in the late 1980s and 1990s, and it came from data1. The idea was to stop writing rules and instead estimate probabilities from large collections of text — corpora — so that the system learned which analyses were likely rather than which were licensed1. A part-of-speech tagger, for instance, could decide that "book" in "book a flight" is a verb by counting how often that word appears as a verb in that context across millions of words, rather than by consulting a rule.
Several things made this possible at once. Machine-readable text became abundant; statistical methods from speech recognition, where hidden Markov models had already displaced rule-based approaches, were imported into language work; and evaluation became quantitative, with shared benchmark tasks and scored results that let methods be compared. The probabilistic parser, the statistical machine-translation system trained on parallel text, and the maximum-entropy classifier for text categorization all date from this period. The field's self-image changed: NLP became an empirical science of models fitted to data, and the older symbolic tradition was pushed to the margins, though it never disappeared — hybrid systems combining hand-built resources with statistics remained common, and linguistically informed features often improved statistical models.
Statistical NLP's own ceiling was the feature engineering it required1. Practitioners had to decide by hand which properties of the text mattered — the previous two words, the word's suffix, its capitalization — and encode them. Progress depended on human ingenuity in designing those features, and the models could not see anything the designer had not thought to show them.
Neural sequence models
The next turn replaced hand-designed features with learned representations. Neural network methods for language had been tried since the 1980s and 1990s — a neural probabilistic language model that learned word representations and predicted the next word was published by Yoshua Bengio and colleagues in 2003 — but they became central in the 2010s as computation and data grew.
The key idea is the distributed representation: a word is a vector of numbers, learned from context, so that words used similarly end up near each other in the vector space. These word embeddings, popularized by the word2vec methods of 2013, gave every downstream task a shared, dense input and removed much feature engineering2. Recurrent neural networks, which process a sequence one element at a time while carrying a memory of what came before, became the standard architecture for translation, tagging, and language modeling; the addition of gating mechanisms (long short-term memory, introduced in 1997) let them handle longer dependencies. By the mid-2010s, neural machine translation — a single network trained end to end to map a source sentence to a target sentence — had overtaken the statistical phrase-based systems that had dominated translation for two decades.
Sequence models still had a weakness: they read text strictly left to right, one step at a time, which made them slow to train and forgetful over long passages. That constraint set up the next break.
Attention and the transformer
In 2014–2015, attention mechanisms were introduced into neural translation: instead of compressing a whole source sentence into one fixed vector, the decoder learned to look back at the relevant parts of the input at each step3. The 2017 paper "Attention Is All You Need" by Vaswani and colleagues went further and removed recurrence entirely. The transformer architecture processes all positions in a sequence in parallel and uses self-attention — each token weighing every other token — to build context4. This made training far more parallelizable and let models scale to sizes that recurrent networks could not reach4.
The transformer made possible a new way of building NLP systems: pre-train one large model on vast unlabelled text, then adapt it to specific tasks5. BERT (2018) pre-trained a transformer to predict masked words from their context and then fine-tuned it, setting new records across a wide range of understanding benchmarks6. GPT models took the other route, pre-training a transformer to predict the next word and then generating text3. The practical consequence was that a single pre-trained model, adapted with modest task-specific training, replaced the bespoke architecture built for each task — and the field's center of gravity moved from task-specific systems to general-purpose language models.
Alignment, instruction tuning, and the assistant era
A model trained only to predict the next word is not automatically a useful assistant: it will continue text in whatever style the training data suggests, and it has no notion of following an instruction or declining a harmful request. The response, developed from around 2020–2022, was alignment: further training that shapes the model's behavior toward what users want. Instruction tuning trains a model on many examples of instructions paired with good responses, so that it learns to treat a request as a request3. Reinforcement learning from human feedback (RLHF) trains a model to produce outputs that human raters prefer, using a reward model learned from those preferences73. The result was the conversational assistant that could be asked to do a task in plain language — the form in which large language models reached the public from late 20225.
Retrieval, tools, and multimodality
Two limitations of a pure language model became the frontier. First, a model's knowledge is frozen at its training data and it cannot reliably cite or verify facts; retrieval-augmented generation addresses this by having the model search an external document collection and condition its answer on what it finds8. Second, a model that only emits text cannot act; tool use gives it the ability to call a calculator, a search engine, a database, or an API, so that the language model becomes the reasoning layer over other systems. In parallel, multimodal language modeling extended the same architecture beyond text — models that take images, audio, or video as input alongside language, and increasingly produce them, so that "language model" now often means a model of language grounded in other modalities9.
Where the field stands
Today the dominant approach is pre-trained transformer language models, adapted by prompting, fine-tuning, or alignment, and increasingly combined with retrieval and external tools. The lineage is visible at every step: the symbolic tradition's insistence that structure and meaning matter survives in work on parsing, formal semantics, and neuro-symbolic hybrids; the statistical turn's commitment to evaluation on data survives in every benchmark; the neural turn's learned representations are the substrate of everything current. The open questions are the old ones in new dress. How does a system trained to predict text come to represent meaning and reference? How can its outputs be made reliable and its errors detected? How much can scale alone deliver, and where does grounding in the world, or explicit structure, remain necessary? The field has changed its methods repeatedly, but it has not stopped asking what it would take for a machine to actually understand.
References
- Natural language processing (introduction, Approaches: Symbolic, statistical, neural networks, Statistical NLP (1990s–present), Statistical approach, Neural networks), Wikipedia
- Word2Vec (introduction, History), Wikipedia
- Large language model (History, introduction, Fine-tuning), Wikipedia
- Transformer (deep learning), Wikipedia
- Generative pre-trained transformer (Background, introduction), Wikipedia
- BERT (language model), Wikipedia
- Reinforcement learning from human feedback, Wikipedia
- Retrieval-augmented generation, Wikipedia
- Multimodal learning, Wikipedia
The frameworks of Natural Language Processing
Each framework is a way of working the field has used: when it held, and the question it answers.
- 1954-1995
Rule-based and symbolic NLP
Can grammar rules let a machine parse and translate language?
- 1988-2015
Statistical NLP
Can probabilities estimated from text corpora resolve linguistic ambiguity?
About Statistical NLP
Statistical Natural Language Processing (NLP) emerged in the 1990s as a paradigm shift from hand-crafted rules to probabilistic models learned from large text corpora. Its central premise is that language behavior can be captured by estimating probability distributions over words, sequences, and structures. This framework addressed the scalability limitations of symbolic and rule-based systems by grounding linguistic knowledge in observed frequencies, enabling systems that could handle ambiguity, variability, and open-domain text.Core Curriculum The following concepts form the teaching spine of Statistical NLP, organized from foundational to advanced:- Language Modeling and Evaluation:
- N-gram Language Models estimate the probability of a word given the previous words, forming the backbone of many tasks.
- Smoothing Techniques (e.g., Add-k, Good-Turing, Kneser-Ney) adjust maximum likelihood estimates to avoid zero probabilities for unseen events.
- Perplexity as Evaluation Metric measures how well a model predicts a test set; lower perplexity indicates better fit.
- Maximum Likelihood Estimation (MLE) is the principle of choosing parameters that maximize the probability of observed training data, applied throughout statistical models.
- Sequence Labeling with Hidden Markov Models:
- Hidden Markov Models for POS Tagging represent sequences of hidden states (e.g., parts of speech) with transition and emission probabilities.
- The Viterbi Algorithm efficiently finds the most likely state sequence, while the Baum-Welch Algorithm (an instance of EM) learns HMM parameters from unlabeled data.
- Probabilistic Parsing:
- Probabilistic Context-Free Grammars (PCFGs) extend context-free grammars with rule probabilities, enabling stochastic parsing to disambiguate syntactic structures.
- Statistical Machine Translation:
- The Noisy Channel Model frames translation as a decoding problem: find the most likely target sentence given a source sentence, using a language model and a translation model.
- IBM Translation Models (Model 1–5) progressively increase alignment complexity, from word-to-word to fertility-based models.
- Statistical Alignment (e.g., HMM Alignment for MT) estimates word correspondences between parallel sentences.
- Distributional and Clustering Methods:
- n-gram and Skip-gram Models (Distributional) learn word representations from co-occurrence statistics, predicting either surrounding words (skip-gram) or contiguous sequences.
- Brown Clustering (Hard Clustering) groups words into classes based on hierarchical mutual information, producing a hard partition of the vocabulary.
- Maximum Entropy Models (Log-Linear Models) combine multiple overlapping features by maximizing entropy subject to constraints, offering a flexible framework for classification and sequence labeling.
- Evaluation with Precision, Recall, F-measure (Early Statistical Evaluation) adapted these metrics from information retrieval to assess tasks like tagging, parsing, and translation.
Intellectual History Statistical NLP crystallized in the late 1980s and early 1990s, as researchers in speech recognition and machine translation turned to probabilistic methods. The Noisy Channel Model originated with Claude Shannon's information theory (1948) and was adapted for speech by Jelinek et al. in the 1970s, but its application to NLP exploded with the IBM Candide project. Peter F. Brown, John Cocke, Stephen Della Pietra, Vincent Della Pietra, and Robert Mercer published "A Statistical Approach to Machine Translation" (1990), introducing IBM Models 1-5 and framing MT as a decoding problem. Frederick Jelinek's work on statistical language models, including n-grams and smoothing, led to the canonical textbook "Statistical Methods for Speech Recognition" (1997) and influenced NLP widely.Hidden Markov Models were adopted from speech recognition for tasks like part-of-speech tagging, with early taggers by Kenneth Church (1988) and Eric Brill (1992, though Brill's tagger used transformation-based learning). The Baum-Welch algorithm had been developed in the 1960s, but its use for unsupervised HMM training in NLP became standard. Probabilistic Context-Free Grammars were formalized by John Lafferty and others (1992), and parsing with PCFGs was advanced by Eugene Charniak (1993) and Michael Collins (1997).Smoothing techniques evolved from add-k to Good-Turing (introduced by Good in 1953, applied to NLP by Church and Gale in 1991) to Kneser-Ney smoothing (1995), which became a standard. Maximum Entropy models were introduced to NLP by Adam Berger, Stephen Della Pietra, and Vincent Della Pietra (1996) in "A Maximum Entropy Approach to Natural Language Processing". Brown clustering emerged from Peter Brown et al. (1992) in the con of class-based n-grams.Statistical NLP coexisted with Corpus-Based NLP, which provided the data, and Probabilistic Graphical Models, which offered general inference tools. It peaked in the 2000s, then was gradually supplemented by neural frameworks after 2010. Key textbooks include Christopher Manning and Hinrich Schütze's "Foundations of Statistical Natural Language Processing" (1999), which codified the curriculum. By 2015, neural methods had largely absorbed statistical approaches, but the statistical toolkit—especially language models, HMMs, and evaluation metrics—remains foundational.The 15 concepts its tower teaches
- Baum-Welch Algorithm
- Brown Clustering
- Evaluation with Precision, Recall, F-measure
- Hidden Markov Models for POS Tagging
- IBM Translation Models
- Maximum Entropy Models
- Maximum Likelihood Estimation
- n-gram and Skip-gram Models (Distributional)
- N-gram Language Models
- Noisy Channel Model
- Perplexity as Evaluation Metric
- Probabilistic Context-Free Grammars
- Smoothing Techniques
- Statistical Alignment
- Viterbi Algorithm
- 2013-2018
Neural sequence modeling
Can learned word vectors and recurrent networks replace hand-built features?
- 2017-Present
Transformer and pre-training paradigm
Can one pre-trained model serve every language task?
- 2020-Present
Alignment and instruction following
How do we make a language model follow instructions and behave well?
- 2020-Present
Retrieval-augmented and tool-using language models
Can a language model look things up and use tools?
- 2021-Present
Multimodal language modeling
Can one model learn language together with images and sound?
How the ideas connect
Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.
Statistical NLP reacted against Rule-based and symbolic NLP
The statistical turn defined itself against hand-written grammar and dictionary rules, arguing that probabilities estimated from corpora should replace rule-writing. Practitioners who had built symbolic parsers and translation systems moved to probabilistic models trained on the same tasks.
Carried by Frederick Jelinek; IBM Candide.
Neural sequence modeling grew out of Statistical NLP
Neural sequence models kept statistical NLP's empirical, corpus-trained, benchmark-scored program but replaced hand-designed features with learned distributed representations. Word embeddings and end-to-end neural machine translation were trained and evaluated on the same parallel corpora and shared tasks that statistical systems had established.
Carried by Yoshua Bengio; word2vec (2013).
Transformer and pre-training paradigm replaced Neural sequence modeling
The transformer removed the recurrence that defined neural sequence modeling, processing all positions in parallel with self-attention. Attention had been added to recurrent translation models in 2014-2015, and the 2017 transformer showed recurrence could be dropped entirely, making sequence models a limiting case.
Carried by Attention Is All You Need (2017); Ashish Vaswani.
Alignment and instruction following grew out of Transformer and pre-training paradigm
Alignment and instruction following grew directly out of pre-trained transformer language models: instruction tuning and reinforcement learning from human feedback were applied to GPT-style next-word predictors to make them follow requests. The base pre-trained model is the object that alignment training reshapes.
Carried by InstructGPT (2022).
Multimodal language modeling grew out of Transformer and pre-training paradigm
Multimodal language models extended the transformer pre-training recipe to images and other modalities, tokenizing them and training jointly with text. The architecture and pre-training objective came straight from the transformer paradigm.
Transformer and pre-training paradigm influenced by Statistical NLP
The pre-training paradigm inherited statistical NLP's practice of fitting models to very large unlabelled text corpora and scoring them on shared benchmarks. Next-word and masked-word prediction are probabilistic language models in the statistical tradition, scaled up and trained end to end.
Retrieval-augmented and tool-using language models grew out of Transformer and pre-training paradigm
Retrieval-augmented generation combined a pre-trained transformer generator with a document retriever, building on the pre-train-then-adapt paradigm rather than replacing it. Tool use likewise wraps a pre-trained language model in an external interface it calls.
Carried by Retrieval-Augmented Generation (2020).
Neural sequence modeling influenced by Rule-based and symbolic NLP
Symbolic NLP left behind large hand-built lexicons and annotated treebanks whose grammatical categories and structures were reused as training and evaluation data for neural taggers and parsers. The linguistic distinctions the rule-based tradition codified carried into the labels neural models learned to predict.
Retrieval-augmented and tool-using language models influenced by Alignment and instruction following
Instruction-tuned and aligned models made reliable tool calling and retrieval use practical, since following a request to consult a document or invoke a tool is itself an instruction-following behavior. Tool-using systems were built on models already shaped by alignment training.
Beyond the field
Where Natural Language Processing's ideas came from in other fields, and what they led to there: the strongest 12 of 27 links.
Rule-based and symbolic NLP influenced History of machine translation in Translation History
Early machine translation was built directly on rule-based symbolic NLP methods such as dictionary lookup and syntactic transfer rules, and the Georgetown-IBM experiment of 1954 embodied that approach.
Carried by Georgetown-IBM experiment (1954).
Statistical NLP influenced by Statistical signal processing in Signal Processing
Statistical NLP imported the hidden Markov model, n-gram estimation and smoothing directly from statistical signal processing, where they had been developed for speech recognition; speech researchers moving into language processing carried the methods over.
Carried by Fred Jelinek.
Statistical NLP led to Statistical machine translation in Machine Translation Studies
Statistical machine translation was a direct application of statistical NLP methods, especially the noisy-channel model and IBM alignment models developed in speech and language processing in the late 1980s and early 1990s.
Carried by IBM Model 1 (1993); Peter F. Brown.
Neural sequence modeling influenced Sequence-to-function machine learning in Regulatory Genomics
Sequence-to-function models borrowed the convolutional and recurrent architectures, training procedures and sequence-embedding ideas developed for neural sequence modeling in natural language processing. These were transferred to DNA and RNA sequences as a new kind of text.
Neural sequence modeling led to Neural machine translation in Machine Translation Studies
Neural machine translation grew directly out of neural sequence modeling: the encoder-decoder architecture with attention developed for sequence-to-sequence learning was applied to translation by Sutskever and colleagues and by Bahdanau and colleagues in 2014.
Carried by Ilya Sutskever; Dzmitry Bahdanau; Sequence to Sequence Learning with Neural Networks (2014).
Neural sequence modeling led to Transformer in Deep Learning
The Transformer grew out of neural sequence modeling, replacing recurrent and convolutional sequence models with self-attention. Sequence-to-sequence work supplied the architecture it superseded.
Carried by Sequence to Sequence Learning (2014).
Neural sequence modeling influenced by Recurrent neural network in Deep Learning
Neural sequence modeling in NLP was built directly on recurrent neural networks, using LSTM and gated recurrent architectures developed in deep learning for sequence transduction and language modeling.
Carried by Sepp Hochreiter; Jürgen Schmidhuber.
Neural sequence modeling influenced by Deep Learning in Data Science
Neural sequence models for NLP were applications of the deep learning toolkit: distributed word embeddings, stacked recurrent layers and end-to-end gradient training came from the deep learning revival.
Carried by Yoshua Bengio.
Transformer and pre-training paradigm influenced Vision transformer in Computer Vision
The transformer and pre-training paradigm from natural language processing supplied both the self-attention architecture and the pretrain-then-finetune recipe that the vision transformer adopted for images.
Transformer and pre-training paradigm influenced by Neural machine translation in Machine Translation Studies
The transformer was introduced for neural machine translation and then generalized to pretraining; NMT supplied the encoder-decoder task formulation and the attention mechanism the paradigm scaled up.
Carried by Ashish Vaswani.
Transformer and pre-training paradigm influenced by Deep Learning in Data Science
The transformer and pretraining paradigm is a direct outgrowth of deep learning: residual connections, layer normalization, dropout and large-scale gradient training all came from that toolkit.
Retrieval-augmented and tool-using language models influenced by Large language model in Artificial Intelligence
Retrieval-augmented and tool-using systems are built on pretrained large language models, using them as the generator that retrieval conditions and tools extend.

Natural Language Processing is a planet in Nousfarer
Its city has a tower for each of these 7 frameworks. Land, walk in, and climb one concept at a time.
Land on Natural Language ProcessingRelated fields
Fields its ideas are linked to
- Artificial Intelligence 5 links
- Deep Learning 4 links
- Machine Translation Studies 4 links
- Computer Vision 2 links
- Data Science 2 links
- Ai Safety 1 link
- Machine Learning 1 link
- Signal Processing 1 link
- Regulatory Genomics 1 link
- Information Science 1 link
- Corpus Linguistics 1 link
- Linguistics 1 link
More in Artificial Intelligence
- Ai Ethics And Fairness
- Ai Safety
- Artificial Intelligence
- Computer Vision
- Deep Learning
- Evolutionary Computation
- Knowledge Representation Reasoning
- Machine Learning
- Multiagent Systems
- Planning And Search
- Probabilistic Ai
- Reinforcement Learning
- Speech Recognition And Synthesis
See where Natural Language Processing sits on the Map of Human Knowledge