Speech Recognition And Synthesis

Speech recognition and speech synthesis are the two directions of the same conversation between people and machines.

12 frameworks, from 1770 to today · Artificial Intelligence · Computer Science

still a working view ended · each leads to its entry below

Overview

Speech recognition and speech synthesis are the two directions of the same conversation between people and machines. Recognition, often called automatic speech recognition (ASR), turns a recording of a voice into text or commands1. Synthesis, or text-to-speech (TTS), turns text into an audible voice2. What is at stake is not only convenience—dictating a document, asking a phone a question, listening to an article—but a test of whether machines can handle language as it is actually spoken: fast, slurred, accented, noisy, and full of ambiguity. For most of the twentieth century the two problems were attacked with hand-built rules and physical devices; since the 1980s they have been rebuilt around statistics and, most recently, deep neural networks trained on enormous quantities of recorded speech.

Mechanical voices and the first analyzers

Synthesis came first, because making a voice is easier to engineer than understanding one. In the late eighteenth century, inventors built mechanical speaking machines: Christian Kratzenstein produced vowel sounds with resonating tubes in the 1770s, and Wolfgang von Kempelen's "speaking machine," described in 1791, used a bellows for lungs, a reed for the vocal cords, and a leather resonator shaped by the hand to form different sounds. These devices demonstrated that speech could be decomposed into an airflow, a vibration, and a filter—an insight that later became the source-filter model of speech production.

Electronic synthesis arrived with the vocoder, developed by Homer Dudley at Bell Labs in the late 1930s23. The vocoder analyzed speech into a small set of slowly varying control signals—the energy in a few frequency bands and whether the sound was voiced or unvoiced—and could resynthesize intelligible speech from them. This was the conceptual breakthrough: speech could be represented as a compact parametric signal rather than a waveform. In 1939 Dudley demonstrated the Voder, a keyboard-operated vocoder that a trained operator played to produce sentences at the New York World's Fair23.

Recognition began in parallel. In 1952, Davis, Biddulph, and Balashek at Bell Labs built a system that recognized spoken digits by measuring the energy in two frequency bands during vowel segments. It worked for a single speaker14. The problem of different speakers, different speeds, and connected speech was not yet on the table.

Formants, phonemes, and the symbolic dream

In the 1950s and 1960s, speech research was dominated by the idea that speech could be described by a small set of acoustic features tied to linguistic units. The key concept was the formant: a resonance peak of the vocal tract, visible on a spectrogram as a dark band, whose frequencies largely determine which vowel is heard. Gunnar Fant's Acoustic Theory of Speech Production (1960) gave a systematic account of how vocal-tract shape maps to formants, and the source-filter model became the standard picture.

Formant synthesis built voices directly from this theory. The OVE series of synthesizers in Japan, and later the Klatt synthesizer developed by Dennis Klatt at MIT in the 1970s and 1980s, generated speech by driving a set of resonators with a buzz or noise source and specifying formant frequencies and bandwidths over time. Klatt's system was rule-based and parametric: a phonetician wrote rules converting phonemes into formant trajectories. It produced intelligible, if robotic, speech, and it made every parameter inspectable—useful for research even when it sounded unnatural.

Recognition followed the same symbolic instinct. The idea was to segment the signal into phonetic units and classify them. Early systems recognized isolated words or digits by template matching: store a recording of each word, compare a new recording to each template, and pick the closest. The difficulty was that the same word spoken twice never has the same duration or timing. The solution, dynamic time warping (DTW), introduced in the speech-recognition context in the 1970s (notably by Hiroaki Sakoe and Seibi Chiba in 1978), aligns two sequences by stretching and compressing time so that similar parts line up. DTW made speaker-dependent isolated-word recognition practical, and it powered commercial products in the 1980s.

But template matching could not handle connected speech or many speakers. The alternative was knowledge-based recognition: encode what linguists know—phoneme inventories, phonological rules, lexical pronunciation dictionaries—and search for the sequence of units that best explains the acoustics. The ARPA Speech Understanding Research program, funded from 1971 to 1976, pursued this vision with systems such as CMU's Harpy, which recognized connected speech over a constrained vocabulary by searching a network of phonetic and lexical rules. Harpy was a landmark, but the approach hit a wall: hand-written rules could not capture the variability of real speech, and every new vocabulary or speaker required new labor.

Statistics takes over

The turning point came from a different direction: treat speech as a random process and learn its statistics from data. The central tool was the hidden Markov model (HMM), a probabilistic model of a sequence in which the system passes through hidden states (for speech, roughly, parts of a sound) and at each state emits an observable acoustic feature according to a probability distribution. Recognition becomes: given the observed features, which sequence of words was most likely to have produced them? The HMM had been described in the statistics literature in the 1960s, and its use for speech was pioneered in the 1970s—James Baker's Dragon system at CMU and Frederick Jelinek and colleagues at IBM both applied it to recognition. By the 1980s, with the work of Lawrence Rabiner and others at Bell Labs, HMMs became the dominant framework.

The appeal was that the model could be trained automatically: given recordings and their transcriptions, an algorithm (Baum-Welch training, a form of expectation-maximization) estimated the emission and transition probabilities5. No phonetician had to write rules. The acoustic features were typically cepstra—a compact representation of the short-term spectrum, computed with a cosine transform of the log spectrum—which decorrelated the features and made Gaussian modeling effective1. The lexicon and language model were layered on top: a pronunciation dictionary mapped words to phoneme sequences, and an n-gram language model gave the prior probability of word sequences. The whole thing was searched with the Viterbi algorithm, which finds the most likely state sequence efficiently6.

The statistical approach won because it degraded gracefully. A rule-based system failed outright on an unexpected pronunciation; an HMM assigned it a low but nonzero probability and kept going. Through the 1990s and 2000s, HMM-based systems with Gaussian mixture models (GMMs) for the acoustic distributions, trained on hundreds or thousands of hours of speech, became the standard for dictation software, telephone directory assistance, and eventually voice search. Error rates fell steadily, driven by more data, better features, and refinements such as discriminative training and speaker adaptation.

Concatenation and the sound of synthetic speech

Synthesis took a parallel statistical turn, but its decisive move was toward recorded human speech. Concatenative synthesis builds utterances by stitching together short recorded segments—diphones (the transition between two phonemes) or longer units—selected from a large database to match the desired phonetic context and prosody. The approach was developed in the 1980s and 1990s; the AT&T NextGen system and the Festival and later HMM-based systems are landmarks. Concatenation produced far more natural voices than formant rules, because the segments carried the real acoustics of a human speaker. The cost was a large recorded database and audible seams where segments joined, which required careful smoothing.

A further step was statistical parametric synthesis: instead of storing waveforms, train an HMM on the speech parameters (spectral and prosodic features) and generate new parameter trajectories from the model, then synthesize a waveform with a vocoder. This was developed most prominently by Keiichi Tokuda and colleagues in the HMM-based speech synthesis system (HTS), released in the 2000s. It allowed voice characteristics to be transformed and adapted from small amounts of data, and it produced smooth, if somewhat muffled, speech. The trade-off between the naturalness of concatenation and the flexibility of parametric synthesis defined the field for a decade.

Neural networks reshape both directions

The deep-learning era arrived in speech in the 2010s, and it changed both recognition and synthesis from the ground up.

In recognition, the first inroad was hybrid neural-HMM acoustic modeling. Around 2010–2012, researchers including Geoffrey Hinton's group and others showed that deep neural networks (DNNs) could replace the Gaussian mixture model as the estimator of acoustic state probabilities within an HMM framework14. A DNN took a window of frames as input and output a probability for each HMM state; the HMM still handled sequence alignment and decoding. The gain was large—word error rates dropped by a substantial margin on standard benchmarks—and by the mid-2010s hybrid DNN-HMM systems were the state of the art in commercial and research ASR. The HMM had not been abandoned; it had been given a much stronger acoustic model.

Then the HMM itself was displaced. End-to-end neural sequence modeling treats recognition as a single mapping from acoustic frames to text, trained with a sequence loss. The key architectures were connectionist temporal classification (CTC), introduced by Alex Graves and colleagues in 2006, which lets a network output text aligned to audio without frame-level labels; attention-based encoder-decoder models, which learn to align and decode jointly; and later the recurrent neural network transducer (RNN-T), which streams. By the late 2010s, systems such as Deep Speech and the Listen, Attend and Spell model showed that end-to-end systems could match or beat hybrid systems, and by the 2020s they dominated. The lexicon and the separate language model were increasingly folded into the network or replaced by neural language models.

Synthesis underwent a comparable revolution. Neural parametric vocoding and waveform generation replaced the signal-processing vocoders with learned models. WaveNet, introduced by DeepMind in 2016, generated raw audio sample by sample with a dilated convolutional network and produced speech far more natural than concatenative or HMM systems. It was slow, but subsequent models—WaveRNN, and then flow-based and GAN-based vocoders—made high-quality generation fast enough for practical use. Neural TTS systems such as Tacotron (2017) and its successors mapped text directly to acoustic features, and the vocoder turned those features into sound. The result was synthetic speech that listeners often could not distinguish from recordings.

Where the field stands

Today, recognition is dominated by large neural models trained on tens of thousands of hours of speech, often with self-supervised pretraining on unlabeled audio, and increasingly by end-to-end architectures that fold acoustic modeling, pronunciation, and language modeling into one system. Synthesis is dominated by neural text-to-speech pipelines—text to acoustic features to waveform—with neural vocoders, and by fully end-to-end models that generate waveforms directly. The old debates have not vanished so much as changed shape: how to handle rare words and names, how to recognize speech in noise and across accents, how to make synthetic voices controllable and expressive, and how to do all of this with less data and less computation. The mechanical speaking machines and the hand-written phonetic rules are gone from practice, but the questions they raised—what are the units of speech, how does context shape sound, and how much can be learned from examples rather than specified by hand—are still the questions the field is answering.

References

  1. Speech recognition (introduction, Pre-1970, Hidden Markov models, 2000s), Wikipedia
  2. Speech synthesis (introduction, History), Wikipedia
  3. Vocoder (introduction, History), Wikipedia
  4. Speech processing (History), Wikipedia
  5. Baum–Welch algorithm (introduction, History), Wikipedia
  6. Viterbi algorithm, Wikipedia

The frameworks of Speech Recognition And Synthesis

Each framework is a way of working the field has used: when it held, and the question it answers.

  1. 1770-1930
  2. 1939-1980
  3. 1960-1990
  4. 1970-1990
  5. 1971-1985
  6. 1975-2015
  7. 1985-2010
  8. 1995-2016
  9. 2010-2018
  10. 2014-Present
  11. 2016-Present
  12. 2019-Present
A city in Nousfarer: a red-carpeted stair climbing terraces to a crowned tower on an acropolis, more towers round it, forest and snowy mountains beyond.
A city in Nousfarer, the game played in the browser.

Speech Recognition And Synthesis is a planet in Nousfarer

Its city has a tower for each of these 12 frameworks. Land, walk in, and climb one concept at a time.

Land on Speech Recognition And Synthesis

Related fields

More in Artificial Intelligence

See where Speech Recognition And Synthesis sits on the Map of Human Knowledge