ML Systems

Machine learning systems are the machinery that turns statistical models into working software.

10 frameworks, from 1965 to today · Data Science · Computer Science

still a working view ended · each leads to its entry below

Overview

Machine learning systems are the machinery that turns statistical models into working software. A model on a laptop is a research artifact; a model serving millions of predictions a second, retrained nightly on fresh data, monitored for drift, and rolled back when it misbehaves is an ML system. The subfield's practitioners are not primarily the people who invent learning algorithms. They are the engineers and scientists who ask: how do we train on data too large for one machine, serve predictions fast enough to matter, keep training and serving consistent, and know when a deployed model has gone stale? What is at stake is the gap between a promising result and a dependable product—a gap that has swallowed far more models than it has delivered.

Statistical computing and the batch model

Before there was an ML systems field, there was statistical computing. From the 1960s onward, packages such as BMDP, SPSS, and later S and R gave statisticians a way to fit models by writing a few lines rather than hand-coding matrix algebra. The workflow was batch: assemble a dataset, run a script, read the output, repeat tomorrow. This culture established two habits that persist—models as artifacts produced by scripts, and the analyst as the person who runs them.

The first real systems problem was scale. Fitting a regression on millions of rows exceeded what a single program could hold in memory. The response in the 1990s and 2000s was to push computation to where the data lived. MapReduce, described by Dean and Ghemawat in 2004, split work across a cluster of commodity machines: a map step processes chunks in parallel, a reduce step aggregates. Its open-source implementation, Hadoop, made cluster-scale batch processing available to ordinary teams. Mahout, an early library built on Hadoop, expressed clustering and recommendation algorithms as MapReduce jobs. The lesson was structural: learning at scale is a distributed-systems problem, and the algorithm must be rewritten to fit the execution model.

The rise of the feature pipeline

As models moved from one-off analyses into products, a new problem appeared: the same transformation had to happen twice, once in the training script and once in the serving code, and the two drifted apart. A feature computed one way offline and another way online produces a model that looks accurate in evaluation and fails in production—a failure mode known as training-serving skew.

The response, emerging around the mid-2010s, was to make features first-class infrastructure. A feature store is a system that computes a feature once and serves it both to training jobs (as historical values) and to live prediction (as the latest value), guaranteeing the two agree1. Uber's Michelangelo platform, described publicly in 2017, was an influential example; Feast and similar open-source projects followed. Around the same time, teams began treating models and features as data assets with lineage—records of which data and which transformations produced which artifact—so that a model could be audited, reproduced, or rolled back. This was a conceptual shift: the model was no longer the deliverable but one node in a tracked pipeline.

GPUs and the deep learning stack

The decisive break came from hardware. Graphics processors, built to apply the same operation to many pixels at once, turned out to be well suited to the matrix multiplications at the heart of neural networks. Researchers had used GPUs for neural nets since the mid-2000s, but the field changed when the tooling caught up. Theano, released in 2007, and then TensorFlow (2015) and PyTorch (2016) let practitioners define a computation as a graph and have the framework handle gradients and device placement automatically. A graph-based framework represents the model as nodes (operations) and edges (data flowing between them), which makes it possible to optimize, parallelize, and differentiate the whole computation2.

This stack made deep learning practical, and deep learning made new systems demands. Models grew from millions to billions of parameters, and a single GPU could no longer hold them. Model parallelism—splitting one model across many devices—became necessary, alongside mixed-precision training, which stores some numbers in 16-bit floats instead of 32-bit to halve memory and speed up arithmetic while keeping enough precision to converge. These techniques, developed through the late 2010s, are what allowed the largest models to be trained at all.

Recommendations, embeddings, and sparse systems

Meanwhile, the highest-volume ML systems in the world were not image classifiers but recommenders. A recommender must score millions of items for millions of users in milliseconds, and the dominant representation became the embedding: a dense vector of numbers learned for each user and item, so that similarity becomes a geometric question. The systems challenge is that the embedding tables are enormous—far larger than any single machine's memory—and access is sparse, touching only a handful of rows per request. Industrial systems such as Google's Wide & Deep (2016) and subsequent deep recommendation models drove specialized infrastructure: sharded embedding tables, sparse updates during training, and low-latency lookup at serving. This line of work is where ML systems and large-scale information retrieval meet.

MLOps and the lifecycle

By the late 2010s, practitioners had accumulated a pile of partial solutions and a name for the discipline that binds them: MLOps, machine learning operations, by analogy with DevOps. The claim was that a model has a lifecycle—data collection, training, evaluation, deployment, monitoring, retraining, retirement—and that each stage needs tooling and ownership3.

Two ideas anchor it. The first is continuous training: because the world changes, a deployed model's inputs drift away from what it learned, so the system must detect degradation and retrain automatically. The second is lifecycle observability: logging predictions, inputs, and outcomes so that a team can see, after the fact, why a model behaved as it did4. Together these turn a model from a static file into a monitored service with a feedback loop. The field's vocabulary—model registry, experiment tracking, drift detection, canary rollout—dates largely from this period.

The present

Today the main approaches descend directly from these turning points. Feature stores and lineage tracking are standard in mature organizations; graph-based frameworks (PyTorch, TensorFlow, and JAX) are the default substrate for deep learning; model and data parallelism with mixed precision are routine for large models; and MLOps platforms bundle training, deployment, and monitoring into one pipeline. The open questions are real ones. How do you keep training-serving consistency when features are computed by large language models rather than simple aggregations1? How do you monitor a system whose outputs are open-ended text? And how do you make the whole lifecycle reproducible when the data itself is continuously changing5? The field began by asking how to run a model at scale; it now asks how to keep one trustworthy over time.

References

  1. Feature store, Wikipedia
  2. PyTorch, Wikipedia
  3. MLOps, Wikipedia
  4. Observability (software), Wikipedia
  5. Data version control, Wikipedia

The frameworks of ML Systems

Each framework is a way of working the field has used: when it held, and the question it answers.

  1. 1965-2005
  2. 2004-2016

    MapReduce

    How do we process data too large for one machine?

    Fly to its tower in Nousfarer

  3. 2007-Present

    Graph-Based Deep Learning Frameworks

    How do we represent a model as a computation graph?

    Fly to its tower in Nousfarer

  4. 2012-Present

    GPU-Accelerated Deep Learning Systems

    How do we train deep networks fast with GPUs?

    Fly to its tower in Nousfarer

  5. 2015-Present

    Feature store

    How do we compute features once for training and serving?

    Fly to its tower in Nousfarer

  6. 2016-Present

    Embedding and Sparse Recommendation Systems

    How do we serve recommendations from huge sparse tables?

    Fly to its tower in Nousfarer

  7. 2016-Present

    Model and Feature Lineage as Data Assets

    How do we track which data produced which model?

    Fly to its tower in Nousfarer

  8. 2017-Present
  9. 2018-Present

    Lifecycle Observability and Continuous Training

    How do we know when a deployed model has gone stale?

    Fly to its tower in Nousfarer

  10. 2018-Present

    MLOps

    How do we manage the whole lifecycle of a model?

    Fly to its tower in Nousfarer

How the ideas connect

Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.

Statistical Computing and Batch Analy…MapReduceGraph-Based Deep Learning FrameworksGPU-Accelerated Deep Learning SystemsFeature storeEmbedding and Sparse Recommendation S…Model and Feature Lineage as Data Ass…Model and Data Parallelism with Mixed…Lifecycle Observability and Continuou…MLOps

Beyond the field

Where ML Systems's ideas came from in other fields, and what they led to there: the strongest 12 of 38 links.

A city in Nousfarer: a red-carpeted stair climbing terraces to a crowned tower on an acropolis, more towers round it, forest and snowy mountains beyond.
A city in Nousfarer, the game played in the browser.

ML Systems is a planet in Nousfarer

Its city has a tower for each of these 10 frameworks. Land, walk in, and climb one concept at a time.

Land on ML Systems

Related fields

Fields its ideas are linked to

More in Data Science

See where ML Systems sits on the Map of Human Knowledge