Recommender Systems

Recommender systems are the software that decides what you see next: which film Netflix suggests, which product Amazon puts on the page, which song Spotify queues, which post a social feed surfaces.

13 frameworks, from 1992 to today · Data Science · Computer Science

still a working view ended · each leads to its entry below

Overview

Recommender systems are the software that decides what you see next: which film Netflix suggests, which product Amazon puts on the page, which song Spotify queues, which post a social feed surfaces1. Their job is to estimate, from data about past behaviour, what a particular person will want among a set of options too large to browse1. What is at stake is attention and choice at scale. When a catalogue holds millions of items, no one can search it exhaustively, so the system's guess about your taste becomes a gatekeeper of what you encounter1. The history of the field is a sequence of attempts to make that guess better—first by finding similar people, then by finding hidden structure in ratings, then by learning directly from clicks and sequences, and now by leaning on large language models.

Early Information Filtering and the Tapestry System

The intellectual root lies in information retrieval and "information filtering"—the problem of routing a stream of incoming documents to the people who want them12. In 1992, David Goldberg, David Nichols, Brian Oki, and Douglas Terry at Xerox PARC described Tapestry, a system for filtering Usenet news. Tapestry let users annotate documents and, crucially, let a person's filter draw on other users' annotations: "show me documents that people who liked what I liked have rated highly." This is the seed of collaborative filtering—recommendation based on the overlap of many users' opinions rather than on the content of items.

Collaborative Filtering and the Memory-Based Era

The idea was formalised and made famous by GroupLens, a 1994 system by Paul Resnick, Neophytos Iacovou, Mitesh Suchak, Peter Bergstrom, and John Riedl for filtering Usenet articles by predicted rating. GroupLens introduced the now-standard machinery: users rate items, the system computes similarity between users, and it predicts a rating as a weighted average of neighbours' ratings. The same period saw the Bellcore work of Upendra Shardanand and Pattie Maes (1995) on "social information filtering" for music, and the Ringo music recommender by Jonathan Herlocker and colleagues.

These memory-based methods—so called because they keep the whole rating matrix and compute on it at query time—split into two families. User-based collaborative filtering finds people similar to you and borrows their opinions1. Item-based collaborative filtering, popularised by Badrul Sarwar, George Karypis, Joseph Konstan, and John Riedl around 2001, instead finds items similar to ones you liked, measured by the pattern of who rated them. Item-based similarity proved more stable and scalable, and it powered Amazon's early product recommendations3. The core weakness was cold start: a new user or a new item has no ratings, so similarity cannot be computed1.

Content-Based Filtering and the Cold-Start Problem

Content-based filtering attacked cold start from the other side1. Instead of comparing people, it compares items by their features—text, genre, metadata—and builds a profile of what a user has liked1. Early work on learning user profiles from text, such as the NewsWeeder system by Ken Lang (1995), trained a model on articles a user rated and used it to rank new ones. Content-based methods handle new items well (their features are known immediately) but suffer from overspecialisation: they recommend more of what you already liked and cannot surprise you with something outside your profile. They also require meaningful item features, which is not always available.

Association Rules and the Basket

In parallel, the data-mining community developed association rule mining, crystallised in the Apriori algorithm by Rakesh Agrawal and Ramakrishnan Srikant (1994). Given transactions—market baskets—Apriori finds rules like "buyers of bread and butter often buy milk," characterised by support (how often the items co-occur) and confidence (how reliably the rule holds). Applied to retail, this became the "customers who bought this also bought…" recommendation. It is a co-occurrence method, not a personalisation method: it says nothing about an individual's taste, only about aggregate patterns. Its descendants, including frequent-pattern and sequence-mining methods, remained a practical workhorse for e-commerce for years.

Matrix Factorization and the Netflix Prize

The decisive shift came with latent factor models4. The idea: represent each user and each item as a short vector of hidden factors, and predict a rating as the dot product of the two vectors4. The factors are not named—they might correspond to "serious versus light" or "mainstream versus niche"—but they are learned from the rating data itself. Simon Funk's widely read 2006 blog post during the Netflix Prize showed that a simple gradient-descent factorisation, with regularisation, worked remarkably well4. The Netflix Prize (2006–2009), a $1 million contest to improve rating prediction by 10%, became the field's public laboratory1. The winning team, BellKor's Pragmatic Chaos, reached the threshold in 2009 by blending many models, and the competition established that matrix factorization—formalised in Yehuda Koren, Robert Bell, and Chris Volinsky's 2009 survey—was the state of the art for explicit ratings. It handled sparsity far better than memory-based methods and captured subtle taste structure.

Implicit Feedback and the Shift to Ranking

Ratings are scarce; clicks, views, and purchases are abundant but carry no negative signal—a missing click might mean dislike or simply inattention. Implicit-feedback recommendation reframed the problem as ranking rather than rating prediction. The landmark method was BPR (Bayesian Personalised Ranking) by Steffen Rendle and colleagues (2009), which learns factors from pairs: a user is assumed to prefer an item they interacted with over one they did not. This "pairwise" framing, and the broader move to optimise ranking metrics directly, aligned recommendation with what systems actually do—order a list—rather than predict a number. Learning to rank, developed in information retrieval (with methods such as RankNet and LambdaMART), supplied the machinery for optimising such orderings.

Beyond Accuracy

As systems matured, practitioners noticed that the most accurate model was not always the best product. Beyond-accuracy research asked what else matters: diversity (not showing ten near-identical items), novelty, serendipity, coverage, and fairness. A recommender that maximises predicted relevance can trap users in a narrow loop and starve new items of exposure. This line of work, growing through the 2010s, reframed evaluation itself—accuracy metrics like RMSE or NDCG became necessary but not sufficient, and researchers began measuring long-run effects on users and catalogues.

Neural Collaborative Filtering and Deep Learning

Deep learning reached recommendation in the mid-2010s. Neural collaborative filtering, notably the 2017 paper by Xiangnan He and colleagues, replaced the dot product with a neural network over user and item embeddings, arguing that a learned nonlinear function could capture interactions a simple inner product could not. Neural methods spread across the field: embeddings for users and items, deep models over features, and architectures borrowed from computer vision and language. The gains over well-tuned matrix factorisation were sometimes modest, but the framework proved flexible enough to absorb rich side information—text, images, context—that classic factorisation could not.

Sequences, Sessions, and Two-Stage Serving

Real systems face two problems the rating-prediction literature ignored: order matters, and scale is enormous. Sequential and session-based recommendation models what a user will want next given the recent sequence of their actions1. Early work used Markov chains over item transitions; recurrent networks, and later attention-based models such as SASRec (2018), learned from ordered histories directly. Session-based models handle anonymous users with only a short current session to go on1.

At the same time, industrial systems converged on a two-stage retrieval-and-ranking architecture1. Because scoring millions of items per request is infeasible, a fast retrieval (or candidate-generation) stage narrows the catalogue to hundreds of plausible items—often via approximate nearest-neighbour search over embeddings—and a heavier ranking stage orders that shortlist with a rich model. This division, standard by the late 2010s, is how deep models are made to serve at web scale.

Graphs and Structured Relations

Graph-based recommendation treats users and items as nodes in a network, with interactions as edges, and propagates information along those edges. Methods such as LightGCN (2020) simplified graph convolutional networks for recommendation by stripping away feature transformations and nonlinearities, showing that neighbourhood aggregation alone—smoothing embeddings across the user–item graph—captured collaborative signal effectively. Graph methods naturally exploit higher-order relations (a user's friends, an item's co-purchase network) and became a major research direction.

Large Language Models and the Present

The newest turn brings large language models into recommendation. LLMs can read item descriptions, user histories, and conversational context, and can be prompted to rank or generate recommendations, or used to produce richer embeddings and profiles. Early results are promising for cold start and for explaining recommendations in natural language, though the approach is young and its cost and reliability are open questions.

Today the field runs on a layered stack that descends directly from this history: embedding-based retrieval and neural ranking for the core, sequence models for session context, graph propagation for relational signal, and increasingly LLMs for understanding and explanation—all evaluated not only for accuracy but for diversity, fairness, and long-run user wellbeing. The open questions are the old ones sharpened: how to recommend well from sparse and implicit signals, how to balance relevance against exploration and fairness, how to measure effects that unfold over months rather than a single click, and how much of the problem can be handed to a general-purpose language model without losing the precision that specialised systems spent three decades building.

References

  1. Recommender system (introduction, Content-based filtering, Collaborative filtering, The Netflix Prize, Session-based recommender systems, Two-Tower Model), Wikipedia
  2. Information filtering system, Wikipedia
  3. Item-item collaborative filtering, Wikipedia
  4. Matrix factorization (recommender systems), Wikipedia

The frameworks of Recommender Systems

Each framework is a way of working the field has used: when it held, and the question it answers.

  1. 1992-Present
  2. 1994-2015
  3. 1995-Present
  4. 1997-Present
  5. 2005-Present
  6. 2006-Present
  7. 2009-Present
  8. 2010-Present
  9. 2010-Present
  10. 2013-Present
  11. 2016-Present
  12. 2018-Present
  13. 2022-Present
A city in Nousfarer: a red-carpeted stair climbing terraces to a crowned tower on an acropolis, more towers round it, forest and snowy mountains beyond.
A city in Nousfarer, the game played in the browser.

Recommender Systems is a planet in Nousfarer

Its city has a tower for each of these 13 frameworks. Land, walk in, and climb one concept at a time.

Land on Recommender Systems

Related fields

More in Data Science

See where Recommender Systems sits on the Map of Human Knowledge