Data Mining

Data mining is the practice of searching large datasets for patterns that are useful, non-obvious, and not simply the product of chance.

11 frameworks, from 1957 to today · Data Science · Computer Science

still a working view ended · each leads to its entry below

Overview

Data mining is the practice of searching large datasets for patterns that are useful, non-obvious, and not simply the product of chance1. Its practitioners build and apply algorithms that sort, group, predict, and flag: they classify customers, cluster documents, predict equipment failure, surface unusual transactions, and discover rules such as "people who buy diapers also buy beer" (a famous but largely apocryphal example from retail folklore)1. What is at stake is the difference between data that merely exists and data that informs a decision. Data mining sits at the meeting point of statistics, machine learning, and database engineering, and its history is the story of how those three communities learned to talk to one another1.

Statistics and Exploratory Data Analysis Before Computing

Long before the term existed, statisticians were mining data by hand1. In the nineteenth and early twentieth centuries, regression, correlation, and the analysis of variance gave researchers tools for finding relationships in tables of numbers. But these methods were built for confirmatory work: state a hypothesis, then test it. The exploratory side of statistics was less formalized until John Tukey, working at Bell Labs and Princeton, pushed it forward in the 1960s and 1970s. Tukey's 1977 book Exploratory Data Analysis argued that analysts should first look at data—with stem-and-leaf plots, box plots, and resistant summaries that are not thrown off by a few extreme values—before fitting models. The box plot, which shows a distribution's median, quartiles, and outliers in one compact graphic, is Tukey's most visible legacy. His message, that visual inspection and robust summaries reveal structure that formal tests can miss, became a founding assumption of data mining2.

Databases Grow Too Large to Inspect

Through the 1970s and 1980s, the relational database, formalized by Edgar Codd at IBM in 1970, made it possible to store and query enormous tables. But SQL queries answer questions you already know how to ask. By the late 1980s, businesses held gigabytes of transaction records and had no systematic way to ask "what patterns are in here?" The first workshop explicitly devoted to knowledge discovery in databases was held in 1989, and the field's vocabulary crystallized around that time1. Gregory Piatetsky-Shapiro, who organized that workshop, and others drew a distinction that has stuck: knowledge discovery in databases (KDD) is the whole process—selecting data, cleaning it, transforming it, mining it, and interpreting the results—while data mining is the algorithmic step in the middle1. The KDD conference series began in 1995, and the journal Data Mining and Knowledge Discovery followed in 1997. The process model that many practitioners still follow, CRISP-DM (Cross-Industry Standard Process for Data Mining), was published in 1999 by a consortium of European companies and research groups; it lays out six phases from business understanding through deployment, and its emphasis on iteration and business framing reflects the field's applied origins1.

Decision Trees and the Rise of Inductive Learning

The most influential early mining algorithm was the decision tree. A decision tree asks a sequence of yes/no questions about a record's attributes and ends in a prediction3. The idea traces to work in the 1960s, but the version that spread through data mining was ID3, published by Ross Quinlan in 1986, and its successor C4.5, published in 1993. ID3 chooses each split by information gain—a measure from Claude Shannon's information theory of how much a question reduces uncertainty about the outcome4. A concrete case: given records of weather conditions and whether a game was played, ID3 builds a tree whose first question might be "is it sunny?" because that attribute separates the outcomes most cleanly. Quinlan's trees were fast, produced rules humans could read, and handled real datasets with noise and missing values. They became the default first algorithm in countless courses and commercial tools, and their readable output made them the field's answer to the complaint that mining produces black boxes.

Nearest Neighbors and the Instance-Based View

A rival philosophy held that you need not build an explicit model at all5. Instance-based learning, articulated by David Aha, Dennis Kibler, and Marc Albert in 1991, and the k-nearest-neighbor classifier, whose statistical foundations were laid by Evelyn Fix and Joseph Hodges in the early 1950s and extended by Thomas Cover and Peter Hart in 1967, simply store the training examples. To classify a new case, find the k most similar stored cases and take a vote6. The method needs a notion of distance—usually Euclidean distance across numeric attributes—and it makes no assumptions about the shape of the underlying pattern67. Its weakness is equally plain: with millions of records, comparing a new case to every stored one is slow, and irrelevant attributes distort the distance. These trade-offs, interpretability versus flexibility, explicit model versus memorized examples, recur throughout the field's history.

Clustering: Finding Groups Without Labels

Cluster analysis addresses a different question: given data with no outcome to predict, can we find natural groupings18? The k-means algorithm, which alternates between assigning points to the nearest of k centers and recomputing those centers, was independently devised in the 1950s and 1960s (Stuart Lloyd's version circulated at Bell Labs from 1957; James MacQueen named it in 1967)9. Hierarchical clustering, which builds a tree of nested groupings, dates to the 1960s work of Joe Ward and others. The difficulty is that "natural grouping" is not self-defining: k-means will happily partition any dataset into k clusters whether or not real structure exists9. The field's response was to develop internal validation measures—silhouette scores, the gap statistic—and to insist that clustering results be checked against domain knowledge. Density-based methods such as DBSCAN, published by Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu in 1996, addressed a specific failure: k-means assumes clusters are roughly spherical and of similar size, while DBSCAN finds arbitrarily shaped clusters and labels sparse points as noise.

Association Rules and the Basket Problem

The algorithm that made data mining a household term in industry came from retail1. In 1993, Rakesh Agrawal, Tomasz Imieliński, and Arun Swami published AIS, and in 1994 Agrawal and Ramakrishnan Srikant published Apriori, for finding association rules: statements of the form "if a basket contains X, it often contains Y." Apriori's insight was a pruning principle—if an itemset is rare, every larger itemset containing it is also rare—which let it avoid checking the exponential number of possible combinations. The measure of interest was support (how often the rule's items appear together) and confidence (how often Y appears given X)1011. The method worked, and it spawned a decade of variants for efficiency and for other pattern types: sequential patterns, episodes, and frequent subgraphs. But it also exposed a problem that shaped the next phase: with low thresholds, the number of "interesting" rules explodes into the millions, most of them trivial or redundant. The study of interestingness measures—lift, conviction, and dozens of others—grew up around this problem, and researchers such as Sergey Brin, Rajeev Motwani, and Craig Silverstein argued in the late 1990s that statistical dependence, not raw frequency, should define what counts as a pattern worth reporting.

Outliers, Ensembles, and the Problem of Trust

Anomaly and outlier detection began as a statistical concern: an observation far from the rest may be an error or may be the most important record in the file. Tukey's exploratory work treated outliers as signals to investigate, and the field developed methods suited to high dimensions—distance-based definitions, local density measures such as the Local Outlier Factor of Breunig and colleagues in 2000, and later isolation-based methods. The practical stakes are concrete: fraud detection, network intrusion, and equipment failure all depend on flagging the rare case.

Meanwhile, the single-model approach ran into a wall. A decision tree trained on one sample of data can be unstable; small changes in the data produce different trees. Ensemble learning answered this by combining many models12. Bagging, introduced by Leo Breiman in 1996, trains models on bootstrap resamples and averages their votes. Boosting, developed by Yoav Freund and Robert Schapire in the late 1990s (AdaBoost, 1997), trains models sequentially, each focusing on the cases the previous ones got wrong. Breiman's Random Forests (2001) combined bagging with random attribute selection at each split and became one of the most reliable off-the-shelf miners. The empirical finding that ensembles of mediocre models often beat a single carefully tuned one was a genuine surprise and reshaped practice13.

Validation: The Discipline That Keeps Mining Honest

The field's credibility depended on a methodological turn as much as an algorithmic one. Mining millions of patterns guarantees that some will look significant by chance1. Statistical learning and validation supplied the safeguards: cross-validation, in which data is repeatedly split into training and held-out portions; the distinction between training error and generalization error; and the bias-variance trade-off, formalized in the 1990s, which explains why a model that fits its training data too closely fails on new data. The 2001 book The Elements of Statistical Learning by Trevor Hastie, Robert Tibshirani, and Jerome Friedman became the standard synthesis, and Vladimir Vapnik's support vector machines, developed from the 1960s onward and popularized in the 1990s, offered a principled alternative to trees and neighbors. The lesson that a pattern is only as good as its out-of-sample performance is now the field's first principle1.

Streams, Scale, and the Present

By the 2000s, data arrived continuously—clicks, sensor readings, transactions—and could no longer be stored and mined in batches. Data stream mining developed algorithms that process each record once, in bounded memory, and adapt as the distribution shifts1415. The landmark system was MOA (Massive Online Analysis), released in 2010 by Albert Bifet, Geoff Holmes, Richard Kirkby, and Bernhard Pfahringer, which collected stream classifiers and change detectors. The problem it solved is concrete: a model trained on last year's customer behavior silently degrades as behavior changes, and stream methods detect that drift and update14.

The present field is continuous with this history. Decision trees survive inside gradient-boosted ensembles such as XGBoost (2016), which dominate tabular prediction. Nearest-neighbor and clustering methods remain standard for exploration68. Association mining has receded from the spotlight but persists in recommendation and market-basket analysis. Deep learning has taken over domains with images, text, and speech, where hand-engineered features proved inadequate, but on the structured tables that data mining was built for, tree ensembles still often win. The open questions are the old ones sharpened: how to make complex models interpretable, how to mine data without amplifying bias, how to distinguish real structure from the patterns that any large dataset will contain, and how to keep models honest as the world they describe keeps changing1.

References

  1. Data mining (introduction, Data mining, Background, Etymology, Process, Standards, Results validation), Wikipedia
  2. Exploratory data analysis (Development), Wikipedia
  3. Decision tree learning, Wikipedia
  4. ID3 algorithm, Wikipedia
  5. Instance-based learning, Wikipedia
  6. K-nearest neighbors algorithm, Wikipedia
  7. Nearest neighbor search, Wikipedia
  8. Cluster analysis, Wikipedia
  9. K-means clustering (History), Wikipedia
  10. Association rule learning (History), Wikipedia
  11. Affinity analysis, Wikipedia
  12. Ensemble learning, Wikipedia
  13. Ensemble averaging (machine learning), Wikipedia
  14. Data stream mining, Wikipedia
  15. Streaming algorithm, Wikipedia

The frameworks of Data Mining

Each framework is a way of working the field has used: when it held, and the question it answers.

  1. 1957-Present

    Cluster Analysis

    What natural groups exist in unlabeled data?

    Fly to its tower in Nousfarer

  2. 1965-Present

    Exploratory Data Analysis

    What does the data show before we model it?

    Fly to its tower in Nousfarer

  3. 1967-Present

    Instance-Based Learning

    Can we classify by comparing to stored examples?

    Fly to its tower in Nousfarer

  4. 1980-Present

    Anomaly detection

    Which records are unusual enough to flag?

    Fly to its tower in Nousfarer

  5. 1986-Present

    Decision tree learning

    What sequence of questions predicts an outcome?

    Fly to its tower in Nousfarer

  6. 1989-Present

    Knowledge Discovery in Databases

    How do we turn stored data into knowledge?

    Fly to its tower in Nousfarer

  7. 1990-Present

    Statistical Learning and Validation

    How do we know a pattern is real and not chance?

    Fly to its tower in Nousfarer

  8. 1993-Present

    Association rule learning

    Which items occur together more than expected?

    Fly to its tower in Nousfarer

  9. 1996-Present

    Ensemble Learning

    Can combining many models beat one?

    Fly to its tower in Nousfarer

  10. 1999-Present

    Cross-industry standard process for data mining

    What steps turn a business problem into a deployed model?

    Fly to its tower in Nousfarer

  11. 2000-Present

    Data Stream Mining

    How do we mine data that never stops arriving?

    Fly to its tower in Nousfarer

How the ideas connect

Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.

Cluster AnalysisExploratory Data AnalysisInstance-Based LearningAnomaly detectionDecision tree learningKnowledge Discovery in DatabasesStatistical Learning and ValidationAssociation rule learningEnsemble LearningCross-industry standard process for d…Data Stream Mining

Beyond the field

Where Data Mining's ideas came from in other fields, and what they led to there: the strongest 12 of 29 links.

A city in Nousfarer: a red-carpeted stair climbing terraces to a crowned tower on an acropolis, more towers round it, forest and snowy mountains beyond.
A city in Nousfarer, the game played in the browser.

Data Mining is a planet in Nousfarer

Its city has a tower for each of these 11 frameworks. Land, walk in, and climb one concept at a time.

Land on Data Mining

Related fields

Fields its ideas are linked to

More in Data Science

See where Data Mining sits on the Map of Human Knowledge