Medical Image Analysis

Medical image analysis is the part of medical imaging that tries to make a computer answer clinical questions about a picture: where is the organ, how big is the tumour, has it grown since last year, is this tissue normal or abnormal.

10 frameworks, from 1960 to today · Medical Imaging · Medicine & Health

still a working view ended · each leads to its entry below

Overview

Medical image analysis is the part of medical imaging that tries to make a computer answer clinical questions about a picture: where is the organ, how big is the tumour, has it grown since last year, is this tissue normal or abnormal1. The scanner and the reconstruction step produce an image; analysis is what turns that image into measurements, labels, and decisions1. Its history is a long migration of methods—from hand-designed rules, through statistical models fitted to populations of images, to deep networks trained on large datasets—driven each time by a problem the previous approach could not solve.

Reading images by hand and by rule

For most of the twentieth century the analysis of medical images was done by radiologists' eyes, and the first computational efforts imitated their reasoning with explicit rules. In the 1960s and 1970s, as computers became available in hospitals, researchers wrote programs that detected edges, thresholded grey levels, and measured regions. A typical early task was counting cells in a microscope image or outlining a heart silhouette on a chest radiograph. These handcrafted feature engineering approaches encoded what an expert believed mattered: a boundary is where intensity changes sharply, a round dark spot on a mammogram may be a mass. The program then applied a classifier—a decision rule learned from labelled examples—to those features. The limitation was that every new organ, scanner, or imaging protocol demanded new features, and features that worked on one dataset often failed on another.

Deformable models: making the boundary an object of physics

A decisive step came in the late 1980s, when Michael Kass, Andrew Witkin, and Demetri Terzopoulos introduced active contours, or "snakes": a curve that moves through the image under forces pulling it toward edges while its own stiffness keeps it smooth. The idea reframed segmentation as an energy-minimisation problem, and it was quickly taken up in medical imaging, where boundaries are often faint and broken. In the 1990s, deformable models in the medical literature—geometric active contours that could split and merge, and later level-set methods—let a single framework find the inner and outer wall of the heart or the surface of the liver. The appeal was that prior knowledge of shape could be built into the physics of the curve, so the model did not depend on a perfect edge map.

Registration and atlases: using other people's anatomy

A second strand addressed a different problem: images of the same patient taken at different times, or images of different patients, do not line up2. Image registration is the computation of a spatial transformation that maps one image onto another, and by the 1990s it had become a mature field, with rigid, affine, and non-rigid methods, and with mutual information as a similarity measure that worked across modalities such as CT and MRI. Registration made atlas-based segmentation possible: build an average anatomical template from many labelled scans, register the template to a new patient, and carry the labels across1. This was powerful for structures with stable anatomy, such as brain regions in MRI, and it established the habit of treating a population of images as a statistical object1.

Statistical shape and appearance models

That habit crystallised in the 1990s in statistical shape and appearance models, most influentially the active shape and active appearance models developed by Timothy Cootes, Christopher Taylor, and colleagues13. From a training set of outlines or images, principal component analysis extracted the main modes of variation—how a set of hand outlines of the same organ typically bends and stretches34. A model could then search a new image only within plausible shapes, which made segmentation robust to noise and partial occlusion. The same statistical logic spread to voxel-wise statistical machine learning: instead of modelling a boundary, treat every voxel's intensity as a feature and learn a classifier that labels each voxel as tissue or background. This "pattern recognition on voxels" approach, popular in the 2000s, was simple and general, and it made brain lesion and tumour segmentation a standard benchmark task.

Benchmarks, challenges, and the culture of comparison

As methods multiplied, the field confronted a problem of evidence: papers reported results on private datasets, so claims could not be compared. The response, from the late 2000s onward, was the challenge dataset: a common training set, a held-out test set, and a leaderboard. The MICCAI conference's grand challenges—on liver segmentation, brain tumour segmentation, and later many others—became the field's proving ground, and the practice spread to journals and societies. This changed what counted as a contribution: a method now had to be evaluated under shared rules, and the datasets themselves became community resources. The same period saw the rise of radiomics, the extraction of large numbers of quantitative features from images—texture, shape, intensity statistics—to build imaging biomarkers that predict diagnosis, prognosis, or treatment response56. Radiomics made explicit what had been implicit: that an image is a measurement, and that its quantitative content can be mined5.

Deep learning: the end of handcrafted features

The turning point came in the 2010s with deep convolutional neural networks. A convolutional network learns its own features from raw pixels by stacking filters that detect increasingly abstract patterns, and it needs no hand-designed edge or texture descriptors7. The demonstration that deep networks could match or beat human readers on natural-image recognition, and then on medical tasks, was decisive. In 2016 and after, networks won major challenges in skin lesion classification, diabetic retinopathy grading, and lung nodule detection; the field's methods sections shifted almost wholesale from feature engineering to network architecture. The earlier statistical and deformable approaches did not vanish—they were absorbed as priors, loss functions, or post-processing—but the centre of gravity moved to learned representations.

Uncertainty, validation, and the clinical question

Deep networks brought a new problem: they are confident even when wrong, and their behaviour on data unlike their training set is unpredictable. The field responded with uncertainty quantification—methods that estimate how sure a model is, such as Bayesian approximations, ensembles, and test-time augmentation—and with a renewed emphasis on clinical validation: prospective trials, reader studies, and regulatory review. The lesson of the 2020s is that a high benchmark score is not evidence of clinical benefit; the question is whether the analysis changes a decision and improves an outcome.

Foundation models and the present

The most recent turn is toward medical foundation and self-supervised models: large networks pre-trained on vast unlabelled image collections, then adapted to specific tasks with little labelled data. Self-supervised learning—training a model to predict parts of its input, such as masked patches, without labels—lets the field exploit the enormous archives of clinical images that are never annotated8. These models descend from the same statistical instinct as atlases and shape models: learn the distribution of normal anatomy, then detect departures from it. Today the main approaches coexist—deep networks for segmentation and detection, registration and atlas methods for anatomy, radiomics for quantitative biomarkers—and the open questions are familiar in new form: how to generalise across scanners and populations, how to quantify uncertainty reliably, and how to prove that an analysis helps a patient.

References

  1. Medical image computing (introduction, Segmentation, Registration), Wikipedia
  2. Image registration, Wikipedia
  3. Active shape model, Wikipedia
  4. Statistical shape analysis, Wikipedia
  5. Radiomics, Wikipedia
  6. Imaging biomarker, Wikipedia
  7. Convolutional neural network, Wikipedia
  8. Self-supervised learning, Wikipedia

The frameworks of Medical Image Analysis

Each framework is a way of working the field has used: when it held, and the question it answers.

  1. 1960-2012

    Handcrafted Feature Engineering

    Which designed features best capture what a clinician sees?

    Fly to its tower in Nousfarer

  2. 1988-2015

    Deformable Models

    How can a curve find a faint boundary?

    Fly to its tower in Nousfarer

  3. 1990-Present

    Image registration

    How do we line up two images of the same anatomy?

    Fly to its tower in Nousfarer

  4. 1995-2015
  5. 2000-2015

    Voxel-Wise Statistical Pattern Recognition

    Can each voxel be classified from its intensity?

    Fly to its tower in Nousfarer

  6. 2007-Present
  7. 2010-Present

    Radiomics

    What quantitative biomarkers can be mined from images?

    Fly to its tower in Nousfarer

  8. 2012-Present

    Convolutional neural network

    Can networks learn features directly from pixels?

    Fly to its tower in Nousfarer

  9. 2016-Present
  10. 2019-Present

    Foundation model

    Can one large model adapt to many tasks?

    Fly to its tower in Nousfarer

How the ideas connect

Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.

Handcrafted Feature EngineeringDeformable ModelsImage registrationStatistical Shape and Appearance Mode…Voxel-Wise Statistical Pattern Recogn…Challenge Benchmarks and Shared Evalu…RadiomicsConvolutional neural networkUncertainty Quantification and Clinic…Foundation model

Beyond the field

Where Medical Image Analysis's ideas came from in other fields, and what they led to there: the strongest 12 of 31 links.

A city in Nousfarer: a red-carpeted stair climbing terraces to a crowned tower on an acropolis, more towers round it, forest and snowy mountains beyond.
A city in Nousfarer, the game played in the browser.

Medical Image Analysis is a planet in Nousfarer

Its city has a tower for each of these 10 frameworks. Land, walk in, and climb one concept at a time.

Land on Medical Image Analysis

Related fields

Fields its ideas are linked to

More in Medical Imaging

See where Medical Image Analysis sits on the Map of Human Knowledge