- Atlas
- Medicine & Health
- Medical Image Analysis
Medical Image Analysis
Medical image analysis is the part of medical imaging that tries to make a computer answer clinical questions about a picture: where is the organ, how big is the tumour, has it grown since last year, is this tissue normal or abnormal.
10 frameworks, from 1960 to today · Medical Imaging · Medicine & Health
- Handcrafted Feature Engineering 1960-2012
- Deformable Models 1988-2015
- Image registration 1990-Present
- Statistical Shape and Appearance Models 1995-2015
- Voxel-Wise Statistical Pattern Recognition 2000-2015
- Challenge Benchmarks and Shared Evaluation 2007-Present
- Radiomics 2010-Present
- Convolutional neural network 2012-Present
- Uncertainty Quantification and Clinical Validation 2016-Present
- Foundation model 2019-Present
Overview
Medical image analysis is the part of medical imaging that tries to make a computer answer clinical questions about a picture: where is the organ, how big is the tumour, has it grown since last year, is this tissue normal or abnormal1. The scanner and the reconstruction step produce an image; analysis is what turns that image into measurements, labels, and decisions1. Its history is a long migration of methods—from hand-designed rules, through statistical models fitted to populations of images, to deep networks trained on large datasets—driven each time by a problem the previous approach could not solve.
Reading images by hand and by rule
For most of the twentieth century the analysis of medical images was done by radiologists' eyes, and the first computational efforts imitated their reasoning with explicit rules. In the 1960s and 1970s, as computers became available in hospitals, researchers wrote programs that detected edges, thresholded grey levels, and measured regions. A typical early task was counting cells in a microscope image or outlining a heart silhouette on a chest radiograph. These handcrafted feature engineering approaches encoded what an expert believed mattered: a boundary is where intensity changes sharply, a round dark spot on a mammogram may be a mass. The program then applied a classifier—a decision rule learned from labelled examples—to those features. The limitation was that every new organ, scanner, or imaging protocol demanded new features, and features that worked on one dataset often failed on another.
Deformable models: making the boundary an object of physics
A decisive step came in the late 1980s, when Michael Kass, Andrew Witkin, and Demetri Terzopoulos introduced active contours, or "snakes": a curve that moves through the image under forces pulling it toward edges while its own stiffness keeps it smooth. The idea reframed segmentation as an energy-minimisation problem, and it was quickly taken up in medical imaging, where boundaries are often faint and broken. In the 1990s, deformable models in the medical literature—geometric active contours that could split and merge, and later level-set methods—let a single framework find the inner and outer wall of the heart or the surface of the liver. The appeal was that prior knowledge of shape could be built into the physics of the curve, so the model did not depend on a perfect edge map.
Registration and atlases: using other people's anatomy
A second strand addressed a different problem: images of the same patient taken at different times, or images of different patients, do not line up2. Image registration is the computation of a spatial transformation that maps one image onto another, and by the 1990s it had become a mature field, with rigid, affine, and non-rigid methods, and with mutual information as a similarity measure that worked across modalities such as CT and MRI. Registration made atlas-based segmentation possible: build an average anatomical template from many labelled scans, register the template to a new patient, and carry the labels across1. This was powerful for structures with stable anatomy, such as brain regions in MRI, and it established the habit of treating a population of images as a statistical object1.
Statistical shape and appearance models
That habit crystallised in the 1990s in statistical shape and appearance models, most influentially the active shape and active appearance models developed by Timothy Cootes, Christopher Taylor, and colleagues13. From a training set of outlines or images, principal component analysis extracted the main modes of variation—how a set of hand outlines of the same organ typically bends and stretches34. A model could then search a new image only within plausible shapes, which made segmentation robust to noise and partial occlusion. The same statistical logic spread to voxel-wise statistical machine learning: instead of modelling a boundary, treat every voxel's intensity as a feature and learn a classifier that labels each voxel as tissue or background. This "pattern recognition on voxels" approach, popular in the 2000s, was simple and general, and it made brain lesion and tumour segmentation a standard benchmark task.
Benchmarks, challenges, and the culture of comparison
As methods multiplied, the field confronted a problem of evidence: papers reported results on private datasets, so claims could not be compared. The response, from the late 2000s onward, was the challenge dataset: a common training set, a held-out test set, and a leaderboard. The MICCAI conference's grand challenges—on liver segmentation, brain tumour segmentation, and later many others—became the field's proving ground, and the practice spread to journals and societies. This changed what counted as a contribution: a method now had to be evaluated under shared rules, and the datasets themselves became community resources. The same period saw the rise of radiomics, the extraction of large numbers of quantitative features from images—texture, shape, intensity statistics—to build imaging biomarkers that predict diagnosis, prognosis, or treatment response56. Radiomics made explicit what had been implicit: that an image is a measurement, and that its quantitative content can be mined5.
Deep learning: the end of handcrafted features
The turning point came in the 2010s with deep convolutional neural networks. A convolutional network learns its own features from raw pixels by stacking filters that detect increasingly abstract patterns, and it needs no hand-designed edge or texture descriptors7. The demonstration that deep networks could match or beat human readers on natural-image recognition, and then on medical tasks, was decisive. In 2016 and after, networks won major challenges in skin lesion classification, diabetic retinopathy grading, and lung nodule detection; the field's methods sections shifted almost wholesale from feature engineering to network architecture. The earlier statistical and deformable approaches did not vanish—they were absorbed as priors, loss functions, or post-processing—but the centre of gravity moved to learned representations.
Uncertainty, validation, and the clinical question
Deep networks brought a new problem: they are confident even when wrong, and their behaviour on data unlike their training set is unpredictable. The field responded with uncertainty quantification—methods that estimate how sure a model is, such as Bayesian approximations, ensembles, and test-time augmentation—and with a renewed emphasis on clinical validation: prospective trials, reader studies, and regulatory review. The lesson of the 2020s is that a high benchmark score is not evidence of clinical benefit; the question is whether the analysis changes a decision and improves an outcome.
Foundation models and the present
The most recent turn is toward medical foundation and self-supervised models: large networks pre-trained on vast unlabelled image collections, then adapted to specific tasks with little labelled data. Self-supervised learning—training a model to predict parts of its input, such as masked patches, without labels—lets the field exploit the enormous archives of clinical images that are never annotated8. These models descend from the same statistical instinct as atlases and shape models: learn the distribution of normal anatomy, then detect departures from it. Today the main approaches coexist—deep networks for segmentation and detection, registration and atlas methods for anatomy, radiomics for quantitative biomarkers—and the open questions are familiar in new form: how to generalise across scanners and populations, how to quantify uncertainty reliably, and how to prove that an analysis helps a patient.
References
- Medical image computing (introduction, Segmentation, Registration), Wikipedia
- Image registration, Wikipedia
- Active shape model, Wikipedia
- Statistical shape analysis, Wikipedia
- Radiomics, Wikipedia
- Imaging biomarker, Wikipedia
- Convolutional neural network, Wikipedia
- Self-supervised learning, Wikipedia
The frameworks of Medical Image Analysis
Each framework is a way of working the field has used: when it held, and the question it answers.
- 1960-2012
Handcrafted Feature Engineering
Which designed features best capture what a clinician sees?
- 1988-2015
- 1990-Present
- 1995-2015
Statistical Shape and Appearance Models
What shapes are plausible for this organ?
- 2000-2015
Voxel-Wise Statistical Pattern Recognition
Can each voxel be classified from its intensity?
- 2010-Present
- 2012-Present
Convolutional neural network
Can networks learn features directly from pixels?
- 2016-Present
Uncertainty Quantification and Clinical Validation
How sure is the model, and does it help?
- 2019-Present
How the ideas connect
Each arrow runs from an earlier idea to the later one it shaped. The list below says what passed between them.
Convolutional neural network replaced Handcrafted Feature Engineering
Deep convolutional networks replaced hand-designed features by learning filters directly from pixels, and after 2012 the field's methods shifted almost wholesale from feature engineering to network architectures. Handcrafted descriptors survived only as inputs or baselines.
Carried by Alex Krizhevsky; Ilya Sutskever; Geoffrey Hinton.
Foundation model grew out of Convolutional neural network
Medical foundation models are large convolutional or transformer networks pre-trained with self-supervision on vast image collections, then fine-tuned; they are a direct extension of the deep-learning paradigm to unlabelled data at scale. The architecture and training machinery come straight from the CNN era.
Voxel-Wise Statistical Pattern Recognition grew out of Handcrafted Feature Engineering
Voxel-wise pattern recognition kept the handcrafted-feature-and-classifier pipeline but moved the feature from designed descriptors to raw voxel intensities, applying standard classifiers to each voxel. It was a direct generalisation of the earlier feature-engineering approach to dense image data.
Radiomics grew out of Handcrafted Feature Engineering
Radiomics grew out of handcrafted feature engineering by scaling up the extraction of texture, shape and intensity descriptors to hundreds of quantitative features per tumour, then mining them as biomarkers. The feature definitions themselves (Haralick textures, shape descriptors) came from the earlier image-analysis literature.
Statistical Shape and Appearance Models influenced by Deformable Models
Active shape and appearance models by Cootes and Taylor built on the deformable-model idea of a curve constrained by internal forces, but replaced hand-set stiffness with modes of shape variation learned from a training set. The search procedure still deforms a model to fit image evidence.
Carried by Timothy Cootes; Christopher Taylor.
Convolutional neural network replaced Voxel-Wise Statistical Pattern Recognition
Voxel-wise classifiers that treated each voxel independently were displaced by convolutional networks that learn spatial context and features end-to-end. The patch-based CNN segmentation literature of the mid-2010s explicitly replaced per-voxel classifiers with learned hierarchical features.
Convolutional neural network influenced by Challenge Benchmarks and Shared Evaluation
Challenge benchmarks provided the large, standardised, annotated datasets and held-out test sets on which deep networks were trained and compared, and the leaderboard culture pushed the shift to CNNs. Victories in MICCAI and other challenges (e.g., brain tumour, lung nodule) demonstrated the advantage of deep learning.
Uncertainty Quantification and Clinical Validation influenced by Challenge Benchmarks and Shared Evaluation
The benchmark culture established shared evaluation and leaderboards, and the recognition that high scores on curated test sets did not guarantee clinical benefit led directly to demands for uncertainty quantification and prospective clinical validation. Challenge organisers and participants drove the call for stronger evidence.
Uncertainty Quantification and Clinical Validation influenced by Convolutional neural network
Deep networks' overconfidence and unpredictable behaviour on out-of-distribution data created the problem that uncertainty quantification and clinical validation were designed to solve. Methods such as Bayesian approximations, ensembles and test-time augmentation were applied to CNN outputs.
Statistical Shape and Appearance Models influenced by Image registration
Statistical shape models were often built from point correspondences established by registering training images to a template, so registration supplied the correspondence step that made population shape statistics possible. Atlas-based methods also shared the habit of treating a registered population as a statistical object.
Voxel-Wise Statistical Pattern Recognition influenced by Image registration
Voxel-wise classification in the brain typically relied on registering each subject to a common space so that voxel locations were comparable across the training population. Registration provided the spatial normalisation that made voxel-wise learning meaningful.
Voxel-Wise Statistical Pattern Recognition rival of Statistical Shape and Appearance Models
In the 2000s, shape-model segmentation and voxel-wise pattern recognition competed as alternative ways to segment the same structures: one modelled boundaries with learned shape priors, the other labelled each voxel independently with a classifier. Both were evaluated on the same brain-lesion and tumour tasks.
Uncertainty Quantification and Clinical Validation influenced by Radiomics
Radiomics had promoted quantitative imaging biomarkers for clinical decision-making, and its struggles with reproducibility and validation reinforced the later emphasis on uncertainty and clinical validation. The radiomics literature's reporting guidelines and multi-centre validation efforts fed into the broader validation movement.
Beyond the field
Where Medical Image Analysis's ideas came from in other fields, and what they led to there: the strongest 12 of 31 links.
Statistical Shape and Appearance Models influenced by Statistical Appearance Models and Subspace Learning in Computer Vision
Statistical appearance models and subspace learning (eigenfaces, PCA appearance models) provided the direct methodological template for Active Appearance Models and statistical shape/appearance models in medical imaging. Cootes and Taylor's framework generalized these subspace techniques to medical anatomy.
Carried by Timothy F. Cootes; Christopher J. Taylor.
Voxel-Wise Statistical Pattern Recognition influenced by Statistical Parametric Mapping in Neuroimaging
Statistical Parametric Mapping established the voxel-wise mass-univariate framework and its later multivariate extensions that voxel-wise pattern recognition in neuroimaging built upon and reacted against. SPM's voxel-wise statistical machinery is the direct antecedent.
Carried by Karl Friston.
Foundation model influenced by Large language model in Artificial Intelligence
Large language models supplied the transformer architecture, pretraining-then-adaptation recipe and scaling behaviour that the foundation model concept generalised beyond text. The foundation model idea was formulated by researchers extending LLM practice to other modalities.
Carried by Attention Is All You Need (2017).
Foundation model grew out of Deep learning in Artificial Intelligence
Foundation models are a direct outgrowth of deep learning: large pretrained neural networks adapted to many downstream tasks are the deep learning programme pushed to scale and generality. The concept generalises deep learning's pretrain-and-finetune practice.
Foundation model grew out of Deep Learning at Scale in Deep Learning
Deep learning at scale is the immediate parent of the foundation model idea: the observation that scaling data, parameters and compute yields emergent general capabilities is what the framework formalised. The scaling results carried directly into the foundation model concept.
Handcrafted Feature Engineering influenced by Stereology and quantitative image analysis in Materials Characterization
Stereology supplied the quantitative morphometric toolkit (point counting, feature measurement on sections) that early handcrafted medical image features drew on. Texture and shape descriptors in medical image analysis borrowed these stereological measurement conventions.
Handcrafted Feature Engineering influenced by Frequency-domain analysis in Biomedical Signal Processing
Frequency-domain methods such as Fourier descriptors and Gabor filters, developed in signal analysis, became standard handcrafted texture and shape features in medical image analysis. The same transform machinery passed from signal processing into image feature engineering.
Handcrafted Feature Engineering influenced Geometric Feature Matching in Computer Vision
Geometric feature matching built on the handcrafted local descriptors and detectors developed in medical and general image analysis, such as scale-invariant keypoints and their descriptors, using them as the primitives whose correspondences were then matched under geometric constraints.
Carried by David G. Lowe.
Deformable Models influenced by Soft Tissue Constitutive Modeling in Biomechanics
Continuum-mechanical soft tissue constitutive models (elasticity, energy functionals) supplied the internal deformation energies used in snakes and active contour models. Kass, Terzopoulos and colleagues framed deformable models as energy-minimizing elastic bodies drawing on this biomechanics tradition.
Carried by Michael Kass; Demetri Terzopoulos.
Deformable Models influenced by Implicit Surfaces and Distance Fields in Geometric Modeling
Implicit surfaces and level-set distance fields provided the numerical machinery for geometric deformable models, letting contours evolve as level sets of a higher-dimensional function. This passed into medical image segmentation in the mid-1990s.
Image registration influenced by Magnetic Resonance Imaging in Medical Imaging
MRI's multi-modal, multi-sequence acquisition created the practical need and the data for intensity-based registration, and MRI physics informed similarity measures and distortion correction. Registration methods were developed and validated largely on MR images.
Voxel-Wise Statistical Pattern Recognition influenced by Pattern recognition in Biomedical Signal Processing
Pattern recognition methods (feature extraction, classifiers) developed in biomedical signal processing were carried over to voxel-wise classification of brain images. The classifier toolkit passed into voxel-based pattern recognition.

Medical Image Analysis is a planet in Nousfarer
Its city has a tower for each of these 10 frameworks. Land, walk in, and climb one concept at a time.
Land on Medical Image AnalysisRelated fields
Fields its ideas are linked to
- Medical Imaging 5 links
- Computer Vision 4 links
- Artificial Intelligence 3 links
- Deep Learning 2 links
- Biomedical Signal Processing 2 links
- Geometric Modeling 2 links
- Quantitative Imaging 2 links
- Statistical Learning 2 links
- Machine Learning 1 link
- Numerical Analysis 1 link
- Biomechanics 1 link
- Clinical Informatics 1 link
More in Medical Imaging
- Diagnostic Radiology
- Image Reconstruction
- Imaging Informatics
- Imaging Physics
- Interventional Radiology
- Medical Imaging
- Molecular Imaging
- Nuclear Medicine
- Quantitative Imaging
See where Medical Image Analysis sits on the Map of Human Knowledge