- Atlas
- Computer Science
- Malware Analysis
Malware Analysis
Malware analysis is the practice of taking a program that was written to do harm—steal data, hold files for ransom, spy on a user, spread itself—and working out what it is, what it does, and how it does it.
9 frameworks, from 1987 to today · Cybersecurity · Computer Science
- Signature-based detection 1987-Present
- Static program analysis 1990-Present
- Dynamic program analysis 1995-Present
- Emulation and heuristic analysis 1995-Present
- Unpacking and de-obfuscation 1995-Present
- Memory forensics 2007-Present
- Automated sandboxing 2010-Present
- Machine learning classification 2010-Present
- Threat intelligence and attribution 2010-Present
Overview
Malware analysis is the practice of taking a program that was written to do harm—steal data, hold files for ransom, spy on a user, spread itself—and working out what it is, what it does, and how it does it1. The analyst's product is understanding: a description of behavior, a set of indicators that let defenders recognize the same thing elsewhere, and often a decision about whether and how to remove it1. What makes the field distinctive is that its object is deliberately hostile. A malicious program is built to resist being read, to hide its purpose, and to change shape when it is detected. So malware analysis has always been a contest, and its history is a history of the analyst's tools being answered by the attacker's countermeasures, and then by new tools.
The age of boot-sector viruses and the first scanners
Malicious code appeared almost as soon as personal computers were networked by floppy disk. The first widely recognized PC virus, Brain, emerged in 1986 and infected the boot sector of floppy disks23. Through the late 1980s and early 1990s, most malware was of this self-replicating kind: boot-sector viruses, file infectors that attached themselves to executable programs, and later macro viruses that spread through documents. The dominant defensive response was the antivirus scanner, and its core technique was signature-based detection: a signature is a short, distinctive byte pattern—a "fingerprint"—taken from a known piece of malware, which a scanner searches for in files. The approach worked because early malware was largely static; the same bytes appeared on every infected machine.
Signature analysis is itself a form of malware analysis. To write a signature, an analyst has to disassemble or inspect the sample and choose a byte sequence that is unique to the malware and unlikely to appear in legitimate software. The weakness was built in from the start: a signature only recognizes what has already been seen, so defenders were always one step behind, and attackers learned that changing a few bytes could defeat a scanner.
Static analysis and the disassembler
The foundational analyst's skill became static analysis: examining a program without running it. The tools are the reverse engineer's tools—a disassembler turns machine code back into assembly language, a debugger lets the analyst step through execution, and a hex editor exposes raw bytes. The lineage runs back to the earliest reverse-engineering work; the interactive debugger that became the standard for Windows analysis, OllyDbg, appeared in the early 2000s, and the disassembler IDA Pro, first released in the mid-1990s, became the workhorse for reading compiled binaries. Static analysis can reveal everything a program is capable of, including code paths that only trigger under rare conditions, but it demands expertise and time, and it struggles against code that is packed or encrypted.
Packers, obfuscation, and the unpacking problem
Attackers answered static analysis with packing: compressing or encrypting the real malicious code and wrapping it in a small "stub" that decrypts and runs it in memory at startup. A packed sample shows the analyst almost nothing useful in its stored form. The countermeasure was unpacking—running the program under a debugger until the stub has decrypted the payload, then dumping the now-exposed code from memory and analyzing that. Related tricks multiplied: obfuscation that mangles control flow, junk code inserted to confuse disassembly, and anti-debugging checks that detect an analyst's tools and refuse to run. Unpacking and de-obfuscation became a specialized craft of their own, and the arms race between packers and unpackers is one of the field's enduring themes.
Dynamic analysis, emulation, and heuristics
Because static reading is slow and can be defeated, analysts turned to dynamic analysis: running the malware in a controlled environment and observing what it actually does—which files it touches, which network addresses it contacts, which registry keys it changes1. The problem is containment. Running live malware on a real machine risks infecting it and letting the malware escape onto the network1. The solution was emulation and isolation: execute the sample inside a simulated or sandboxed environment where its actions can be watched safely14.
Emulation also enabled a smarter kind of detection. Rather than matching bytes, a scanner could run a file inside a CPU emulator for a short time and watch for suspicious behavior—heuristic analysis, which judges a program by what it appears to do rather than by a fixed fingerprint. This caught variants that byte-matching missed, at the cost of false positives, since legitimate programs sometimes behave suspiciously.
Automated sandboxing
The next step was to make dynamic analysis automatic and high-volume. Automated sandboxes—services that accept an uploaded sample, run it in an instrumented virtual machine, and return a report of its behavior—turned malware analysis from a craft performed one sample at a time into something closer to an assembly line. The open-source Cuckoo sandbox, released in the early 2010s, and a number of commercial services made this routine. The reports they produce—file-system changes, network connections, process activity—became a standard currency of the field. Attackers responded with sandbox evasion: code that detects it is running in a virtual machine or under instrumentation and stays dormant, or waits, until it believes it is on a real victim's machine1.
Memory forensics and fileless malware
As attackers grew better at evading disk-based detection, a class of malware that lives only in memory—fileless malware—became prominent. It may arrive through a script, a malicious document, or a compromised legitimate tool, and it never writes an executable file to disk, so file scanning finds nothing. The answer was memory forensics: capturing a running system's RAM and analyzing it for injected code, hidden processes, and malicious artifacts that exist only while the machine runs. The Volatility framework, an open-source memory-analysis tool, became central to this work. Memory analysis also exposed techniques like process injection, where malware hides inside a legitimate running process.
Machine learning and classification at scale
The sheer volume of new samples—millions per year—made manual analysis of everything impossible. Analysts and vendors turned to machine learning to classify files automatically, training models on features extracted from known malware and benign software to predict whether a new sample is malicious and which family it belongs to. The appeal is scale; the difficulty is that attackers can probe and adapt to a model, and that a classifier's reasoning is often opaque. Machine learning became a standard layer in detection pipelines rather than a replacement for human analysis.
Threat intelligence and attribution
Finally, malware analysis increasingly fed a broader practice: threat intelligence. Indicators extracted from a sample—domain names, IP addresses, file hashes, code similarities—are shared among defenders so that one organization's analysis protects others. Analysts also attempt attribution, linking campaigns to particular groups or actors by shared tools, infrastructure, and code. Attribution is contested and often uncertain, since attackers deliberately reuse and borrow code to confuse it, but the effort to connect individual samples to ongoing campaigns reshaped how the field thinks about its work: not just "what does this file do" but "who is behind it and what are they after."
The present
Today's malware analysis combines all of these layers1. A typical pipeline runs a sample through static inspection, an automated sandbox, and a classifier, then hands the hard cases to a human analyst who unpacks, debugs, and reads code, often finishing in memory forensics when nothing touches disk. The open questions are the old ones in new form: how to analyze malware that detects and evades every automated environment; how to trust machine-learning verdicts that attackers can game; how to attribute attacks without overclaiming; and how to keep pace when the volume of new samples outstrips any team of humans. The field's history is a record of each answer being met by a counter, and of analysts building the next tool in response.
References
- Malware analysis (introduction, Applications, Dynamic analysis, Combined analysis), Wikipedia
- Malware (History), Wikipedia
- Rootkit (History), Wikipedia
- Sandbox (computer security), Wikipedia
The frameworks of Malware Analysis
Each framework is a way of working the field has used: when it held, and the question it answers.
- 1987-Present
- 1990-Present
- 1995-Present
- 1995-Present
- 1995-Present
- 2007-Present
- 2010-Present
- 2010-Present
- 2010-Present

Malware Analysis is a planet in Nousfarer
Its city has a tower for each of these 9 frameworks. Land, walk in, and climb one concept at a time.
Land on Malware AnalysisRelated fields
More in Cybersecurity
- Applied Cryptography
- Cybersecurity
- Digital Forensics
- Digital Privacy
- Network Security
- Security Economics
- Security Theory
- Software Security
- Systems Security
See where Malware Analysis sits on the Map of Human Knowledge