Malware Analysis

Malware analysis is the practice of taking a program that was written to do harm—steal data, hold files for ransom, spy on a user, spread itself—and working out what it is, what it does, and how it does it.

9 frameworks, from 1987 to today · Cybersecurity · Computer Science

still a working view ended · each leads to its entry below

Overview

Malware analysis is the practice of taking a program that was written to do harm—steal data, hold files for ransom, spy on a user, spread itself—and working out what it is, what it does, and how it does it1. The analyst's product is understanding: a description of behavior, a set of indicators that let defenders recognize the same thing elsewhere, and often a decision about whether and how to remove it1. What makes the field distinctive is that its object is deliberately hostile. A malicious program is built to resist being read, to hide its purpose, and to change shape when it is detected. So malware analysis has always been a contest, and its history is a history of the analyst's tools being answered by the attacker's countermeasures, and then by new tools.

The age of boot-sector viruses and the first scanners

Malicious code appeared almost as soon as personal computers were networked by floppy disk. The first widely recognized PC virus, Brain, emerged in 1986 and infected the boot sector of floppy disks23. Through the late 1980s and early 1990s, most malware was of this self-replicating kind: boot-sector viruses, file infectors that attached themselves to executable programs, and later macro viruses that spread through documents. The dominant defensive response was the antivirus scanner, and its core technique was signature-based detection: a signature is a short, distinctive byte pattern—a "fingerprint"—taken from a known piece of malware, which a scanner searches for in files. The approach worked because early malware was largely static; the same bytes appeared on every infected machine.

Signature analysis is itself a form of malware analysis. To write a signature, an analyst has to disassemble or inspect the sample and choose a byte sequence that is unique to the malware and unlikely to appear in legitimate software. The weakness was built in from the start: a signature only recognizes what has already been seen, so defenders were always one step behind, and attackers learned that changing a few bytes could defeat a scanner.

Static analysis and the disassembler

The foundational analyst's skill became static analysis: examining a program without running it. The tools are the reverse engineer's tools—a disassembler turns machine code back into assembly language, a debugger lets the analyst step through execution, and a hex editor exposes raw bytes. The lineage runs back to the earliest reverse-engineering work; the interactive debugger that became the standard for Windows analysis, OllyDbg, appeared in the early 2000s, and the disassembler IDA Pro, first released in the mid-1990s, became the workhorse for reading compiled binaries. Static analysis can reveal everything a program is capable of, including code paths that only trigger under rare conditions, but it demands expertise and time, and it struggles against code that is packed or encrypted.

Packers, obfuscation, and the unpacking problem

Attackers answered static analysis with packing: compressing or encrypting the real malicious code and wrapping it in a small "stub" that decrypts and runs it in memory at startup. A packed sample shows the analyst almost nothing useful in its stored form. The countermeasure was unpacking—running the program under a debugger until the stub has decrypted the payload, then dumping the now-exposed code from memory and analyzing that. Related tricks multiplied: obfuscation that mangles control flow, junk code inserted to confuse disassembly, and anti-debugging checks that detect an analyst's tools and refuse to run. Unpacking and de-obfuscation became a specialized craft of their own, and the arms race between packers and unpackers is one of the field's enduring themes.

Dynamic analysis, emulation, and heuristics

Because static reading is slow and can be defeated, analysts turned to dynamic analysis: running the malware in a controlled environment and observing what it actually does—which files it touches, which network addresses it contacts, which registry keys it changes1. The problem is containment. Running live malware on a real machine risks infecting it and letting the malware escape onto the network1. The solution was emulation and isolation: execute the sample inside a simulated or sandboxed environment where its actions can be watched safely14.

Emulation also enabled a smarter kind of detection. Rather than matching bytes, a scanner could run a file inside a CPU emulator for a short time and watch for suspicious behavior—heuristic analysis, which judges a program by what it appears to do rather than by a fixed fingerprint. This caught variants that byte-matching missed, at the cost of false positives, since legitimate programs sometimes behave suspiciously.

Automated sandboxing

The next step was to make dynamic analysis automatic and high-volume. Automated sandboxes—services that accept an uploaded sample, run it in an instrumented virtual machine, and return a report of its behavior—turned malware analysis from a craft performed one sample at a time into something closer to an assembly line. The open-source Cuckoo sandbox, released in the early 2010s, and a number of commercial services made this routine. The reports they produce—file-system changes, network connections, process activity—became a standard currency of the field. Attackers responded with sandbox evasion: code that detects it is running in a virtual machine or under instrumentation and stays dormant, or waits, until it believes it is on a real victim's machine1.

Memory forensics and fileless malware

As attackers grew better at evading disk-based detection, a class of malware that lives only in memory—fileless malware—became prominent. It may arrive through a script, a malicious document, or a compromised legitimate tool, and it never writes an executable file to disk, so file scanning finds nothing. The answer was memory forensics: capturing a running system's RAM and analyzing it for injected code, hidden processes, and malicious artifacts that exist only while the machine runs. The Volatility framework, an open-source memory-analysis tool, became central to this work. Memory analysis also exposed techniques like process injection, where malware hides inside a legitimate running process.

Machine learning and classification at scale

The sheer volume of new samples—millions per year—made manual analysis of everything impossible. Analysts and vendors turned to machine learning to classify files automatically, training models on features extracted from known malware and benign software to predict whether a new sample is malicious and which family it belongs to. The appeal is scale; the difficulty is that attackers can probe and adapt to a model, and that a classifier's reasoning is often opaque. Machine learning became a standard layer in detection pipelines rather than a replacement for human analysis.

Threat intelligence and attribution

Finally, malware analysis increasingly fed a broader practice: threat intelligence. Indicators extracted from a sample—domain names, IP addresses, file hashes, code similarities—are shared among defenders so that one organization's analysis protects others. Analysts also attempt attribution, linking campaigns to particular groups or actors by shared tools, infrastructure, and code. Attribution is contested and often uncertain, since attackers deliberately reuse and borrow code to confuse it, but the effort to connect individual samples to ongoing campaigns reshaped how the field thinks about its work: not just "what does this file do" but "who is behind it and what are they after."

The present

Today's malware analysis combines all of these layers1. A typical pipeline runs a sample through static inspection, an automated sandbox, and a classifier, then hands the hard cases to a human analyst who unpacks, debugs, and reads code, often finishing in memory forensics when nothing touches disk. The open questions are the old ones in new form: how to analyze malware that detects and evades every automated environment; how to trust machine-learning verdicts that attackers can game; how to attribute attacks without overclaiming; and how to keep pace when the volume of new samples outstrips any team of humans. The field's history is a record of each answer being met by a counter, and of analysts building the next tool in response.

References

  1. Malware analysis (introduction, Applications, Dynamic analysis, Combined analysis), Wikipedia
  2. Malware (History), Wikipedia
  3. Rootkit (History), Wikipedia
  4. Sandbox (computer security), Wikipedia

The frameworks of Malware Analysis

Each framework is a way of working the field has used: when it held, and the question it answers.

  1. 1987-Present
  2. 1990-Present
  3. 1995-Present
  4. 1995-Present
  5. 1995-Present
  6. 2007-Present
  7. 2010-Present
  8. 2010-Present
  9. 2010-Present
A city in Nousfarer: a red-carpeted stair climbing terraces to a crowned tower on an acropolis, more towers round it, forest and snowy mountains beyond.
A city in Nousfarer, the game played in the browser.

Malware Analysis is a planet in Nousfarer

Its city has a tower for each of these 9 frameworks. Land, walk in, and climb one concept at a time.

Land on Malware Analysis

Related fields

More in Cybersecurity

See where Malware Analysis sits on the Map of Human Knowledge