Malware analysis: concepts

Every SOC alert that says "suspicious file" ends with someone answering one question: what does this program do? Malware analysts answer it for incident response (what did it do here?), for threat intel (what can this family do?), and for detection engineering (what rule catches the next sample?). The work splits into two complementary approaches — static (read it without running it) and dynamic (run it and watch) — and the professional skill is knowing which to use when, and how each defeats the other's blind spots.

Level 3 · Apply — do it unaided on an unfamiliar authorized target, and explain why it worked and what would have stopped it

Walkthrough run 2026-10-09 · synthetic benign training files only; real YARA via yara-python; detonation is reader-side in their isolated lab

1. Where this shows up

Every SOC alert that says "suspicious file" ends with someone answering one question: what does this program do? Malware analysts answer it for incident response (what did it do here?), for threat intel (what can this family do?), and for detection engineering (what rule catches the next sample?). The work splits into two complementary approaches — static (read it without running it) and dynamic (run it and watch) — and the professional skill is knowing which to use when, and how each defeats the other's blind spots.

2. Before you start

• Authorization and safety: this lesson analyzes only synthetic, benign training files you create yourself on this machine. Real malware analysis happens exclusively in an isolated lab: dedicated VMs (FLARE VM for Windows, REMnux for Linux), host-only networking with no route to the real internet, clean snapshots taken before anything runs. You never analyze a real sample on a machine you care about, and you never upload a sensitive sample to a public scanner by anything but its hash. Detonation walkthroughs are reader-side in your own lab — this lesson marks exactly where that boundary is.
• You can use a shell, compute hashes, and read a log (a-linux-cli).
• You know why the implant exists (d-redteam-concepts): persistence, C2, collection — analysis is how you learn which of those a given sample implements.
• You can read network evidence and log formats (c-log-sources) — dynamic analysis produces exactly those streams: DNS queries, connections, process telemetry.
• You can write small Python scripts (a-python-security) — the walkthrough's entropy scorer and YARA test harness are yours to build.
• Tooling: Python 3 (stdlib only), plus the yara-python package used in the walkthrough (installed in a venv here; the lesson gives the exact command).

3. How it works

Static vs dynamic: the two approaches and their blind spots

Static analysis reads the program without executing it: file type, hashes, readable strings, the executable's structure, which system functions it imports. It is fast and safe, and it is defeated by packing and obfuscation — the analyst sees the wrapper, not the payload. Dynamic analysis runs the program in isolation and watches: which files it touches, which registry keys it writes, which network connections it opens, which processes it injects into. It is truthful about behavior, and it is defeated by sandbox evasion — malware that sleeps, checks for VMs, or waits for user interaction before misbehaving. Professionals do static first (triage: what might this be?), dynamic second (behavior: what did it actually do?), and iterate: static tells you where to look, dynamic tells you what happened, and the gap between them tells you what the sample is hiding.

The triage workflow, in order

1. Hash first, upload never. Compute the SHA-256. A hash is a pivot, not just an ID: look it up (by hash) in threat-intel sources to see if anyone has analyzed this sample before. Never upload the file to a public service when the sample came from your organization — the upload itself can leak sensitive data. Hash lookups are free; file uploads are a disclosure decision.

2. Identify the file. Read magic bytes, not the extension — file on Linux, and the PE headers on Windows. The Portable Executable format (Windows) has a DOS header, NT headers, and a section table: .text holds code, .data holds data, .rsrc holds resources. What each section should contain versus what anomalies mean is the analyst's first read: an executable section with a suspicious name and very high entropy is the classic packed-binary tell.

3. Strings. Extract readable text: URLs, IP addresses, registry paths, error messages, PDB paths (PDB: the compiler's Program Database debug file — its path records the folder on the author's machine where the binary was built, which sometimes names the author). Few readable strings plus high entropy means packing or string encryption — itself a finding, and it tells you static analysis alone won't finish the job.

4. Imports. The Import Address Table lists the system functions the binary calls. A small, security-relevant import set — process, memory, network, registry, crypto APIs — sketches the binary's capabilities before a single instruction is read. This is the professional output of static analysis: a capability assessment (what this family can do), not just hashes and strings.

5. Entropy. Shannon entropy per section, on a 0–8 scale, measures randomness — 8 because a byte has 256 possible values and log₂(256) = 8: maximum entropy means every byte value is equally likely, which is what encrypted or compressed data looks like. Normal code sits in a middle band; packed or encrypted sections approach 8.0. It is a triage signal, never a verdict — legitimate installers and packed software also score high.

Packing: the concept

A packer compresses or encrypts the real payload; the on-disk binary is a small stub that unpacks the payload in memory at runtime. Static analysis sees the stub; the real code only exists in memory. The analyst's counters are conceptual: let it unpack itself (run to the end of the unpacking stub in a debugger, then dump the memory region) or use automated unpackers. The meta-reading matters too: packed means "the author didn't want static analysis to work" — which is itself intelligence about the sample's intent.

Dynamic analysis: what to watch

Run the sample in the isolated lab and collect three streams: API/process behavior (which system functions it calls, in what sequence — persistence writes followed by network connections tells a story), filesystem and registry changes (snapshot before and after: dropped files, Run keys, services), and network behavior (DNS queries, connections — captured against fake internet services, since the lab has no real route out). Then memory forensics: dumped memory shows injected code, unpacked payloads, and hidden processes that static analysis never sees. Watch for sandbox evasion — sleep timers, VM checks — and answer it with patience and a realistic lab, not cleverer tools.

Sandboxing: dynamic analysis, automated

A sandbox is dynamic analysis with the analyst taken out of the loop: the sample detonates in a fresh, instrumented virtual machine, and the sandbox emits a behavioral report — API calls in sequence, files created, registry keys written, mutexes, network connections attempted — without anyone watching it run. It automates exactly the three streams above, which makes it the triage-speed version of the manual lab: minutes per sample instead of hours. What it doesn't change: the sample still runs against fake internet (the sandbox answers DNS and HTTP itself, like INetSim), and sandbox evasion still applies — a sample that sleeps past the sandbox's time limit, or refuses to misbehave on a VM, produces a clean-looking report for a dirty sample. So the professional read of a sandbox report is the same as the manual read: behavior observed is fact, behavior absent is only as trustworthy as the detonation was faithful. Use the sandbox for scale and first-pass triage; use the manual lab when the report and the static read disagree.

From behavior to detection

Analysis produces behavioral indicators, which outlive any single sample: the persistence mechanism used, the C2 channel shape, the process-injection targets, distinctive mutex names (a mutex is a named lock programs use so only one copy runs at a time — malware often creates one with a distinctive name, which becomes a detection string) and registry keys, user-agent strings. Hashes die when the sample is recompiled; behaviors persist across the family. These become the detection content — Sigma rules, YARA rules, hunt hypotheses — which is the whole point: analysis feeding detection is the job's output, not the report sitting in a drawer.

YARA: classification as a discipline

YARA is the pattern-matching language for identifying and classifying files. A rule has an identifier, an optional meta section (descriptive only), a strings section (text, hex with wildcards/jumps/alternatives, regex, with modifiers like nocase, wide, ascii, xor, base64), and a required condition — boolean logic over the strings, plus file properties (filesize, uint16(0)==0x5A4D for MZ headers, PE-module attributes like imports and section entropy). The professional discipline has three rules: anchor on strings unique to the family, never generic API names; combine weak indicators (2 of ($a, $b, $c)) rather than trusting one strong one; and always test against a benign corpus before deploying — a rule that fires on legitimate software is worse than no rule.

Worked example: the triage sheet

A fictional training sample, invoice-reminder.exe, arrives from the phishing lesson's world. Triage: SHA-256 computed and looked up (unknown — proceed). file says PE32 executable. Strings show two URLs, a Run registry path, and the PDB path C:\Users\dev\Desktop\proj\Release\pay.pdb — the author built it as "pay", not "invoice-reminder". Imports include process-memory and network APIs — capability sketch: injection + C2. One section has entropy 7.9 with a nonstandard name — packed. Verdict on the sheet: packed dropper, likely injector, needs dynamic analysis in the lab; do not trust the strings. Every line on the sheet is a checkable fact, and the sheet itself is the artifact.

4. Walkthrough

Run by us on this machine — transcript: runs/d-malware-analysis.log. Clean environment: stock Python 3.12, yara-python in a venv, 2026-10-09. All samples below are synthetic benign files created for this exercise — no real malware is involved at any step.

Step 1 — Build the training set. Create three labeled text files: sample-a.txt (contains the marker strings of fictional family "GRAYWIDOW": gw-c2-channel, gw-persist-run), sample-b.txt (family "BLUEFALCON": bf-exfil-staging), benign.txt (a normal-looking config file that happens to contain the word channel):

mkdir -p ~/tmp-guidebook/ma && cd ~/tmp-guidebook/ma
printf 'config marker gw-c2-channel gw-persist-run\n' > sample-a.txt
printf 'config marker bf-exfil-staging\n' > sample-b.txt
printf 'app config: channel=stable\nlog_level=info\n' > benign.txt

Step 2 — Hash and identify. The triage opening, exactly as done for real:

sha256sum sample-a.txt sample-b.txt benign.txt | tee triage-hashes.txt
file sample-a.txt
strings sample-a.txt

Expected: three hashes recorded; file reports ASCII text; strings surfaces the markers.

Step 3 — Entropy. Write entropy.py (Shannon entropy over bytes) and score two synthetic files — random.bin (2KB of os.urandom) vs text.bin (2KB of English text):

python3 entropy.py

Expected: random.bin ≈ 8.0, text.bin ≈ 4–5. The number is the packed-vs-not triage signal from the lesson.

Step 4 — Write the YARA rule. Create graywidow.yar:

rule GRAYWIDOW_dropper {
    meta:
        author = "guidebook walkthrough"
        description = "Fictional family marker - training only"
    strings:
        $c2 = "gw-c2-channel"
        $persist = "gw-persist-run"
    condition:
        all of them
}

Compile and scan the training set with yara-python:

python3 scan.py

Expected: sample-a.txt matches; sample-b.txt and benign.txt do not.

Step 5 — Meet the false positive. Loosen the rule (change the condition to any of them and add the generic string "channel"), rescan, and watch benign.txt match:

python3 scan_loose.py

Expected: benign.txt now matches — the lesson's core warning, demonstrated: a generic string makes a bad rule.

Step 6 — Tune it back. Restore the tight rule, add a filesize < 100KB guard and the 2 of form, rescan the full set:

python3 scan_tuned.py

Expected: only sample-a.txt matches. The tuned rule plus the test output is the artifact.

Step 7 — Dynamic analysis (reader-side). Detonation, ProcMon/Wireshark/INetSim collection (INetSim: a fake-internet server for the lab — it answers the sample's DNS and HTTP requests so you can watch the network behavior with no real route out), and memory forensics happen in your isolated lab: FLARE VM (github.com/mandiant/flare-vm — install scripts for a Windows VM, verified 2026-10-09; the project states it must only be installed on a VM) for the Windows side, REMnux (remnux.org — Linux toolkit for reverse-engineering malware, verified 2026-10-09) for the Linux side and fake internet services. Lab setup here is not run by us — this machine is not an isolated lab. The exact procedure: two VMs, host-only networking, snapshot before detonation, training samples only (published training corpora), revert after.

5. On your own

Write a YARA rule for the fictional "BLUEFALCON" family (marker: bf-exfil-staging, plus invent two more family-specific markers and embed them in a new synthetic sample). Test it against a benign corpus of at least 10 files you create (configs, logs, source files). Success test: the rule fires on the family sample, fires on nothing else, and your test log shows both runs — a rule without a negative test is not finished.

6. Where it goes wrong

• Uploading the sample. The file upload to a public scanner is the classic beginner disclosure — hash lookups first, always.
• Generic strings. http, cmd, password appear in benign software by the megaton. The walkthrough's step 5 exists so you feel this before you deploy a rule.
• Entropy worship. A packed section is a signal to switch methods, not a verdict of malice. Legitimate software packs too.
• Static-only conclusions. "The strings show no URLs, so it doesn't do networking" — the sample was packed; the strings were encrypted. The gap between static and dynamic is where the truth lives.
• Lab escapes. Shared folders, NAT networking, "just this once" on the host — the isolation rules are not paranoia, they are the license to do this work at all.
• YARA version drift. Rule syntax is stable, but module features move; the docs page used here is the 4.4.0 stable line — name the version your rules target.

7. Read the source

Primary, in this order:

1. YARA documentation (yara.readthedocs.io — the "Writing YARA rules" page, stable line; read the strings section, then conditions, then the PE module) — the rule language from its authors.
2. FLARE VM repository (github.com/mandiant/flare-vm) — read the README's requirements and installation sections; note the VM-only rule and the snapshot guidance.
3. REMnux (remnux.org) — read what the toolkit is (a curated Linux toolkit for reverse-engineering malware) and the distro vs container options.

Free resources for going deeper (at most three):

• MalwareBazaar (bazaar.abuse.ch) — the free sample and threat-intel sharing platform; hash lookups and training corpora. For: the pivot in step 2 of every triage.
• Practical Malware Analysis (the book's free labs) — the classic training exercises, widely mirrored. For: structured practice beyond this lesson's synthetic samples.
• ATT&CK technique pages for observed behaviors (attack.mitre.org) — map what the sample did to technique IDs; the bridge from analysis to detection. For: turning the behavioral report into detection content.

8. Check yourself

1. A sample's .text section has entropy 7.9 and the strings output is nearly empty. What does each fact tell you, and what is your next method — and why that one?
2. Your YARA rule fires on a sample and on three benign installers. Name two distinct fixes at the rule level, and explain what each costs you in detection terms.
3. Why is "hash lookup first, file upload never (for org samples)" the correct order? What exactly does the upload risk that the lookup doesn't?
4. Dynamic analysis shows the sample writing a Run key and then opening an HTTPS connection. Static analysis showed neither. Give two different explanations for the gap, and say how you'd distinguish them.
5. A colleague proposes skipping the isolated lab because "we only do static analysis." Explain the two ways this still goes wrong.

9. Prove it

Artifact: a triage sheet for a training sample you craft (not the walkthrough's invoice-reminder.exe): invent the markers, pack or obfuscate it lightly, then triage it cold — hashes, file ID, strings of interest, import notes, entropy reading, packer verdict, capability sketch, next-method recommendation — plus one tuned YARA rule for your invented family with its positive and negative test log. The walkthrough taught the motions on labeled files; this artifact proves you can run them on a sample you didn't pre-solve.

You can honestly claim: you can triage a training sample statically, write a YARA rule with false-positive discipline, and describe the isolated-lab workflow for dynamic analysis. You cannot yet claim: you can perform dynamic analysis or handle real samples — that needs the FLARE VM/REMnux lab built and snapshotted, which this lesson specifies but does not build.

10. Leads to

• d-redteam-concepts — the other half of the implant: what it is for once it runs.
• c-sigma-detection — behavioral indicators become Sigma rules; YARA covers files, Sigma covers logs.
• Stage E — the governance of a malware-analysis capability (handling, retention, disclosure).
• Map: Stage D malware-analysis track (lab build → static triage → YARA → dynamic → IOCs → detection package); YARA also serves the DFIR track's file-classification needs — written once, referenced twice.