Chapter 31 — Digital forensics and malware analysis: implementation II
Lesson map
Lesson 31 published. Map, example, EX091.
flowchart TD
N0["Learning outcomes"]
N1["31.1 Entropy"]
N2["31.2 Triage use"]
N3["Lab — block entropy"]
N4["31.3 Reading the result"]
N5["31.4 Adding to EvidenceDesk"]
N6["31.5 False positives"]
N7["31.6 Block size"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX091"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["31.1 Entropy"]
N2["31.2 Triage use"]
N3["Lab — block entropy"]
N4["31.3 Reading the result"]
N5["31.4 Adding to EvidenceDesk"]
N6["31.5 False positives"]
N7["31.6 Block size"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX091"]
[Learning outcomes]
|
v
[31.1 Entropy]
|
v
[31.2 Triage use]
|
v
[Lab — block entropy]
|
v
[31.3 Reading the result]
|
v
[31.4 Adding to EvidenceDesk]
|
v
[31.5 False positives]
|
v
[31.6 Block size]
|
v
[Worked example] --> [Exercise EX091]
Learning outcomes
You will (1) compute Shannon entropy of data blocks; (2) interpret entropy in triage; (3) add an entropy flag to an intake report; (4) avoid over-interpreting high entropy.
31.1 Entropy
Shannon entropy measures how unpredictable bytes are, from 0 (all identical) to 8 bits per byte (uniformly random). Text sits in between; compressed and encrypted data approach 8.
For a block of bytes, entropy is H = −Σ p(b) log₂ p(b) summed over the 256 possible byte values, where p(b) is the share of the block equal to b. If only one byte value appears, H is zero. If all 256 values appear equally often, H is eight. English text uses a limited set of characters unevenly, which keeps it around four to five bits per byte.
31.2 Triage use
High entropy across a disk region may indicate encryption, compression or packed executables. Malware analysts note that packed samples often show high entropy in sections, as discussed by Sikorski and Honig (2012). Engineers use entropy as a flag for examiners, not a conclusion.
On a disk, a large contiguous high-entropy region inside an otherwise ordinary file system might be an encrypted container or a large archive. In an executable, one high-entropy region among otherwise normal code and data suggests packed code that unpacks at run time. Either case tells examiners where to look next, not what they will find there.
Lab — block entropy
"""Shannon entropy of synthetic blocks (bits per byte)."""
import math, random
def H(b):
c = [0]*256
for x in b: c[x] += 1
return -sum(n/len(b)*math.log2(n/len(b)) for n in c if n)
random.seed(7)
blocks = {"text": b"the quick brown fox " * 205, "zeros": bytes(4096),
"random": bytes(random.randrange(256) for _ in range(4096))}
for k, v in blocks.items():
print(f"{k:7} {abs(H(v)):.2f}", "HIGH" if H(v) > 7.5 else "")
Run with python3 lab_entropy.py. Verified output:
text 3.82
zeros 0.00
random 7.96 HIGH
31.3 Reading the result
Repeated English text scores 3.82 bits per byte; a block of zeros scores 0.00; pseudo-random bytes score 7.96 and are flagged HIGH above 7.5. The flag says the block is unpredictable — consistent with encryption or compression — and nothing more.
Repeated text scores below normal English because the repetition reduces variety. The pseudo-random block scores just under eight because a finite sample of 4 KiB rarely hits a perfectly even distribution. The 7.5 threshold is a heuristic chosen to separate encrypted and compressed data from most other content. Priya records it as a setting, not as a fact about data.
31.4 Adding to EvidenceDesk
Priya adds an optional summary: the proportion of blocks above 7.5 in an image. An image that is almost entirely high entropy suggests full-disk encryption, prompting examiners to check for lawful access routes early.
The scan reads the image as raw bytes and reports only the proportion of blocks above the threshold, which keeps EvidenceDesk out of interpreting content. An image with 98 percent high-entropy blocks is most consistent with full-disk encryption, though a drive filled with archives or video could look similar. Flagging it at intake lets the case officer ask about recovery keys before the examiner's time is booked.
31.5 False positives
JPEG, ZIP and video files are high entropy by design. The flag must be read with file types.
A photo library or a folder of video files can make large parts of a disk look encrypted. Reading the flag alongside file-system information, such as which files occupy the high-entropy regions, separates expected compression from unexplained regions. Only high-entropy space that no known file accounts for is worth raising with the examiner. The report lists the largest unexplained regions by offset and size.
31.6 Block size
Small blocks give noisy estimates; Priya uses 4 KiB.
With very small blocks, such as 256 bytes, the sample is too small for byte frequencies to settle, so random data can score well below eight and text can look more varied than it is. Large blocks smooth over boundaries between different kinds of content. A 4 KiB block matches common file-system cluster sizes, which keeps each block within one allocation unit. Priya tests 512 bytes, 4 KiB and 64 KiB on the synthetic set and records how often each misclassifies text blocks.
31.7 Safety
The entropy scan reads bytes; it never executes content.
Reading bytes to count their values cannot trigger code in them, so the scan is safe to run on images that contain malware. The scan also opens images read-only and writes its summary to a separate output folder. Those properties are tested in the same way as the verifier's read-only discipline, by checking input metadata before and after a run. No preview or file-type handler is ever invoked. Output never includes raw evidence bytes.
Visualising entropy
A simple strip chart of block entropy across an image shows regions at a glance: low-entropy system areas, high-entropy media folders, uniform high entropy where a container sits. Priya prints it as text bands in the summary so it works offline, and she labels it as a triage aid.
Worked example
An intake summary reports 97% of blocks above 7.5. The examiner checks and finds a full-disk encryption header, records the encryption state, and requests the organisation's recovery key through the proper legal route.
Synthetic case. The image is invented.
Errors and limits
Entropy cannot distinguish encryption from compression; structure and headers must be examined.
Exercises
- EX091 — Lab values. Report the three entropy values and the flag. Rubric: 3 = 3.82, 0.00, 7.96 HIGH; 2 = two; 1 = one; 0 = none.
- EX092 — Flag meaning. What does a HIGH flag mean? Rubric: 3 = unpredictable data, consistent with encryption/compression, not conclusion; 2 = partial; 1 = malware; 0 = none.
- EX093 — 97% response. What did the examiner do on seeing 97% high-entropy blocks? Rubric: 3 = checked header, recorded encryption, sought key lawfully; 2 = partial; 1 = attempted bypass; 0 = none.
Synthesis
Entropy is a cheap triage flag that guides examiners when read with file types and structure.
Continue studying — chapter 31 of 50
← Chapter 30 (draft) · Course chapter index · Chapter 32 (draft) →