OpenCyber Academy
Bachelor's · Chapter 34 · English
Course code: OCA-109Chapter 34: OCA-109-C034

Chapter 34 — Comparative project

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 34 published. Map, example, EX100.

flowchart TD
  N0["Learning outcomes"]
  N1["34.1 Running the tools"]
  N2["Lab — comparing tool outputs"]
  N3["34.2 Reading the result"]
  N4["34.3 Investigating differences"]
  N5["34.4 Reporting"]
  N6["34.5 Practical advice"]
  N7["34.6 Sharing with developers"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX100"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["34.1 Running the tools"]
  N2["Lab — comparing tool outputs"]
  N3["34.2 Reading the result"]
  N4["34.3 Investigating differences"]
  N5["34.4 Reporting"]
  N6["34.5 Practical advice"]
  N7["34.6 Sharing with developers"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX100"]

[Learning outcomes]
    |
    v
[34.1 Running the tools]
    |
    v
[Lab — comparing tool outputs]
    |
    v
[34.2 Reading the result]
    |
    v
[34.3 Investigating differences]
    |
    v
[34.4 Reporting]
    |
    v
[34.5 Practical advice]
    |
    v
[34.6 Sharing with developers]
    |
    v
[Worked example] --> [Exercise EX100]

Learning outcomes

You will (1) execute the comparative study; (2) compare tool outputs with set operations; (3) investigate each difference; (4) report fairly.

34.1 Running the tools

Both tools process the synthetic image on the same workstation. Outputs are normalised to a common list of artefact identifiers by a script.

Normalising matters because each tool names artefacts differently; one might call an item "Run key: Updater" and another "HKCU Run". The script maps both to the identifier run-key, so comparison is exact. The mapping is part of the archive, so a reviewer can check that no item was mislabelled. Running both tools on the same workstation, one after the other, removes hardware as a source of difference.

Lab — comparing tool outputs

"""Compare artefacts reported by two synthetic tools."""
a = {"Q1_budget.xlsm", "update.dll", "notes.txt", "run-key", "prefetch:EXCEL"}
b = {"Q1_budget.xlsm", "update.dll", "run-key", "prefetch:EXCEL", "prefetch:RUNDLL32"}
print("both:", len(a & b))
print("only tool A:", sorted(a - b))
print("only tool B:", sorted(b - a))

Run with python3 lab_compare.py. Verified output:

both: 4
only tool A: ['notes.txt']
only tool B: ['prefetch:RUNDLL32']

34.2 Reading the result

Four artefacts are reported by both tools. Tool A alone reports notes.txt; tool B alone reports a prefetch entry for RUNDLL32. Neither difference is a verdict yet: each must be checked against ground truth and against the raw image.

The lab's set operations show the comparison directly: the intersection gives the shared items and the two differences give each tool's unique reports. Python's set operators make this a few lines of code, and the output can be checked by eye because the sets are small. In a larger study the same code would scale to hundreds of items, while the investigation of each difference would remain manual.

34.3 Investigating differences

notes.txt is in ground truth as a planted user file; tool B's default profile excluded text files from its "user activity" view. The RUNDLL32 prefetch entry is also planted; tool A's prefetch parser skipped it because of a format variation. Both tools therefore missed one planted artefact each, for different reasons.

Without ground truth, the two differences might have looked like false reports by each tool. Checking against the planted list showed both were real artefacts. The causes are different in kind: one is a default setting the user can change, the other a parser limitation that needs a fix by the developers.

34.4 Reporting

Completeness: tool A 5 of 6 relevant planted artefacts, tool B 5 of 6. False reports: none for either. Leah describes each miss and its cause, rather than ranking tools.

Equal scores hide very different problems, which is why Leah describes each miss rather than ranking. An examiner who knows that tool B hides text files by default can change the view; one who knows tool A skips a prefetch variant can check prefetch with another tool. Ranking would have given less useful information than the descriptions.

34.5 Practical advice

Examiners using either tool should know these blind spots; using two tools, as in this study, caught both.

The lesson for casework is concrete: when an artefact type matters to a conclusion, confirm it with a second tool or a raw check. Leah's own verification of the run key in the ws-114 case followed this practice. Knowing a tool's blind spots in advance is better than discovering them during a case under time pressure. Leah adds both blind spots to the lab's tool notes.

34.6 Sharing with developers

With Dr Ferreira's approval, she reports the prefetch parsing issue to the tool's maintainers with a synthetic test file.

A synthetic test file lets the maintainers reproduce the problem without any real data. Leah describes the format variation, the expected output and the actual output. Reporting helps every user of the tool, and if a fix is released, she can rerun her test to confirm it. The report and any reply are kept in her project archive with their dates.

34.7 Reproducibility

The image, ground truth, scripts and outputs are archived.

Archiving everything lets another student, or Leah herself later, rerun the comparison and get the same results. The manifest lists each file with its hash. If a tool is updated, the archive allows a clean comparison between old and new versions on the same image, which would show whether the prefetch issue was fixed. The archive is stored with the course's project records and listed in her thesis appendix, so it can be found after she graduates.

Time and usability

Tool A took 14 minutes, tool B 22, on the same hardware. Leah notes that time differences matter for triage but not for completeness, and records clarity issues separately.

Worked example

Her conclusion: "On this synthetic image, each tool missed one planted artefact — tool A a prefetch variant, tool B a text file excluded by its default view. Using both tools, or adjusting tool B's view, achieved full coverage."

Synthetic case. Outputs come from the lab script.

Errors and limits

Six artefacts are too few to estimate general completeness; the value is in the specific blind spots found.

Exercises

Synthesis

Compare outputs item by item, explain every difference and report blind spots, not rankings.

Continue studying — chapter 34 of 40

← Chapter 33 (draft) · Course chapter index · Chapter 35 (draft) →

Worked answers

EX100. Four artefacts reported by both; notes.txt only by tool A; prefetch:RUNDLL32 only by tool B.

EX101. Tool B's default view excluded text files, and tool A's prefetch parser skipped an entry with a format variation.

EX102. Both found five of six with no false reports, the sample is tiny, and the useful result is the specific blind spot each has.