Chapter 8 — Independent review and defense | Master's Security operations and blue team

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 08 published. Map, example, EX045.

flowchart TD
  N0["Learning outcomes"]
  N1["8.1 Independent review"]
  N2["8.2 The defense format"]
  N3["8.3 Likely questions"]
  N4["8.4 Answering well"]
  N5["8.5 Returning results"]
  N6["8.6 Sharing safely"]
  N7["8.7 Afterwards"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX045"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["8.1 Independent review"]
  N2["8.2 The defense format"]
  N3["8.3 Likely questions"]
  N4["8.4 Answering well"]
  N5["8.5 Returning results"]
  N6["8.6 Sharing safely"]
  N7["8.7 Afterwards"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX045"]

[Learning outcomes]
    |
    v
[8.1 Independent review]
    |
    v
[8.2 The defense format]
    |
    v
[8.3 Likely questions]
    |
    v
[8.4 Answering well]
    |
    v
[8.5 Returning results]
    |
    v
[8.6 Sharing safely]
    |
    v
[8.7 Afterwards]
    |
    v
[Worked example] --> [Exercise EX045]

Learning outcomes

You will (1) arrange independent review before submission; (2) prepare an oral defense; (3) respond to examiners' questions; (4) hand results back to the host organisation responsibly.

8.1 Independent review

Before submission, a SOC analyst from another organisation reviews the method and recommendations, and a fellow student reruns the analysis scripts from the archive. The analyst's main comment: "Your allow-list review process assumes a mature change process; smaller SOCs may not have one." Tariq adds a paragraph scaling the recommendations.

The two reviews test different things. The fellow student's rerun checks that the scripts produce the reported figures from the archive. The external analyst checks whether the recommendations make sense outside Pelham. Her comment led to a concrete change, which is recorded in the change log alongside the AL-07 narrowing.

8.2 The defense format

A 20-minute presentation and 40 minutes of questions from two examiners.

Tariq's twenty minutes follow the dissertation: question, design, the three results together, the AL-07 case and the limits. He keeps the bootstrap interval on the same slide as the 136.5 estimate, so the examiners see the uncertainty with the number. The forty minutes of questions are where most of the assessment happens, so he prepares answers for the questions he expects most. The two examiners have read the restricted appendix with host and account names.

8.3 Likely questions

"Why not randomise?" "How do you know the holiday affected both groups equally?" "Is one suppressed event in eight a lot or a little?" "Would you deploy these allow-lists at another utility?"

Each question targets a design choice. Randomisation was rejected because the SOC wanted its noisiest rules fixed first. The holiday is handled by the control group. The one-in-eight question tests whether Tariq can state a result's meaning without overstating it. The last question tests whether he understands what in his work transfers and what is specific to Pelham.

8.4 Answering well

Tariq answers directly, gives the evidence, then the limit. On "one in eight": "In this exercise it was 12.5% of injected behaviours, but the events were few and chosen; the important finding is that a broad entry could hide a realistic misuse, and narrowing it fixed that case."

His answer on deployment elsewhere follows the same pattern: not the allow-lists as they are, because they encode Pelham's hosts and accounts, but the process of conditional entries, peer review and quarterly replay against authorised simulations. Answering this way shows the examiners that he can separate Pelham-specific artefacts from the transferable method.

8.5 Returning results

Pelham receives a summary report, the narrowed allow-list policy and the replay script. Tariq briefs analysts first and managers the following week, with the same summary report, so the people who did the work hear results first.

The replay script lets Pelham repeat the AL-07 test on every entry each quarter without Tariq. Briefing analysts first is also practical: they maintain the allow-lists day to day and are the people who will notice whether the narrowed entries still remove routine noise.

8.6 Sharing safely

The public dissertation omits host names, account names and rule logic details that would help an attacker evade Pelham's detections; these remain in a restricted appendix for the examiners.

The rule is the same one Tariq applied to his own data: share the method, not the means of evasion. A public reader learns that a broad host-account entry hid a misuse of a patching account and how narrowing fixed it, without learning which host, which account or which rule logic would let an attacker repeat it at Pelham.

8.7 Afterwards

Examiner corrections are tracked and completed; the archive is tagged with the final version.

The final tag ties the archive to the version the examiners approved, so the scripts, counts and change log anyone downloads later match the dissertation's figures exactly. Pelham receives the same tagged version of the replay script. Corrections are listed in a short table with the examiner's comment, the change made and the page, which goes into the archive with the final tag. If a correction changed any figure, the scripts are rerun and the new output archived beside the old.

Worked example

Asked "Would you deploy these allow-lists elsewhere?", Tariq replies: "Not as they are — they encode Pelham's hosts and accounts. I would deploy the process: conditional entries, peer review and quarterly replay against authorised simulations."

Synthetic case. The review and defense are invented.

Errors and limits

Restricting details protects the host but limits reproducibility; the archive explains what was withheld and why.

Exercises

EX045 — External comment. What did the external analyst point out, and what changed?
Show marking criteria
Rubric: 3 = assumes mature change process; scaling paragraph added; 2 = partial; 1 = vague; 0 = none.
EX046 — Defense questions. List three likely examiner questions.
Show marking criteria
Rubric: 3 = randomise, holiday equality, one in eight, deploy elsewhere (three); 2 = two; 1 = one; 0 = none.
EX047 — One in eight. How did Tariq frame the one-in-eight result?
Show marking criteria
Rubric: 3 = 12.5% of few chosen events; key point broad entry hid realistic misuse; fixed by narrowing; 2 = partial; 1 = percentage only; 0 = none.
EX048 — Returning results. What did Pelham receive, and who heard first?
Show marking criteria
Rubric: 3 = report, policy, replay script; analysts before managers; 2 = partial; 1 = vague; 0 = none.
EX049 — Restricted appendix. What was withheld from the public version, and why?
Show marking criteria
Rubric: 3 = host/account names, rule details; would aid evasion; 2 = partial; 1 = vague; 0 = none.
EX050 — Deploy elsewhere. Why not reuse the allow-lists at another utility?
Show marking criteria
Rubric: 3 = encode Pelham-specific hosts/accounts; transfer the process instead; 2 = partial; 1 = would reuse; 0 = none.

Synthesis

Independent review, direct answers with limits, and responsible return and sharing complete a master's project.

Continue studying — chapter 08 of 08

← Chapter 07 (draft) · Course chapter index · Worked results ↓ · Next course via chapter index →