Chapter 37 — Evidence review seminar
Lesson map
Lesson 37 published. Map, example, EX109.
flowchart TD
N0["Learning outcomes"]
N1["37.1 The seminar"]
N2["37.2 Lucas's evidence"]
N3["37.3 Peer critique"]
N4["37.4 Weak inferences"]
N5["37.5 Missing data"]
N6["37.6 Responding"]
N7["37.7 Critiquing others"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX109"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["37.1 The seminar"]
N2["37.2 Lucas's evidence"]
N3["37.3 Peer critique"]
N4["37.4 Weak inferences"]
N5["37.5 Missing data"]
N6["37.6 Responding"]
N7["37.7 Critiquing others"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX109"]
[Learning outcomes]
|
v
[37.1 The seminar]
|
v
[37.2 Lucas's evidence]
|
v
[37.3 Peer critique]
|
v
[37.4 Weak inferences]
|
v
[37.5 Missing data]
|
v
[37.6 Responding]
|
v
[37.7 Critiquing others]
|
v
[Worked example] --> [Exercise EX109]
Learning outcomes
You will (1) present evidence from projects for peer review; (2) critique others' evidence constructively; (3) identify weak inferences and missing data; (4) improve evidence before final reports.
37.1 The seminar
Final-year students present their capstone evidence so far. Each presentation shows claims, the evidence behind them and open questions.
Students present to classmates, Dr. Kim and an academic supervisor. Questions focus on whether claims match evidence, not on how much work was done. Presenting before the final evaluation means criticism can still change the project, which is more useful than receiving it in the final examination. Each presenter receives written comments from two assigned peers in addition to spoken questions, so quieter classmates contribute too and the presenter has a record to work from afterwards.
37.2 Lucas's evidence
He shows before-scenario results (11 of 20 detected), monitor-only false alerts (four in two weeks) and the design of the mass-mailing detection.
The before result shows that existing rules caught just over half the scenarios. Four false alerts in two weeks is a rate of two a week, under the three-per-week requirement, but over a short period. The mass-mailing design was presented as a plan, since it had not yet been built. He presented each figure with its source in the project log, so peers could see which numbers were measured and which were still plans.
37.3 Peer critique
A classmate asks whether the 20 scenarios include slow attackers like s3310. Another asks whether four false alerts in two weeks meets the "three per week" requirement or whether two weeks is too short to judge.
Both questions were fair. The scenarios had been based mainly on s2047's fast behaviour, so slow attackers were under-represented. Two weeks of monitor-only data gives only a rough sense of the false-alert rate. Lucas had not considered either point carefully until classmates raised them.
37.4 Weak inferences
Dr. Kim notes Lucas's slide says "the package will detect most compromises"; the evidence supports only "detects most test scenarios".
The difference matters because test scenarios are designed by defenders and may be easier than real attacks. Claiming the package will detect most compromises would invite readers to rely on it more than the evidence warrants. Precise wording protects both the reader and the project's credibility. Lucas rewrote the slide during the seminar break, and Dr. Kim asked him to apply the same check to every claim in his draft report before submitting it.
37.5 Missing data
Lucas has not yet recorded how long each detection took; the 30-minute requirement cannot be assessed.
Detection time had been in the requirements from the start but was not being captured, because the test harness recorded only whether each scenario alerted. Lucas changed the harness to log the time of the triggering action and the alert, so the before and after runs would both have times. He also realised that the before run had to be repeated to collect times, since comparing times from only the after run would have nothing to compare against.
37.6 Responding
Lucas adds two slow-attacker scenarios (replacing two duplicates), starts recording detection times, and changes the wording.
Replacing duplicates rather than adding scenarios kept the total at twenty, so the before and after runs remain comparable. The before run was repeated with the new scenarios to keep the comparison fair. Each change was recorded in the project log with the seminar as its source. The wording change was applied across his draft report as well as the slides. He sent a short note to both classmates thanking them and listing what had changed, which also documented the response for his supervisor.
37.7 Critiquing others
He questions a classmate's claim that a new dashboard "reduced workload" when the only evidence was analysts' impressions; he suggests counting alerts handled per shift.
Lucas framed his comment as a suggestion with a method attached, not as a dismissal. The classmate agreed that impressions might reflect novelty rather than real change, and added a count of alerts handled per shift before and after the dashboard. Good critique gives the presenter something to do. Lucas also noted something the classmate had done well: the dashboard's design was based on interviews with analysts.
Preparing for critique
Before the seminar, each student writes the three questions they most fear being asked and brings the evidence they have for each. Lucas's list included false-alert rates, which turned out to be exactly what a classmate asked about. Preparing this way makes critique a working session rather than a defence.
Recording
Notes from the seminar are added to the project log.
Worked example
Lucas's revised claim: "In authorised test scenarios, the existing rules detected 11 of 20; after the package, we expect to report detection with a confidence interval and detection times for each scenario." The wording now matches what the evidence can show.
Synthetic case. The seminar is invented.
Errors and limits
Peer review improves evidence but cannot replace testing.
Exercises
- EX109 — Peer questions. What two peer questions challenged Lucas's evidence? Rubric: 3 = slow attackers in scenarios; two weeks too short for false-alert rate; 2 = one; 1 = vague; 0 = none.
- EX110 — Weak claim. What overclaim did Dr. Kim identify? Rubric: 3 = "will detect most compromises" vs "detects most test scenarios"; 2 = partial; 1 = vague; 0 = none.
- EX111 — Changes made. What did Lucas change after the seminar? Rubric: 3 = added slow scenarios, recorded detection times, reworded; 2 = two; 1 = one; 0 = none.
Synthesis
Evidence review seminars sharpen claims, expose gaps and improve projects before final reporting.
Continue studying — chapter 37 of 40
← Chapter 36 (draft) · Course chapter index · Chapter 38 (draft) →