OpenCyber Academy
Bachelor's · Chapter 37 · English
Course code: OCA-104Chapter 37: OCA-104-C037

Chapter 37 — Evidence review seminar

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 37 published. Map, example, EX109.

flowchart TD
  N0["Learning outcomes"]
  N1["37.1 Seminar"]
  N2["37.2 Aisha's evidence"]
  N3["37.3 Challenge"]
  N4["37.4 Improvement"]
  N5["37.5 Lesson"]
  N6["Limitations"]
  N7["Generalising carefully"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX109"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["37.1 Seminar"]
  N2["37.2 Aisha's evidence"]
  N3["37.3 Challenge"]
  N4["37.4 Improvement"]
  N5["37.5 Lesson"]
  N6["Limitations"]
  N7["Generalising carefully"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX109"]

[Learning outcomes]
    |
    v
[37.1 Seminar]
    |
    v
[37.2 Aisha's evidence]
    |
    v
[37.3 Challenge]
    |
    v
[37.4 Improvement]
    |
    v
[37.5 Lesson]
    |
    v
[Limitations]
    |
    v
[Generalising carefully]
    |
    v
[Worked example] --> [Exercise EX109]

Learning outcomes

You will (1) review evidence critically; (2) challenge claims; (3) improve evidence quality.

37.1 Seminar

Students present evidence for their projects; peers challenge.

Each student has a short slot to present the evidence behind one claim, and peers then question how it was collected. The seminar rewards students who find the weaknesses in their own evidence before examiners do. Aisha chooses her encryption claim, 45 of 45 verified, because the whole project rests on it and it depends heavily on one data source. Aisha uses the peers' questions to list weaknesses she then addresses in her limitations chapter.

37.2 Aisha's evidence

Agent reports, logs, lab outputs and staff feedback.

Aisha's evidence comes from four kinds of source: management server agent reports, server and sign-in logs, lab outputs rerun on synthetic data, and staff feedback from the survey and desk time. Each item of evidence is dated and named by its source in her evidence folder. She shows which claims rest on one source only, because those are the claims peers are most likely to challenge. Staff feedback is the weakest source, because it is self-selected and comes from few responses.

37.3 Challenge

A peer asks whether agent reports can be wrong. Aisha spot-checks five laptops by hand; all match.

The spot check uses the same method as the encryption verification: Gareth opens the encryption status on five laptops chosen across teams and compares it with the agent report. All five laptops show the same encryption state as their agent reports. Aisha is careful to say that five matching laptops make an error less likely but do not prove that all 45 reports are correct. A peer then asks whether five is enough; Aisha answers that it is a check on the report, not a sample for estimating a rate.

37.4 Improvement

She adds spot checks to the routine.

The routine now includes a monthly spot check of five laptops, with different laptops chosen each month so the whole fleet is covered over time. The result of each spot check is recorded beside that month's encryption verification run. If a spot check disagrees with a report, the laptop is investigated and the report treated as doubtful until the cause is found. The monthly spot check takes Gareth about ten minutes and is recorded on the routine checklist.

37.5 Lesson

Evidence from one source should be cross-checked.

Aisha applies the lesson to her other measures and to staff feedback, which she compares with help-request counts: patch figures are compared with laptops' own update history on the spot-checked machines, and the inventory comparison is checked against the asset register. Non-reporting devices are counted separately in every table, because a silent device can make one source look complete when it is not. Claims resting on one source are worded more cautiously in the dissertation than claims supported by two.

Limitations

The project has clear limits. It covers laptops and servers but not phones or the website. Data come from one organisation over a few months, so results may not generalise. The reminder study had no control group. Some measures depend on the management server's reports, which can miss devices that do not check in.

Aisha states these limits plainly in her dissertation. Dr Grant explains that examiners respect honest limits more than inflated claims, and that limits also point to future work.

She proposes three follow-ups: extend management to phones, run the reminder comparison for a full year, and test recovery for the finance server again after its replacement.

Generalising carefully

Other small housing associations may find the approach useful, but should adapt targets and tools to their size and risk.

Threats to validity

Seasonal patterns and staff changes may affect patch results.

Reflection

Aisha notes that she underestimated how long checking with staff would take.

Project log entry

In the evidence review seminar, peers challenge each other's claims. A classmate asks whether Bramley's encryption rate would have risen anyway after the theft. Aisha agrees that the theft created urgency and says verification, not motivation, is what her lab measures.

Review with the tutor

Dr Grant asks which limitation worries her most. She names non-reporting devices, because they can make results look better than reality, and explains how each table counts them separately.

Worked example

Another peer asks about staff feedback bias; Aisha notes it is self-selected.

Synthetic case. Synthetic.

Errors and limits

Unchecked evidence can be confidently wrong.

Exercises

Synthesis

Cross-check evidence sources.

Continue studying — chapter 37 of 40

← Chapter 36 (draft) · Course chapter index · Chapter 38 (draft) →

Worked answers

EX109. Whether agent reports can be wrong.

EX110. Five laptops checked by hand; all matched.

EX111. It is self-selected.