◇ OpenCyber Academy
Technologist · Chapter 28 · English

Chapter 28 — Measuring the result

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 28 — Measuring the result

Lesson map

Lesson 28 published. Map, example, EX082.

flowchart TD
  N0["Learning outcomes"]
  N1["28.1 Results"]
  N2["28.2 Fairness"]
  N3["28.3 Causes"]
  N4["28.4 Remaining gap"]
  N5["Reading the numbers against targets"]
  N6["What did not improve, and what staff said"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX082"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["28.1 Results"]
  N2["28.2 Fairness"]
  N3["28.3 Causes"]
  N4["28.4 Remaining gap"]
  N5["Reading the numbers against targets"]
  N6["What did not improve, and what staff said"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX082"]

[Learning outcomes]
    |
    v
[28.1 Results]
    |
    v
[28.2 Fairness]
    |
    v
[28.3 Causes]
    |
    v
[28.4 Remaining gap]
    |
    v
[Reading the numbers against targets]
    |
    v
[What did not improve, and what staff said]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX082]

Learning outcomes

You will (1) repeat measurements; (2) compare fairly; (3) discuss causes.

28.1 Results

Stale accounts 4 to 0, SSH compliance 2 to 4 of 4, baseline 31 to 93 of 94, encryption 87 to 94 of 94, backups on time 4 to 6 of 6.

These are the same five measures that were recorded at the start, now taken again at the end of term. Each result is written as a count out of the same total used in the baseline, so the before and after numbers can be compared directly. The comparison table lists all five rows with both dates. Every target set in the plan was met, with laptops on the baseline at 93 against a target of at least 90.

28.2 Fairness

The same scripts and sources are used.

The same scripts read the same kinds of export, with the same thresholds, such as ninety days for stale accounts and one day for backup age. The five laptops offline at the start were treated as unknown and checked on return, so neither measurement counts an unmeasured laptop as compliant. Any change to a script after the baseline would have broken this comparison, so none was made. The method notes written at the start are what made it possible to repeat each step exactly.

28.3 Causes

Each improvement links to a specific change, which is easier here than in larger studies because changes were separate.

Stale accounts fell after the disabling and the leavers form; SSH compliance rose after the Saturday changes; encryption reached all laptops after the seven individual fixes. Because each change touched one measure, a reader can follow each number to an entry in the change log rather than relying on Tomás's judgement. The laptop baseline is the one measure where two things changed together, the settings and the whiteboard app update, but both belonged to the same planned rollout.

28.4 Remaining gap

One laptop missed the baseline due to a failed agent, since fixed.

The one laptop below the baseline had a failed management agent, which is why it could not receive settings; reinstalling the agent brought it into line. Tomás reports the gap at the time of measurement and the later fix separately, so the result is not quietly rounded up to 94 of 94. The gap is small, but naming it shows readers that the table reports what was measured rather than what was hoped. The agent failure also suggests adding last check-in dates to the weekly routine, which the routine handed over in the next chapter already does.

Reading the numbers against targets

The improvements are large, but Tomás is careful how he describes them. The measurements cover one term at one trust, and some values, such as encryption, were already high. Laptop compliance depends on laptops being online, so the five offline laptops were checked when they returned. The plan had set targets of zero stale accounts, all servers SSH-compliant and at least 90 of 94 laptops on the baseline. Every target was met, with laptops at 93. He writes that the project achieved its targets for this term and that keeping them depends on the routines in the next chapter.

What did not improve, and what staff said

Not everything moved. The classroom tablets stayed outside the project, and the outside company's shared VPN account was replaced only in the final week, so its effect cannot yet be measured. Reporting these openly shows the limits of the work. Dev's time on the project was about twenty-five hours, more than planned, mainly on the laptop pilot. A short question to staff about disruption gets mostly neutral answers, with a few comments about the screen lock. This suggests the changes were acceptable, though the sample is small, and Tomás says so rather than claiming staff approval.

Worked example

The comparison table is attached to the report.

Synthetic case. Figures are synthetic.

Errors and limits

Improvements may fade without routines; chapter 29 addresses this.

Exercises

Synthesis

Repeating the method exactly allows confident comparison.

Continue studying — chapter 28 of 30

← Chapter 27 (draft) · Course chapter index · Chapter 29 (draft) →