Chapter 40 — Verification laboratory II
Lesson map
Lesson 40 published. Map, example, EX118.
flowchart TD
N0["Learning outcomes"]
N1["40.1 Purpose"]
N2["40.2 Method"]
N3["Lab — timing five runs"]
N4["40.3 Reading the result"]
N5["40.4 Stage analysis"]
N6["40.5 Fixes"]
N7["40.6 Re-verification"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX118"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["40.1 Purpose"]
N2["40.2 Method"]
N3["Lab — timing five runs"]
N4["40.3 Reading the result"]
N5["40.4 Stage analysis"]
N6["40.5 Fixes"]
N7["40.6 Re-verification"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX118"]
[Learning outcomes]
|
v
[40.1 Purpose]
|
v
[40.2 Method]
|
v
[Lab — timing five runs]
|
v
[40.3 Reading the result]
|
v
[40.4 Stage analysis]
|
v
[40.5 Fixes]
|
v
[40.6 Re-verification]
|
v
[Worked example] --> [Exercise EX118]
Learning outcomes
You will (1) verify an end-to-end requirement under varied conditions; (2) analyse which stage causes failures; (3) report pass or fail with evidence; (4) propose fixes and re-verify.
40.1 Purpose
Laboratory II verifies requirement R-07 — possible-takeover alert acknowledged within 15 minutes — not just in quiet conditions but during busy periods, at night and at shift handover.
Quiet-condition tests from the first laboratory showed the rule works; they did not show the whole response chain meets its time target when conditions are difficult. Busy periods slow pipelines and distract analysts; nights have fewer staff; handovers split attention. Testing the requirement under those conditions shows whether it holds when it matters most. Each condition gets its own run.
40.2 Method
Five timed synthetic injections run in production under a test customer account, with analysts aware a test may happen but not when. Times are captured at ingestion, detection and acknowledgement.
Running in production is necessary because the laboratory cannot reproduce real load or real analyst behaviour. A reserved test account keeps the injections away from real customers. Telling analysts a test may happen, without saying when, respects them while keeping their behaviour natural. Timestamps at each stage let the team see where any delay occurred.
Lab — timing five runs
"""End-to-end verification of requirement R-07 from five timed synthetic injections (seconds after injection)."""
runs = { # run: (siem_received, rule_fired, analyst_ack)
"R1 quiet": (95, 310, 690),
"R2 quiet": (102, 298, 610),
"R3 busy": (240, 520, 1180),
"R4 night": (98, 305, 820),
"R5 handover": (110, 330, 1010),
}
limit = 15 * 60
for name, (s, f, a) in runs.items():
print(f"{name:12} ingest={s:4}s detect={f-s:4}s ack={a-f:4}s total={a/60:5.1f} min {'PASS' if a <= limit else 'FAIL'}")
Run with python3 lab_e2e.py. Verified output:
R1 quiet ingest= 95s detect= 215s ack= 380s total= 11.5 min PASS
R2 quiet ingest= 102s detect= 196s ack= 312s total= 10.2 min PASS
R3 busy ingest= 240s detect= 280s ack= 660s total= 19.7 min FAIL
R4 night ingest= 98s detect= 207s ack= 515s total= 13.7 min PASS
R5 handover ingest= 110s detect= 220s ack= 680s total= 16.8 min FAIL
40.3 Reading the result
Three runs pass (11.5, 10.2 and 13.7 minutes). The busy-period run fails at 19.7 minutes: ingestion slowed to 240 seconds and acknowledgement took 660 seconds. The handover run fails at 16.8 minutes, almost entirely from slow acknowledgement.
The total for the busy run, 19.7 minutes, breaks down into four minutes of ingestion, about five minutes of detection and eleven of acknowledgement. Both slow stages exceeded their budgets. The handover failure is a single-stage problem: the pipeline was fine, but nobody picked up the alert quickly while shifts changed over.
40.4 Stage analysis
Detection time stays near 200–280 seconds in all runs. The failures come from ingestion under load and from analyst acknowledgement under load or handover.
Stable detection times mean the rule itself is not the problem; effort spent making it faster would not fix the failures. Ingestion slows under load because portal logs compete with everything else in the queue. Acknowledgement slows when analysts are busy or changing shifts. Separating stages makes the right fixes obvious. Amara plots each run as a stacked bar, one segment per stage, which makes the pattern visible at a glance.
40.5 Fixes
Ingestion: raise the priority of portal logs in the pipeline queue. Acknowledgement: page the possible-takeover alert to a dedicated device rather than the shared queue, and keep the 15-minute handover overlap.
Raising priority for portal logs means they move ahead of bulk logs when the pipeline is busy. A dedicated paging device for takeover alerts makes them stand out from the general queue, so they are not lost among routine work. Keeping the handover overlap gives the incoming analyst time to pick up urgent alerts.
40.6 Re-verification
After fixes, two further busy and handover runs pass at 12.9 and 13.4 minutes.
Re-running the conditions that failed is essential; passing quiet-condition tests again would prove nothing new. Both new results fall comfortably under fifteen minutes. Two runs are still a small sample, so the timed injections continue monthly, and the results feed the requirement's verification record. If a later monthly run fails, the same stage analysis is repeated before any fix is chosen. A new failure in a stage that passed before suggests something changed, such as higher volume or a new shift pattern, and that change is investigated first.
Test hygiene in production
The test customer account is flagged so its alerts are excluded from metrics and customer service never calls a real person. After each run, Amara confirms the test hold has been released and records the clean-up in the laboratory log. Analysts are thanked, not judged, for slow acknowledgements; the point is to find system causes.
Worked example
Amara's laboratory report states: "R-07 initially failed in 2 of 5 conditions (busy, handover). Causes: ingestion lag and acknowledgement delay. After prioritised ingestion and dedicated paging, re-runs passed. Requirement status: verified under tested conditions; retest each quarter."
Synthetic case. Times come from the lab script; the re-runs are recorded results.
Errors and limits
Five runs give limited coverage; extreme peaks remain untested.
Exercises
- EX118 — Pass and fail. Which runs passed and which failed? Rubric: 3 = R1, R2, R4 pass; R3 busy 19.7 and R5 handover 16.8 fail; 2 = partial; 1 = wrong; 0 = none.
- EX119 — Failure stages. Which stages caused the failures? Rubric: 3 = ingestion under load; acknowledgement under load and at handover; 2 = partial; 1 = detection; 0 = none.
- EX120 — Fixes applied. What fixes were made, and what did re-verification show? Rubric: 3 = prioritised portal ingestion, dedicated paging; re-runs passed 12.9 and 13.4; 2 = partial; 1 = vague; 0 = none.
Synthesis
Verify end-to-end requirements under realistic conditions, find the failing stage, fix it and verify again.
Continue studying — chapter 40 of 50
← Chapter 39 (draft) · Course chapter index · Chapter 41 (draft) →