OpenCyber Academy
Bachelor's · Chapter 23 · English
Course code: OCA-108Chapter 23: OCA-108-C023

Chapter 23 — Triage and hypothesis testing methods

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 23 published. Map, example, EX067.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["23.1 Hypotheses, not conclusions"]
  N2["23.2 Priors and likelihood ratios"]
  N3["23.3 Corroboration and independence"]
  N4["23.4 Severity"]
  N5["23.5 Stopping rules and reversal conditions"]
  N6["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX067"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["23.1 Hypotheses, not conclusions"]
  N2["23.2 Priors and likelihood ratios"]
  N3["23.3 Corroboration and independence"]
  N4["23.4 Severity"]
  N5["23.5 Stopping rules and reversal conditions"]
  N6["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX067"]

[Learning outcomes and connection]
    |
    v
[23.1 Hypotheses, not conclusions]
    |
    v
[23.2 Priors and likelihood ratios]
    |
    v
[23.3 Corroboration and independence]
    |
    v
[23.4 Severity]
    |
    v
[23.5 Stopping rules and reversal conditions]
    |
    v
[Chapter synthesis]
    |
    v
[Worked example] --> [Exercise EX067]

Learning outcomes and connection

You will be able to (1) state a prior belief and update it with evidence using likelihood ratios; (2) seek corroboration from independent sources; (3) test a favoured hypothesis against a competing explanation; (4) set severity from impact and confidence; and (5) define a stopping rule and a reversal condition. Chapter 12 introduced conditional probability; Chapter 22 assessed source strength. Here they become a triage method. Chapter 24 turns triage results into containment decisions.

23.1 Hypotheses, not conclusions

Triage begins with competing hypotheses: malicious activity, legitimate but unusual activity, misconfiguration, tool error. Chamberlin's 1890 essay on "multiple working hypotheses" and Platt's 1964 "strong inference" both argue for keeping several explanations alive and designing tests that discriminate between them. A test that every hypothesis predicts equally is useless, however much effort it costs.

The Saturday identity alerts at Harbourline show the method: password spraying and a maintenance side effect are both written down at the start, and each piece of evidence is judged by how differently the two explanations predict it. Failure reason codes discriminate well; the bare fact that sign-ins failed does not, because both hypotheses expect it.

23.2 Priors and likelihood ratios

A prior is your belief before new evidence, informed by base rates (how often does this alert turn out malicious here?). A likelihood ratio (LR) is P(evidence | H1) / P(evidence | H2): how much more expected the evidence is under one hypothesis than the other. In odds form, posterior odds = prior odds × LR. You rarely know LRs precisely, but estimating them roughly forces the right question: would I expect to see this if the benign explanation were true?

In the maintenance example the prior came from history: 2 of 40 similar alerts were malicious.

23.3 Corroboration and independence

Two pieces of evidence corroborate only if they are independent — not derived from the same source or failure (Chapter 2). Multiplying LRs assumes independence; if two alerts are both generated from the same sign-in log, they count once.

In the maintenance example, the timing of failures and their reason code both come from the IdP log. They describe different attributes, so they are not duplicates, but a fault in that one log would affect both. That is why the worked example treats its posterior as an order of magnitude. Evidence from a separate system, such as the directory connector's own restart record, would corroborate more strongly than another field from the same log.

23.4 Severity

Severity combines impact (what would be harmed if the hypothesis is true) and confidence (how likely it is true). A low-confidence, high-impact hypothesis may warrant urgent cheap checks; a high-confidence, low-impact one may wait.

In the maintenance example, the 14 accounts include clinical staff, so impact would be high if the attack hypothesis were true. Before the reason codes were checked, confidence was low but the stakes justified quick checks within a short time limit and escalation by default if they could not be completed. Write severity as both parts, impact and confidence, so a reader can see which one is driving urgency.

23.5 Stopping rules and reversal conditions

Triage must end with a decision: escalate, close, or monitor. A stopping rule says when you have enough evidence (for example posterior probability above 0.8 or below 0.05, or a time limit with default escalation for high impact). A reversal condition states what new evidence would change the decision; recording it makes the decision reviewable.

The maintenance case closes with a reversal condition covering 48 hours: a sign-in from a new device or address for those accounts, or any invalid-password failures in the window, reopens it. That turns closure into a testable statement rather than a judgement nobody can revisit.

23.6 Worked example (synthetic): identity alerts during maintenance

Fictional. At 22:10 on a Saturday, Harbourline's IdP raises "multiple failed sign-ins followed by success" for 14 staff accounts within 20 minutes. A database maintenance window ran 21:30–23:00.

H1: password-spraying attack with some successes. H2: maintenance caused authentication failures (the IdP's connector to the staff directory was restarted), and users' devices retried automatically.

Prior: in this organization's last 12 months, 2 of 40 such alerts were malicious → prior odds 2:38 ≈ 0.053 (probability 0.05).

Evidence and rough LRs (H1 vs H2): - E1: failures cluster 21:42–21:48, inside the window when the connector restarted. Under H2 this is expected; under H1 it would be a coincidence. LR ≈ 0.2. - E2: failure reason code "directory unavailable", not "invalid password". Strongly expected under H2. LR ≈ 0.1. - E3: successful sign-ins came from the users' usual devices and addresses. LR ≈ 0.3. Posterior odds ≈ 0.053 × 0.2 × 0.1 × 0.3 ≈ 0.0003 → probability ≈ 0.03%.

Independence check: E1 and E2 both come from the IdP log but measure different attributes (timing and reason code); they are not fully independent, so treat the posterior as an order of magnitude, not a precise value.

Decision: close as maintenance-related, with a note to the IdP owner about the connector restart. Reversal condition: any sign-in in the following 48 hours for these accounts from a new device or address, or any "invalid password" failures in the window, reopens the case.

23.7 Common errors and limits

Exercises — chapter 23

Attempt each exercise before opening the worked results. Every case, organization, person, address and number in an exercise is synthetic unless a cited public source is named.

EX067 — Update by hand. Inputs: prior probability 0.10; two independent pieces of evidence with LRs 4 and 2.5 in favour of H1. Steps: compute the posterior probability. Rubric: 3 = prior odds 0.111, posterior odds 1.11, probability ≈ 0.53; 2 = arithmetic slip; 1 = adds probabilities; 0 = none.

EX068 — Discriminating test. Inputs: §23.6. Steps: propose one additional test that discriminates strongly between H1 and H2 and one that does not. Rubric: 3 = discriminating: failures from external addresses or for non-existent usernames (expected under spraying, not maintenance); non-discriminating: "were there failed sign-ins?" (both predict); 2 = only one; 1 = neither discriminates; 0 = none.

EX069 — Severity and stopping. Steps: write the severity statement and stopping rule you would put in the case record for §23.6 before E2 was known. Rubric: 3 = impact high (14 staff accounts incl. clinical), confidence low; quick checks within 30 minutes; escalate by default if checks cannot complete; 2 = partial; 1 = no rule; 0 = none.

Chapter synthesis

Keep competing hypotheses, start from base rates, update with discriminating and independent evidence, set severity from impact and confidence, and end every triage with a decision, a stopping rule and a reversal condition.

Worked answers

EX067. Prior odds = 0.10 / 0.90 = 0.111. Posterior odds = 0.111 × 4 × 2.5 = 1.11. Posterior probability = 1.11 / 2.11 ≈ 0.53.

EX068. Discriminating: check whether failures include usernames that do not exist or came from addresses outside the users' normal ones — password spraying typically tries many accounts, including invalid ones, from attacker infrastructure; maintenance retries come from users' own devices for valid accounts. Non-discriminating: "were there failed sign-ins followed by success?" — that is the alert itself, predicted by both.

EX069. "Severity: provisional high — 14 staff accounts including clinical users could be affected if H1 is true; confidence in H1 low (base rate 5%). Stopping rule: complete failure-reason and source-address checks within 30 minutes; if posterior probability of H1 < 5% close with reversal condition; if > 50% escalate to incident lead; if checks cannot complete in 30 minutes, escalate by default."