OpenCyber Academy
Postgraduate · Chapter 8 · English
Course code: OCA-154Chapter 08: OCA-154-C008

Chapter 8 — Difficult cases with conflicting evidence

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 08 published. Map, example, EX036.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["8.1 When evidence conflicts"]
  N2["8.2 Analysis of competing hypotheses"]
  N3["8.3 Provenance and missingness"]
  N4["8.4 Seeking disconfirmation"]
  N5["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX036"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["8.1 When evidence conflicts"]
  N2["8.2 Analysis of competing hypotheses"]
  N3["8.3 Provenance and missingness"]
  N4["8.4 Seeking disconfirmation"]
  N5["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX036"]

[Learning outcomes and connection]
    |
    v
[8.1 When evidence conflicts]
    |
    v
[8.2 Analysis of competing hypotheses]
    |
    v
[8.3 Provenance and missingness]
    |
    v
[8.4 Seeking disconfirmation]
    |
    v
[Chapter synthesis]
    |
    v
[Worked example] --> [Exercise EX036]

Learning outcomes and connection

You will be able to (1) build a hypothesis–evidence matrix and use it to find diagnostic evidence; (2) assess provenance and missingness when sources conflict; (3) seek disconfirmation deliberately; and (4) express confidence that reflects unresolved conflict. Chapter 3 reconciled specialist reports; this chapter handles cases where conflict persists. Chapter 9 shows how to brief leaders before it is resolved.

8.1 When evidence conflicts

Conflict has a limited number of sources: different things were measured; one source is wrong (clock error, parsing, configuration); one source was manipulated; the world is more complex than the hypotheses (both sources right, hypotheses incomplete). Each predicts different follow-up results; that is how you discriminate.

Calder Retail's payroll export is such a conflict: identity logs name hr-lead-2, while the HR workstation's host timeline shows only hr-assist-5 logged on at the time. Each of the four sources of conflict gives a hypothesis to test.

8.2 Analysis of competing hypotheses

Heuer's Psychology of Intelligence Analysis (CIA Center for the Study of Intelligence, 1999) describes Analysis of Competing Hypotheses (ACH): list hypotheses, list evidence, mark each cell as consistent, inconsistent or neutral, focus on evidence that is diagnostic (consistent with some hypotheses and inconsistent with others), and prefer the hypothesis with the least inconsistent evidence rather than the most consistent. ACH does not produce truth; it structures attention and exposes assumptions.

Calder's lab shows the method: of four hypotheses, H2 has no inconsistent evidence, H1 and H4 one each, and H3 two.

8.3 Provenance and missingness

Weight evidence by provenance (who recorded it, how, with which dependencies) and by what could have been missing. A log that would have recorded an event if it occurred, but did not, is diagnostic; a log that was not collecting at the time is neutral.

In Calder's payroll case, the application's record of who performed the export is weighted by how it is produced: if the application records the token owner rather than the session user, the identity log's naming of hr-lead-2 is consistent with several hypotheses. That is why H4, a logging or attribution error, stays on the list until the deployed version is tested.

8.4 Seeking disconfirmation

For the leading hypothesis, ask "what would we expect to see if it were false?" and look for that. Confirmation bias pushes teams to collect more consistent evidence, which is often non-diagnostic.

For Calder's leading hypothesis, H2, the team named what would disconfirm it: hr-assist-5 on a video call in another application with keyboard activity at 22:47, or hr-lead-2 admitting to sharing credentials for a legitimate task. It also checks the assumptions behind the diagnostic evidence: DHCP lease records for the workstation address, and a lab test of how the application records users.

8.5 Worked example and lab (synthetic): two accounts, one timeline

Fictional. At Calder Retail, payroll data was exported at 22:47 on 9 June. Identity logs show the export performed by hr-lead-2; the host timeline from the HR workstation shows hr-assist-5 logged on interactively at 22:40–23:05 and no hr-lead-2 logon.

Hypotheses: H1 hr-lead-2's credentials were stolen and used remotely; H2 hr-assist-5 used hr-lead-2's credentials at the workstation (credential sharing or theft by a colleague); H3 a scheduled export ran under hr-lead-2's delegated token, and hr-assist-5's presence is coincidental; H4 a logging or attribution error (the application records the token owner, not the session user).

Save as lab08.py.

# Lab PG-08 - a hypothesis-evidence matrix with diagnosticity (synthetic, stdlib only)
H = ["H1 remote theft", "H2 local use by colleague", "H3 scheduled job", "H4 attribution error"]
# C = consistent, I = inconsistent, N = neutral
E = {
    "E1 IdP: export by hr-lead-2 at 22:47":                 "CCCC",
    "E2 host: hr-assist-5 interactive 22:40-23:05":           "NCNN",
    "E3 IdP sign-in for hr-lead-2 from workstation IP 22:44": "ICIC",
    "E4 no scheduled job in export service config":           "CCIC",
    "E5 app docs: records session user, not token owner":     "CCCI",
}
def tally():
    inc = {h: 0 for h in H}
    for ev, marks in E.items():
        for h, m in zip(H, marks):
            inc[h] += (m == "I")
    return inc
print("inconsistencies:", tally())
diag = [ev for ev, m in E.items() if len(set(m) - {"N"}) > 1]
print("diagnostic evidence:", diag)

Expected output:

inconsistencies: {'H1 remote theft': 1, 'H2 local use by colleague': 0, 'H3 scheduled job': 2, 'H4 attribution error': 1}
diagnostic evidence: ['E3 IdP sign-in for hr-lead-2 from workstation IP 22:44', 'E4 no scheduled job in export service config', 'E5 app docs: records session user, not token owner']

Interpretation: H2 has no inconsistent evidence; H1 and H4 one each; H3 two. E1 is consistent with everything and so is not diagnostic — yet it is the evidence most teams would cite first. The diagnostic items are E3, E4 and E5. Before concluding, test the assumptions behind them: is the workstation IP in E3 really that workstation (DHCP lease records)? Is E5's documentation accurate for the deployed version (test in a lab)? Is there another job scheduler? Seek disconfirmation of H2: if hr-assist-5 was on a video call in another application at 22:47 with keyboard activity, or if hr-lead-2 admits sharing credentials for a legitimate task, the picture changes. Note the ethical and HR dimension: H2 implicates a named person; handle through HR and counsel with due process (Chapter 4), and do not circulate the hypothesis widely.

Confidence statement: "The export at 22:47 was most likely performed using hr-lead-2's credentials from the HR workstation while hr-assist-5 was logged on there (moderate confidence). Remote credential theft is less likely because the sign-in came from the workstation's address. This assessment depends on the workstation address mapping and on the application recording the session user; both are being verified. No conclusion about any individual's intent is drawn."

8.6 Common errors and limits

Calder's matrix shows the first error clearly: E1 was consistent with every hypothesis and so told the team nothing, yet it was the evidence most teams would cite first. H4, the error-in-the-evidence hypothesis, stayed on the list. And because H2 implicates a named person, the confidence statement ends with "no conclusion about any individual's intent is drawn", with HR and counsel handling the next steps.

Exercises — chapter 8

Attempt each exercise before opening the worked results. Every case, organization, person, address and number in an exercise is synthetic unless a cited public source is named.

EX036 — Run and extend. Steps: run Lab PG-08; add evidence E6 "DHCP shows workstation IP assigned to a different host at 22:44" marked appropriately; report the change. Rubric: 3 = sensible marks (for example E6 = "CINC"), output reported, E3's reliability reassessed; 2 = output only; 1 = not run; 0 = none.

EX037 — Missingness. Steps: explain how you would treat the absence of hr-lead-2 logon on the host timeline — diagnostic or neutral — and what determines it. Rubric: 3 = depends on whether the host logs network/remote logons and whether a web session would produce a host logon at all; likely neutral for a web application; 2 = partial; 1 = treats as proof; 0 = none.

EX038 — Disconfirmation plan. Steps: list three tests that could disconfirm H2. Rubric: 3 = specific, non-intrusive first (application session device ID, browser profile on workstation, other activity of hr-assist-5 at 22:47), with HR involvement for interviews; 2 = two; 1 = generic; 0 = none.

EX039 — Assumption register. Steps: list the assumptions behind E3–E5 with how to verify each. Rubric: 3 = IP-to-host mapping, clock alignment, app attribution semantics, completeness of job configs; verification methods; 2 = two; 1 = vague; 0 = none.

EX040 — Communication boundary. Steps: write who should know about H2 at this stage and why. Rubric: 3 = need-to-know: commander, HR lead, counsel, investigators; not the wider team or the named person's manager without HR; reasons: fairness, legal risk, evidence integrity; 2 = partial; 1 = broadcast; 0 = none.

Chapter synthesis

In conflicting cases, list competing hypotheses including evidence error, focus on diagnostic and inconsistent evidence, verify the assumptions that make evidence diagnostic, seek disconfirmation, and state confidence without conclusions about intent.

Worked answers

EX036. If DHCP shows the IP belonged to a different host at 22:44, E3 no longer places the sign-in at the HR workstation. Mark E6 as consistent with H1 (remote or other-host use), inconsistent with H2 (local use at the workstation), neutral or consistent with H3, consistent with H4 — e.g., "CINC". Rerun: H2 now has one inconsistency, and E3's marks should be revisited (E3 becomes less reliable). The lesson: diagnostic evidence rests on assumptions (IP mapping) that themselves need provenance.

EX037. If the export was performed through a web application, an interactive or network logon by hr-lead-2 on the workstation host would not necessarily be created — the browser session uses web authentication. So the absence is neutral unless the host would have logged the relevant activity. It becomes diagnostic only if the authentication path requires a host logon (for example, a thick client using Windows integrated authentication).

EX038. (1) Check the application's session details for device identifier or browser fingerprint and compare with the workstation's browser profile. (2) Review the workstation's browser history/profile for hr-lead-2 sessions (with authorization, minimal scope). (3) Check hr-assist-5's concurrent activity at 22:47 in other systems (for example, an active call or ticket work elsewhere) to see if local use was plausible. Interviews only through HR with due process.

EX039. E3: IP-to-host mapping (verify DHCP/NAC records); clock alignment between IdP and host (verify with shared events). E4: the export service's job configuration is complete and current (verify with service owner and change history; check other schedulers). E5: the application records session user, not token owner (verify in the deployed version via lab test and vendor confirmation).

EX040. At this stage: incident commander, HR lead, counsel and the two investigators — need-to-know. Not the wider incident team, not the line manager without HR's process, not the named employee until HR and counsel decide on an interview. Reasons: fairness and presumption of innocence; legal exposure (defamation, employment law); avoiding contamination of evidence or tipping off.