OpenCyber Academy
Engineering · Chapter 38 · English
Course code: OCA-133Chapter 38: OCA-133-C038

Chapter 38 — Threat intelligence and OSINT: validation

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 38 published. Map, example, EX112.

flowchart TD
  N0["Learning outcomes"]
  N1["38.1 Validation questions"]
  N2["38.2 Acceptance scenarios"]
  N3["38.3 Decision impact"]
  N4["38.4 When requirements were wrong"]
  N5["38.5 Independent observers"]
  N6["Practice note — negative results"]
  N7["Feedback loop"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX112"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["38.1 Validation questions"]
  N2["38.2 Acceptance scenarios"]
  N3["38.3 Decision impact"]
  N4["38.4 When requirements were wrong"]
  N5["38.5 Independent observers"]
  N6["Practice note — negative results"]
  N7["Feedback loop"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX112"]

[Learning outcomes]
    |
    v
[38.1 Validation questions]
    |
    v
[38.2 Acceptance scenarios]
    |
    v
[38.3 Decision impact]
    |
    v
[38.4 When requirements were wrong]
    |
    v
[38.5 Independent observers]
    |
    v
[Practice note — negative results]
    |
    v
[Feedback loop]
    |
    v
[Worked example] --> [Exercise EX112]

Learning outcomes

You will (1) validate Lantern against stakeholders' real needs; (2) design acceptance scenarios with operators; (3) measure decision impact, not only system metrics; (4) handle findings that requirements were wrong.

38.1 Validation questions

Does the SOC block threats sooner? Do OT engineers get vendor warnings they act on? Do executives understand strategic briefs? These questions concern outcomes; system metrics alone cannot answer them.

Kofi takes each question to the people concerned and asks how they would recognise an answer. The SOC lead suggested comparing times from publication to enforcement before and after Lantern. The OT engineers wanted to count vendor warnings that led to an action. Executives were asked directly whether the strategic brief changed any decision during the quarter.

38.2 Acceptance scenarios

With operators, Kofi writes scenarios: "A partner shares a phishing campaign against utilities at 09:00; by 09:30 domain blocks are active, a ticket with ATT&CK context exists and the duty analyst has hunted for prior contacts." Scenarios run on synthetic data in a staging environment that mirrors production.

The scenario includes the human steps as well as the system's, because a block that arrives on time but leaves no ticket would still fail the duty analyst. Running it in staging means the operators can rehearse without risk to production.

38.3 Decision impact

Measure before and after Lantern: median time from external publication to enforcement; percentage of incidents where intelligence context was available at triage; executive decisions that referenced a strategic brief. Some measures are rough; report them with their limits.

Kofi gathered the earlier figures from tickets and email records of the manual process, which were less precise than Lantern's logs. He reported the comparison with that caveat. Counting executive decisions that cited a brief relies on minutes of meetings, which may not mention every influence. Rough measures are still better than none, as long as readers know how rough they are.

38.4 When requirements were wrong

Validation may show that a verified requirement serves no one — for example, a daily digest nobody reads. That is a valuable finding: change the requirement, not the measurement.

The worked example shows a second form of the same lesson: vendor alerts were delivered as required, but to an inbox that nobody watched. The system passed its verification test and still failed its users. Kofi changed the requirement to name the destination the OT team actually uses, then wrote a new verification procedure for it.

38.5 Independent observers

Someone outside the build team observes scenario runs and interviews users, reducing the temptation to declare success.

Kofi asked a colleague from BEN's internal audit team to observe. She noted that during one scenario an analyst took a manual step the design had not described, which revealed a gap in the interface. Her report was attached to the validation record without edits from the build team. Users also spoke more freely to her than to the engineers who had built the system, and two of their complaints became backlog items.

Practice note — negative results

If a scenario shows Lantern adds nothing to a decision, record it plainly. Negative validation results justify retiring features and save maintenance effort.

Feedback loop

Validation findings are scheduled into the next release plan with owners, and the scenario is rerun after the change to confirm the outcome improved.

Worked example

Median time from partner publication to DNS enforcement fell from 9 hours (manual process) to 25 minutes. But OT engineers reported vendor alerts arrived in an inbox they rarely check. The requirement is changed to deliver vendor alerts to the OT team's ticket queue.

Synthetic case. Figures are invented.

Errors and limits

Before-and-after comparisons can be confounded by other changes; note concurrent changes. Users' praise is not measurement.

Exercises

Synthesis

Validate outcomes with operators, measure decisions, accept that requirements can be wrong, and use independent observers.

Continue studying — chapter 38 of 50

← Chapter 37 (draft) · Course chapter index · Chapter 39 (draft) →

Worked answers

EX112. Example: "Do duty analysts have intelligence context available when they triage phishing incidents, and does it change their actions?"

EX113. Change it so vendor alerts are delivered to the OT team's ticket queue, where they actually work.

EX114. Example: BEN also hired two SOC analysts during the same period, which alone could shorten enforcement time.