◇ OpenCyber Academy
Bachelor's · Chapter 39 · English

Chapter 39 — Capstone evaluation

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 39 — Capstone evaluation

Lesson map

Lesson 39 published. Map, example, EX115.

flowchart TD
  N0["Learning outcomes"]
  N1["39.1 Against the proposal"]
  N2["39.2 Results"]
  N3["39.3 Recall"]
  N4["39.4 User feedback"]
  N5["39.5 Limitations"]
  N6["39.6 Future work"]
  N7["39.7 Cost and benefit"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX115"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["39.1 Against the proposal"]
  N2["39.2 Results"]
  N3["39.3 Recall"]
  N4["39.4 User feedback"]
  N5["39.5 Limitations"]
  N6["39.6 Future work"]
  N7["39.7 Cost and benefit"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX115"]

[Learning outcomes]
    |
    v
[39.1 Against the proposal]
    |
    v
[39.2 Results]
    |
    v
[39.3 Recall]
    |
    v
[39.4 User feedback]
    |
    v
[39.5 Limitations]
    |
    v
[39.6 Future work]
    |
    v
[39.7 Cost and benefit]
    |
    v
[Worked example] --> [Exercise EX115]

Learning outcomes

You will (1) evaluate the capstone against its success measures; (2) compute precision and discuss recall limits; (3) gather user feedback; (4) state limitations and future work.

39.1 Against the proposal

Evaluation returns to the proposal's measures. Amara planned to report precision, analyst time and number of relevant lookalikes in a pilot month.

Amara writes the evaluation in the same order as the proposal's measures, so readers can match each result to its target. Where she could not measure something as planned, she says so and explains the substitute. She also reports what was not in the proposal but proved important, such as the time spent tuning rules in the first week.

39.2 Results

Over four weeks, the matcher flagged 25 domains; the supervisor judged 6 suspicious, giving precision of 24%. Analyst time averaged 50 minutes per week. One suspicious domain led to a takedown request.

Amara reports the counts as well as the percentage, so readers can see the precision rests on 25 domains. Most false positives were domains belonging to unrelated organisations with similar names, which the feedback below addresses. Fifty minutes a week is within what the supervisor said the team could sustain. The takedown request was made by HCHT's security lead after reviewing the evidence, as the proposal's scope required.

39.3 Recall

Recall — the share of all real lookalikes found — cannot be measured directly because the total is unknown. Amara estimates partial recall by checking whether three lookalikes reported independently by HSSC during the month were caught: two were, one used a pattern her rules lacked.

Amara reports this as a rough indication, not a recall figure, since three cases are far too few to estimate a rate. The missed lookalike is added to the synthetic test list so that the next version of the rules is tested against it.

39.4 User feedback

The supervisor values the weekly sheet but asks for fewer obvious false positives, such as domains belonging to unrelated organisations with similar initials.

Amara adds a filter for domains registered long ago to organisations with similar initials, tests it on the month's data and finds it would have removed most of the obvious false positives without losing any suspicious domains. The change is recorded in the decision log and will be tested on new data before it is adopted. She thanks the supervisor for specific feedback, which made the fix easy to design.

39.5 Limitations

Only certificate-issuing domains are visible; domains without certificates are missed. Rules are English-centric. One month is short.

Amara explains each limitation's likely effect. Missing domains without certificates means some lookalikes, such as those used only for email, are not seen at all. English-centric rules miss tricks based on other languages common in the region. A single month may not include a large campaign, so precision and time could differ in a busier period. She suggests how each limit could be reduced.

39.6 Future work

Add rules for the missed pattern, filter domains with long legitimate histories, and test for three more months.

Amara writes each item as a task with an expected result, so a successor knows what to do and how to tell if it worked. The missed pattern is a combined hyphen and added word, which a single new rule can cover. Filtering long histories should reduce false positives, as the feedback suggested. Three more months would show whether results hold in different conditions.

39.7 Cost and benefit

Fifty minutes a week is modest compared with the effort of responding to a successful credential-phishing campaign. The evaluation states the cost plainly and avoids claiming savings it cannot measure; the benefit is presented as earlier warning of six suspicious domains, one of which was taken down.

Amara presents both figures side by side. She describes what is uncertain, such as whether any of the six domains would have been used in an attack. The security lead judges that the cost is justified for a further trial, which is the decision the evaluation was meant to support. Amara includes the lead's view as feedback rather than as a measured benefit.

Comparing with the baseline

Before the pilot, lookalikes reached the team only when staff reported suspicious emails. Comparing dates shows whether the new process warned earlier.

Worked example

Amara's draft claims the system "protects HCHT from lookalike phishing". Her tutor asks for evidence. She rewrites: "In a four-week pilot, the process gave early warning of six suspicious domains, one subsequently taken down; its effect on actual phishing success was not measured."

Synthetic case. The results are invented.

Errors and limits

Short pilots overstate novelty effects; long-term value needs longer measurement.

Exercises

Synthesis

Measure against the plan, report precision and partial recall, listen to users and state limits plainly.

Continue studying — chapter 39 of 40

← Chapter 38 (draft) · Course chapter index · Chapter 40 (draft) →