Chapter 30 — Laboratory II
Lesson map
Lesson 30 published. Map, example, EX088.
flowchart TD
N0["Learning outcomes"]
N1["30.1 The task"]
N2["Lab — Tuning an impossible-travel rule"]
N3["30.2 Tuning"]
N4["30.3 Reporting"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX088"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["30.1 The task"]
N2["Lab — Tuning an impossible-travel rule"]
N3["30.2 Tuning"]
N4["30.3 Reporting"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX088"]
[Learning outcomes]
|
v
[30.1 The task]
|
v
[Lab — Tuning an impossible-travel rule]
|
v
[30.2 Tuning]
|
v
[30.3 Reporting]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX088]
Learning outcomes
You will (1) simulate log review for suspicious sign-ins; (2) tune a simple detection rule; (3) measure false positives and misses on synthetic data.
30.1 The task
The second laboratory asks Lucas to write a detection rule for suspicious sign-ins using synthetic logs from Fernway's identity system. Each record has a user, a time, a country and a result. He starts with a rule that flags sign-ins from a country other than the user's usual one. The synthetic data includes labelled events, some of them genuine attacks planted by Dr Bellini, so Lucas can measure how well his rule works. All data is generated, so no real staff activity is examined.
Lab — Tuning an impossible-travel rule
# Tune a sign-in rule on labelled synthetic logs: (user, hour, country, attack?)
logs = [("ana", 9, "UK", 0), ("ana", 10.5, "BR", 1), ("ben", 8, "UK", 0), ("ben", 20, "FR", 0),
("cai", 7, "UK", 0), ("cai", 8.5, "RU", 1), ("dee", 9, "UK", 0), ("dee", 15, "ES", 0),
("eli", 12, "UK", 0), ("eli", 13.5, "CN", 1), ("fay", 9, "DE", 0), ("fay", 18, "DE", 0),
("gus", 9, "UK", 0), ("gus", 30, "IT", 0), ("hal", 9, "UK", 0), ("hal", 9.8, "US", 1)]
home = "UK"
def evaluate(name, flags):
tp = sum(1 for i in flags if logs[i][3]); fp = len(flags) - tp
miss = sum(r[3] for r in logs) - tp
print(f"{name}: flagged {len(flags)}, attacks caught {tp}, false positives {fp}, missed {miss}")
evaluate("any foreign", [i for i, r in enumerate(logs) if r[2] != home])
def travel(window):
out = []
for i, r in enumerate(logs):
for j, p in enumerate(logs[:i]):
if p[0] == r[0] and p[2] != r[2] and r[1] - p[1] <= window: out.append(i)
return out
evaluate("2-hour travel", travel(2)); evaluate("1-hour travel", travel(1))
Run with python3 lab_detect.py. Verified output:
any foreign: flagged 9, attacks caught 4, false positives 5, missed 0
2-hour travel: flagged 4, attacks caught 4, false positives 0, missed 0
1-hour travel: flagged 1, attacks caught 1, false positives 0, missed 3
30.2 Tuning
The first rule flags every foreign sign-in, including buyers on supplier trips, producing many false alarms. Lucas adds a condition: flag only when two successful sign-ins for the same user come from different countries within two hours, which is too fast to travel. The lab shows the change: false positives fall sharply while the planted attacks are still caught. He also tests a tighter one-hour window and finds it misses three of the four attacks, so he keeps two hours. Tuning is a trade between noise and misses, measured on labelled data.
30.3 Reporting
Lucas reports the rule's performance on synthetic data and warns that real data will differ. He recommends a two-week trial in Fernway's identity tool in alert-only mode, with Hannah recording which alerts were real. He notes that the rule misses attackers who sign in from the same country as the user, so it complements rather than replaces phishing-resistant authentication. Dr Bellini notes that stating what a rule cannot detect is as important as its detection rate, because defenders otherwise trust it too much.
Worked example
Lucas writes and tunes an impossible-travel rule on labelled synthetic logs, compares windows by false positives and misses and plans a trial.
Synthetic case. Logs and attacks are synthetic.
Errors and limits
Synthetic data rarely matches real noise. VPNs and travel apps can confuse location rules.
Exercises
- EX088 — First rule problem. What was wrong with flagging all foreign sign-ins? Rubric: 3 = buyers on trips caused many false alarms;; 2 = false positives;; 1 = vague;; 0 = none.
- EX089 — Window choice. Why keep two hours rather than one? Rubric: 3 = one hour missed three of four attacks;; 2 = miss;; 1 = faster;; 0 = none.
- EX090 — Rule blind spot. What does the rule miss? Rubric: 3 = attackers signing in from the user's own country;; 2 = same country;; 1 = vague;; 0 = none.
Synthesis
Detection rules are tuned on labelled data by trading false alarms against misses, and their blind spots must be stated.
Continue studying — chapter 30 of 40
← Chapter 29 (draft) · Course chapter index · Chapter 31 (draft) →