Chapter 17 — Contained laboratory II
Lesson map
Lesson 17 published. Map, example, EX065.
flowchart TD
N0["Learning outcomes"]
N1["17.1 The lab task"]
N2["Lab — Detecting suspicious sign-ins on synthetic rec"]
N3["17.2 Measuring"]
N4["17.3 Recommending"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX065"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["17.1 The lab task"]
N2["Lab — Detecting suspicious sign-ins on synthetic rec"]
N3["17.2 Measuring"]
N4["17.3 Recommending"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX065"]
[Learning outcomes]
|
v
[17.1 The lab task]
|
v
[Lab — Detecting suspicious sign-ins on synthetic rec]
|
v
[17.2 Measuring]
|
v
[17.3 Recommending]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX065]
Learning outcomes
You will (1) test detection of suspicious sign-ins in a lab; (2) measure detection and false alarms; (3) recommend settings for Linden.
17.1 The lab task
Daniel's second lab uses synthetic sign-in records from a cloud practice system. Each record shows user, hour, location and whether it succeeded, with a few planted malicious sign-ins. Nadia must design a rule that alerts on suspicious sign-ins without flooding Grace with alarms. Linden staff work in five towns, mostly between seven in the morning and seven in the evening, so she starts by describing normal behaviour from the data before writing any rule. She writes her description of normal in a few sentences and has Daniel check it before she starts coding.
Lab — Detecting suspicious sign-ins on synthetic records
# Compare two sign-in alert rules on synthetic records: (user, hour, town, attack?)
recs = [("dr.ali", 9, "Elmfield", 0), ("dr.ali", 23, "Elmfield", 0), ("n.ross", 8, "Norbury", 0),
("n.ross", 14, "Norbury", 0), ("dr.ali", 3, "Lisbon", 1), ("k.chan", 10, "Ashby", 0),
("k.chan", 2, "Unknown", 1), ("m.ojo", 12, "Kingsley", 0), ("m.ojo", 22, "Riga", 1),
("p.lee", 18, "Westcott", 0), ("p.lee", 11, "Westcott", 0)]
seen = {}
for u, h, t, _ in recs:
if 7 <= h < 19: seen.setdefault(u, set()).add(t)
def rule_a(r): return not (7 <= r[1] < 19)
def rule_b(r): return rule_a(r) and r[2] not in seen.get(r[0], set())
for name, rule in (("Rule A", rule_a), ("Rule B", rule_b)):
hits = [r for r in recs if rule(r)]
caught = sum(r[3] for r in hits)
print(f"{name}: {len(hits)} alerts, {caught} attacks caught, {len(hits) - caught} false alarms")
Run with python3 lab_signin.py. Verified output:
Rule A: 4 alerts, 3 attacks caught, 1 false alarms
Rule B: 3 alerts, 3 attacks caught, 0 false alarms
17.2 Measuring
Nadia tests two rules. Rule A alerts on any sign-in outside working hours. Rule B alerts on sign-ins outside working hours that also come from a location not used by that user before. The lab counts alerts, planted attacks caught and false alarms for each. Rule A catches all planted attacks but also flags a dentist who checks schedules late at night. Rule B catches the same attacks with fewer false alarms. Measuring both outcomes shows the trade-off clearly. She saves the counts for her report.
17.3 Recommending
Nadia recommends Rule B for Linden, with alerts sent to Grace and a weekly review, and notes its limits: an attacker using a stolen password during working hours from a usual town would not be flagged. She adds that multi-factor sign-in on the practice system matters more than any alert rule, since it blocks most stolen-password attacks outright. Daniel agrees, noting that detection complements prevention but cannot replace it. She writes the recommendation as a one-page note for Grace, listing the rule, who receives alerts, the weekly review and the known blind spot, so the decision can be revisited if Linden's working patterns change.
Worked example
Nadia compares two sign-in alert rules on synthetic data, chooses the one with fewer false alarms and states its blind spot.
Synthetic case. Records and attacks are synthetic.
Errors and limits
Planted attacks may be easier than real ones. Location data can be wrong.
Exercises
- EX065 — Normal first. What did Nadia describe before writing rules? Rubric: 3 = normal behaviour: five towns, 7am to 7pm;; 2 = normal;; 1 = vague;; 0 = none.
- EX066 — Rule A result. What were Rule A's results? Rubric: 3 = 4 alerts, 3 attacks caught, 1 false alarm;; 2 = numbers;; 1 = vague;; 0 = none.
- EX067 — Rule B result. What were Rule B's results? Rubric: 3 = 3 alerts, 3 attacks caught, 0 false alarms;; 2 = numbers;; 1 = vague;; 0 = none.
- EX068 — Blind spot. What would Rule B miss? Rubric: 3 = a stolen password used in working hours from a usual town;; 2 = working hours;; 1 = vague;; 0 = none.
Synthesis
Measuring detection and false alarms on labelled data shows which rule to deploy, while prevention remains essential.
Continue studying — chapter 17 of 18
← Chapter 16 (draft) · Course chapter index · Chapter 18 (draft) →