Chapter 5 — Proposal and pilot
Lesson map
Lesson 05 published. Map, example, EX027.
flowchart TD
N0["Learning outcomes"]
N1["5.1 Pilot purpose"]
N2["5.2 The click lab"]
N3["Lab — Scanner pre-fetch clicks separated from human"]
N4["5.3 Checking the rule"]
N5["5.4 Refining the proposal"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX027"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["5.1 Pilot purpose"]
N2["5.2 The click lab"]
N3["Lab — Scanner pre-fetch clicks separated from human"]
N4["5.3 Checking the rule"]
N5["5.4 Refining the proposal"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX027"]
[Learning outcomes]
|
v
[5.1 Pilot purpose]
|
v
[5.2 The click lab]
|
v
[Lab — Scanner pre-fetch clicks separated from human]
|
v
[5.3 Checking the rule]
|
v
[5.4 Refining the proposal]
|
v
[Worked example] --> [Exercise EX027]
Learning outcomes
You will (1) pilot an outcome measure before main collection; (2) detect automated activity contaminating security metrics; (3) refine a proposal from pilot evidence.
5.1 Pilot purpose
Before the main data collection, Priya pilots her outcome measure on one earlier simulation. She wants to know whether a click recorded by the simulation platform really means a person clicked. Dr. Bello reminds her that many mail security gateways open links automatically to check them for malicious content, and that such scanner activity can appear in logs as clicks. If the pilot finds this, her outcome needs a correction rule before the main study starts. The security team supplies an anonymised export from one earlier quarter.
5.2 The click lab
The pilot lab builds a synthetic log for 1,200 staff, recording the seconds between delivery and any click. It finds 239 raw clicks, a rate of 19.9 percent. Of these, 77 occur within two seconds of delivery, far faster than a person could read an email, and are attributed to the scanner. Removing them leaves 162 human clicks, 13.5 percent. The raw figure would have overstated clicking by more than six percentage points. Priya shows the security team a chart comparing raw and human rates. The team had been reporting raw figures to the board for two years and agrees the correction matters.
Lab — Scanner pre-fetch clicks separated from human clicks in a synthetic simulation log
"""Separate scanner pre-fetch clicks from human clicks in a synthetic phishing simulation log (stdlib only)."""
import random
random.seed(41)
rows = []
for staff in range(1200):
sec = None
r = random.random()
if r < 0.07:
sec = random.uniform(0.2, 1.8) # mail scanner opens the link almost at once
elif r < 0.21:
sec = random.uniform(30, 86400) # a person clicks later
rows.append((staff, sec))
clicked = [s for s in rows if s[1] is not None]
auto = [s for s in clicked if s[1] < 2]
human = len(clicked) - len(auto)
print(f"staff {len(rows)}; raw clicks {len(clicked)} ({len(clicked)/len(rows):.1%})")
print(f"clicks within 2 s (scanner) {len(auto)}; human clicks {human} ({human/len(rows):.1%})")
Run with python3 lab_clicks.py. Verified output:
staff 1200; raw clicks 239 (19.9%)
clicks within 2 s (scanner) 77; human clicks 162 (13.5%)
5.3 Checking the rule
A two-second threshold is a judgement, so Priya checks it. She asks the mail administrators whether the gateway scans links, and they confirm it does and that scans come from known addresses. Matching by address would be better than timing, but the platform does not keep addresses in the export she receives. She keeps the timing rule, reports it as a limitation and tests whether changing the threshold to five seconds changes conclusions in the main analysis. Dr. Bello agrees the approach is reasonable for a master's study.
5.4 Refining the proposal
From the pilot, Priya updates her proposal: the primary outcome becomes human click rate, excluding clicks within two seconds; report rates are measured from the reporting button logs; and simulations use varied templates. She adds a sentence explaining that unadjusted platform figures, sometimes quoted in vendor reports, can overstate risk. Dr. Bello approves the change and confirms that it happened before main data collection, so it remains part of the pre-registered plan. The revised proposal is circulated to the security team and to the ethics committee, which confirms that the change needs no further approval.
Worked example
Priya pilots the click measure, finds 77 of 239 clicks are scanner activity within two seconds, redefines the outcome as human click rate (13.5 percent rather than 19.9 percent) and updates her plan.
Synthetic case. Log data are synthetic; outputs come from lab_clicks.py.
Errors and limits
Counting scanner clicks as human clicks inflates measured risk.
Exercises
- EX027 — Raw clicks. How many raw clicks does the pilot find? Rubric: 3 = 239, 19.9 percent;; 2 = partly correct;; 1 = wrong;; 0 = none.
- EX028 — Scanner clicks. How many clicks occur within two seconds? Rubric: 3 = 77;; 2 = partly correct;; 1 = wrong;; 0 = none.
- EX029 — Human rate. What is the human click rate? Rubric: 3 = 13.5 percent;; 2 = partly correct;; 1 = wrong;; 0 = none.
- EX030 — Better method. What would be better than timing? Rubric: 3 = matching known scanner addresses;; 2 = partly correct;; 1 = vague;; 0 = none.
- EX031 — Threshold test. Which alternative threshold is tested? Rubric: 3 = five seconds;; 2 = partly correct;; 1 = other;; 0 = none.
- EX032 — New outcome. What is the new primary outcome? Rubric: 3 = human click rate excluding two-second clicks;; 2 = partly correct;; 1 = raw;; 0 = none.
Continue studying — lesson 05 (chapter 05 of 08)
← Chapter 04 · Course chapter index · Chapter 06 →
Continue to the next authored lesson.