Chapter 5 — Proposal and pilot
Lesson map
Lesson 05 published. Map, example, EX027.
flowchart TD
N0["Learning outcomes"]
N1["5.1 Proposal"]
N2["5.2 Pilot sensor"]
N3["5.3 Pilot changes"]
N4["5.4 Pilot lessons"]
N5["Choosing what to tune first"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX027"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["5.1 Proposal"]
N2["5.2 Pilot sensor"]
N3["5.3 Pilot changes"]
N4["5.4 Pilot lessons"]
N5["Choosing what to tune first"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX027"]
[Learning outcomes]
|
v
[5.1 Proposal]
|
v
[5.2 Pilot sensor]
|
v
[5.3 Pilot changes]
|
v
[5.4 Pilot lessons]
|
v
[Choosing what to tune first]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX027]
Learning outcomes
You will (1) write the dissertation proposal; (2) pilot tuning on one sensor; (3) refine the procedure.
5.1 Proposal
The proposal states the question, measures, design, governance, ethics and analysis plan. It sets success criteria in advance: at least a 50 percent reduction in normalised alert volume with recall no lower than 0.90 on the labelled set.
Fixing recall at no lower than 0.90 stops a large volume cut from being bought by silencing rules that catch real attacks.
The proposal sets out the question, design, measures, timeline, risks and data handling, and is approved by her supervisor and Corvane before work begins. It becomes the reference against which later changes are judged, and any departure is recorded with a reason.
5.2 Pilot sensor
Tuning is piloted on one sensor for two weeks. Amara reviews the top twenty noisiest rules by volume, consulting closure notes. Most noise comes from three sources: a backup server's scheduled transfers matching a data-exfiltration rule, a monitoring system's scans of its own hosts and rules for services Corvane does not run.
The pilot uses one sensor in one data centre, limiting impact if tuning removes something important, while giving realistic evidence.
Analysts on that sensor agree to report anything unusual.
5.3 Pilot changes
The backup server and monitoring system are added as documented exceptions to the specific rules they trigger, not globally. Rules for absent services are disabled with evidence. Thresholds for one scanning rule are raised from five to twenty connections per minute after analysts confirm legitimate tools exceed five.
Each pilot change is logged with the rule, the reason, the reviewer and the expected effect, then compared with what actually happened.
Unexpected effects are reviewed before further changes.
Patterns guide rollout.
5.4 Pilot lessons
The pilot shows that exceptions must be narrow: an early draft exempted the backup server from all rules, which reviewers rejected. The procedure is revised to require rule-specific exceptions with an expiry date.
The pilot showed which parts of the method worked and which needed revision. Amara records lessons such as rules whose tuning needed more context than expected, and adjusts the procedure for the full rollout. Lessons are written down promptly with evidence, so the rollout plan is based on what happened rather than on recollection.
Choosing what to tune first
With hundreds of rules, Amara needs an order. She ranks rules by alert volume and by how often their alerts were closed as benign, focusing first on rules that are both noisy and almost always benign. Rules that fire rarely but are sometimes real are left alone, even if they produce occasional false alarms, because the cost of missing their true detections is high. This ordering puts the analysts' effort where it reduces the most noise for the least risk, and it makes each change easy to justify in the register.
Worked example
The pilot sensor's normalised alerts fall by 64 percent with no labelled event missed.
Synthetic case. Data are synthetic.
Errors and limits
Global exceptions for noisy hosts create blind spots if those hosts are compromised.
Exercises
- EX027 — Success criteria. State the two success criteria. Rubric: 3 = both;; 2 = one;; 1 = vague;; 0 = none.
- EX028 — Noise sources. Name the three main noise sources. Rubric: 3 = three;; 2 = two;; 1 = one;; 0 = none.
- EX029 — Narrow exceptions. Why reject a global exception for the backup server? Rubric: 3 = blind spot if compromised;; 2 = too broad;; 1 = vague;; 0 = none.
- EX030 — Threshold change. What justified raising the scanning threshold? Rubric: 3 = analyst confirmation legitimate tools exceed five;; 2 = evidence;; 1 = vague;; 0 = none.
- EX031 — Expiry dates. Why give exceptions expiry dates? Rubric: 3 = force review;; 2 = review;; 1 = vague;; 0 = none.
- EX032 — Pilot result. What was the pilot result? Rubric: 3 = 64% reduction, no missed events;; 2 = reduction;; 1 = vague;; 0 = none.
Synthesis
The pilot refines tuning into narrow, documented, expiring changes.
Continue studying — chapter 05 of 08
← Chapter 04 (draft) · Course chapter index · Chapter 06 (draft) →