◇ OpenCyber Academy
Postgraduate · Chapter 11 · English

Chapter 11 — Applied project

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 11 — Applied project

Lesson map

Lesson 11 published. Map, example, EX051.

flowchart TD
  N0["Learning outcomes"]
  N1["11.1 Scoping"]
  N2["11.2 Planning"]
  N3["11.3 Execution"]
  N4["11.4 Evaluation"]
  N5["11.5 Handover"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX051"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["11.1 Scoping"]
  N2["11.2 Planning"]
  N3["11.3 Execution"]
  N4["11.4 Evaluation"]
  N5["11.5 Handover"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX051"]

[Learning outcomes]
    |
    v
[11.1 Scoping]
    |
    v
[11.2 Planning]
    |
    v
[11.3 Execution]
    |
    v
[11.4 Evaluation]
    |
    v
[11.5 Handover]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX051]

Learning outcomes

You will (1) scope an applied AI-security project with an organisation; (2) plan work, evidence and decisions; (3) execute with authorisation and safety; (4) evaluate outcomes against criteria; and (5) hand over results that the organisation can use.

11.1 Scoping

A good project answers a decision the organisation must make: "Should we enable autonomous rebooking?", "Does deduplication reduce leakage from our fine-tuned model?" Agree the decision owner, the systems in scope, authorisation in writing, data rules (synthetic or approved), success criteria and timeline.

Tomás's project in the worked example below was scoped this way, with agreed criteria for advantage and reply quality. For Tomás's project, the decision owner was Amira, the system in scope was an isolated copy of the reply-suggestion model, and the data was synthetic customer messages. Write all of this down and get it signed before any testing starts.

11.2 Planning

Plan in phases: baseline (current state and risk), intervention (design or controls), evaluation (controlled tests), decision support (report and recommendation), handover. Each phase has a deliverable and a review. Keep a risk register for the project itself: test damage, data exposure, schedule slips.

For Tomás's project, the phases were a baseline leakage measurement, deduplication as the intervention, controlled tests on the isolated copy, a recommendation and handover. Each deliverable was reviewed with Amira, the decision owner, and the risk register was updated at the weekly review.

11.3 Execution

Work on isolated copies. Log every test. Stop and escalate if you find a live vulnerability affecting customers; that takes priority over the project timeline. Hold weekly reviews with the decision owner.

Tomás worked only on Corvane's isolated copy and logged each run with versions. Record anything unexpected in the project log, even if it did not change the result, and tell the decision owner about it at the next weekly review. Keep test data and logs on the isolated copy.

11.4 Evaluation

Measure against the criteria agreed at scoping, including side effects.

"""Lab M: membership inference against an overfit 1-nearest-neighbour 'model'
that returns a confidence score. Synthetic records only; teaching model."""
import random, math
rng = random.Random(5)
def rec(): 
    y = rng.random() < 0.5
    return [rng.gauss(0.6 if y else -0.6, 1.0) for _ in range(4)], int(y)
members = [rec() for _ in range(150)]; nonmembers = [rec() for _ in range(150)]
def confidence(x, y, k):
    # k-NN vote share for the true label, computed over the training set
    d = sorted((sum((a-b)**2 for a, b in zip(x, m)), my) for m, my in members)
    return sum(1 for _, my in d[:k] if my == y)/k
def auc(pos, neg):
    s = sum((p > n) + 0.5*(p == n) for p in pos for n in neg)
    return s/(len(pos)*len(neg))
for k in (1, 5, 25):
    pos = [confidence(x, y, k) for x, y in members]
    neg = [confidence(x, y, k) for x, y in nonmembers]
    best = max(sum(p >= t for p in pos)/len(pos) - sum(n >= t for n in neg)/len(neg)
               for t in sorted(set(pos + neg)))
    print(f"k={k:2d} attack_auc={auc(pos, neg):.3f} best_advantage={best:.3f}")

Expected output:

k= 1 attack_auc=0.600 best_advantage=0.200
k= 5 attack_auc=0.521 best_advantage=0.073
k=25 attack_auc=0.520 best_advantage=0.060

Tomás measured advantage and reply quality against the criteria agreed at scoping: advantage below 0.1 with less than 2 points quality loss. He reported both, including the quality cost. If a criterion is missed, say so and explain what would be needed to meet it. Use held-out cases for the final measurement, and include adaptive cases written by someone who did not build the defence. Report the numbers with intervals.

11.5 Handover

Deliver: the report; the evaluation harness and suites so tests can be re-run; configuration changes; runbooks; and a list of open risks with owners. A project that cannot be repeated after you leave has limited value.

Tomás handed over the harness, the deduplication job and a runbook for retesting after each fine-tune, so Corvane can repeat the measurement. Include named owners for open risks and a date for the first retest. Check that someone else can run the harness before you leave.

11.6 Worked example (synthetic Corvane case)

A student on placement, Tomás, takes Amira's question: does deduplicating customer messages before fine-tuning a reply-suggestion model reduce membership-inference leakage? Criteria: attack advantage below 0.1 with less than 2 points loss in reply quality. In the lab analogue, a model that effectively memorises its training records (k = 1) gives advantage 0.20; as it generalises (larger k) the advantage falls to about 0.06–0.07. On Corvane's isolated copy, deduplication cuts advantage from 0.17 to 0.06 with 1 point quality loss. He hands over the harness, the deduplication job and a runbook for retesting after each fine-tune.

The criteria were met.

11.7 Errors and limits

Each error limits the project's value. Without a decision owner, results have nowhere to go. Testing on production can harm customers and breaches authorisation. A report without the harness cannot be repeated after the next fine-tune, which is why Tomás handed over his harness and runbook. Settle all three in the project plan before work starts, and ask your supervisor to confirm them in writing.

Exercises

Synthesis

Applied projects serve a real decision, run safely under authorisation, measure against agreed criteria and leave behind tools the organisation can reuse.

Continue studying — chapter 11 of 12

← Chapter 10 (draft) · Course chapter index · Chapter 12 (draft) →