◇ OpenCyber Academy
Professional · Chapter 6 · English

Chapter 6 — AI and machine-learning security: operational workflow

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 6 — AI and machine-learning security: operational workflow

Lesson map

Lesson 06 published. Map, example, EX026.

flowchart TD
  N0["Learning outcomes"]
  N1["6.1 Workflow"]
  N2["6.2 Triage"]
  N3["6.3 Remediation tracking"]
  N4["6.4 Change management"]
  N5["6.5 Working with operations"]
  N6["6.7b Closing the loop with the reporter"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX026"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["6.1 Workflow"]
  N2["6.2 Triage"]
  N3["6.3 Remediation tracking"]
  N4["6.4 Change management"]
  N5["6.5 Working with operations"]
  N6["6.7b Closing the loop with the reporter"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX026"]

[Learning outcomes]
    |
    v
[6.1 Workflow]
    |
    v
[6.2 Triage]
    |
    v
[6.3 Remediation tracking]
    |
    v
[6.4 Change management]
    |
    v
[6.5 Working with operations]
    |
    v
[6.7b Closing the loop with the reporter]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX026]

Learning outcomes

You will (1) follow an AI-security workflow from intake to closure; (2) triage findings by severity; (3) track remediation; (4) manage changes to AI systems; and (5) work with operations teams.

6.1 Workflow

Intake (a new AI service, change or concern) → context and data flow → risk scenarios → tests → findings → remediation → verification → closure and records. Each step has an owner and a record.

At Meridian, a prompt edit is intake as much as a new service, as Sofia's re-test showed. Record each step's outcome in the tracker. Sofia's tracker has one row per step. A new AI service, a prompt edit or a handler's concern all start at intake, and nothing skips verification before closure. Records at each step let an auditor follow a finding from the first concern to its closure note.

6.2 Triage

Severity considers impact (money, data, people affected, legal), likelihood (how easy, how exposed), and detectability. Findings that let outsiders take actions on others' accounts or see their data are usually high or critical.

Sofia rated urgent callbacks via injected uploads and claim status disclosure as high, and long-message floods as medium. Write a reason for each rating, so the owner can see how impact and likelihood were judged. Ratings use the impact and likelihood scale agreed with the business, so that a high rating means the same thing across teams.

6.3 Remediation tracking

Each finding gets an owner, a due date by severity, a fix, a verification test and a closure note. Accepted risks need a named approver and review date.

At Meridian, each of Sofia's three findings has an owner and due date. Review open findings weekly and escalate any past due. Close a finding only after the verification test passes and the closure note says what changed. Overdue high findings go to the product owner's manager. The tracker shows each finding's status at a glance for the product owner.

6.4 Change management

Model updates, prompt edits, new document sources, new tools and configuration changes all alter behaviour. Route them through change management with a re-test before release.

At Meridian, a prompt edit a month after testing brought back finding 1, which only a re-test caught. Write which tests must be re-run for each kind of change, and keep the results with the change record. A model update usually calls for the full suite, while a new document source calls at least for the retrieval and injection tests on that source.

6.5 Working with operations

Agree how alerts reach you, how you reach on-call, and who can disable a tool in an emergency. Share runbooks.

"""Lab A: spotting extraction-style abuse in a synthetic API log.
Flags clients whose query volume or input diversity is anomalous."""
import random, statistics
rng = random.Random(21)
log = []
for c in range(12):
    for _ in range(rng.randint(40, 90)):
        log.append((f"client{c:02d}", round(rng.gauss(0, 1), 1)))
for i in range(600):                      # synthetic scraper: many, grid-like inputs
    log.append(("client99", round(-3 + 6*(i % 61)/60, 1)))
by = {}
for c, x in log: by.setdefault(c, []).append(x)
vol = {c: len(v) for c, v in by.items()}
med = statistics.median(vol.values()); mad = statistics.median(abs(v-med) for v in vol.values())
for c in sorted(by):
    z = (vol[c]-med)/(1.4826*mad); uniq = len(set(by[c]))/len(by[c])
    flag = "FLAG" if z > 3.5 else "ok"
    print(f"{c} queries={vol[c]:3d} robust_z={z:6.2f} distinct_ratio={uniq:.2f} {flag}")

Expected output:

client00 queries= 50 robust_z= -0.34 distinct_ratio=0.56 ok
client01 queries= 64 robust_z=  0.45 distinct_ratio=0.52 ok
client02 queries= 81 robust_z=  1.41 distinct_ratio=0.44 ok
client03 queries= 46 robust_z= -0.56 distinct_ratio=0.52 ok
client04 queries= 52 robust_z= -0.22 distinct_ratio=0.48 ok
client05 queries= 40 robust_z= -0.90 distinct_ratio=0.57 ok
client06 queries= 42 robust_z= -0.79 distinct_ratio=0.57 ok
client07 queries= 87 robust_z=  1.74 distinct_ratio=0.48 ok
client08 queries= 56 robust_z=  0.00 distinct_ratio=0.52 ok
client09 queries= 50 robust_z= -0.34 distinct_ratio=0.62 ok
client10 queries= 68 robust_z=  0.67 distinct_ratio=0.50 ok
client11 queries= 89 robust_z=  1.85 distinct_ratio=0.42 ok
client99 queries=600 robust_z= 30.58 distinct_ratio=0.10 FLAG

At Meridian, Sofia agrees with operations who can disable ClaimChat's callback tool if urgent bookings spike. Test the procedure in a drill before it is needed, and record how long it took. Runbooks should say how to disable the callback tool, who decides, how customers are told and how to re-enable it once a fix is verified. Practise disabling it in staging. Name a backup contact in case on-call cannot be reached.

6.6 Worked example (synthetic Meridian case)

Sofia's three findings: (1) urgent callbacks via injected uploads — high; (2) ClaimChat reveals claim status for any claim number given a postcode — high; (3) long-message floods — medium. Owners and due dates are set. A month later, the product team edits the system prompt; change management triggers a re-test, which shows finding 1 has returned because the edit removed an instruction the team had relied on. Sofia uses this to argue that the urgent-callback rule must move into the callback tool itself.

The regression shows why controls must live in tools.

6.7 Errors and limits

Each error lets problems return. A finding without an owner or due date drifts. A prompt edit outside change management can undo a fix without anyone noticing, as Meridian's edit did for finding 1. A control that lives only in the prompt can be removed by the next edit or overridden by injected text. Check your tracker and change process for all three.

6.7b Closing the loop with the reporter

When a claims handler reports odd ClaimChat behaviour, the security team tells them what was found and what changed. People keep reporting only when reports visibly lead somewhere, so the feedback step is part of the workflow, not a courtesy.

Record the feedback in the finding, with the date. Sofia's note to the handler who first raised the urgent callbacks named the finding, the tool rule and the re-test result, in plain language, without technical detail the handler did not need. A short note like this takes minutes and keeps reports coming.

Exercises

Synthesis

A clear workflow — intake to closure, with triage, tracking and change management — keeps AI-security work moving and prevents regressions.

Continue studying — chapter 06 of 12

← Chapter 05 (draft) · Course chapter index · Chapter 07 (draft) →