◇ OpenCyber Academy
Technologist · Chapter 27 · English

Chapter 27 — Deployment and validation

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 27 — Deployment and validation

Lesson map

Lesson 27 published. Map, example, EX079.

flowchart TD
  N0["Learning outcomes"]
  N1["27.1 From lab to production"]
  N2["27.2 Validation set"]
  N3["27.3 Acceptance criteria"]
  N4["27.4 Staged rollout"]
  N5["27.5 Rights"]
  N6["27.6 Documentation"]
  N7["27.7 Rollback"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX079"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["27.1 From lab to production"]
  N2["27.2 Validation set"]
  N3["27.3 Acceptance criteria"]
  N4["27.4 Staged rollout"]
  N5["27.5 Rights"]
  N6["27.6 Documentation"]
  N7["27.7 Rollback"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX079"]

[Learning outcomes]
    |
    v
[27.1 From lab to production]
    |
    v
[27.2 Validation set]
    |
    v
[27.3 Acceptance criteria]
    |
    v
[27.4 Staged rollout]
    |
    v
[27.5 Rights]
    |
    v
[27.6 Documentation]
    |
    v
[27.7 Rollback]
    |
    v
[Worked example] --> [Exercise EX079]

Learning outcomes

You will (1) deploy a CTI script or process into production safely; (2) validate outputs against known cases; (3) plan monitoring after deployment; (4) document acceptance.

27.1 From lab to production

A script that worked on synthetic data now meets real feeds and logs. Deployment moves it to a managed server, under a service account with minimum rights, scheduled and monitored.

On Lena's laptop the matcher read a six-line synthetic log; in production it reads the full resolver logs every day, with real formats and real failures. Running it on a managed server means it keeps working when she is on leave, and scheduling and monitoring mean a failed run is noticed. The service account ties the script's actions to the job rather than to Lena personally.

27.2 Validation set

Before go-live, run the script on a validation set: a sample of real past indicators and logs where results are already known — matches that were confirmed, near misses that were not. Outputs must agree.

Known results make the comparison objective: if the manual process confirmed that a host looked up a fraud domain on a given day, the script must find the same match. Near misses that were judged not to match test the other direction, that the script does not raise alerts the analysts already dismissed. Disagreement in either direction stops go-live until the difference is explained.

27.3 Acceptance criteria

Written beforehand: no missed known matches; false-positive rate no worse than the manual process; runtime under ten minutes; failure alerts reach the team.

Writing the criteria before the run prevents them being shaped around whatever the script produces. Each criterion is measurable: missed known matches can be counted, the false-positive rate compared with the manual process, runtime timed and failure alerts tested. The worked example below shows the script meeting them, including a four-minute runtime against the ten-minute limit. Any criterion not met is recorded with the reason, and go-live waits until it is met or formally waived by the manager.

27.4 Staged rollout

Run in report-only mode for two weeks, producing outputs humans check but no automatic exports. Then enable exports.

In report-only mode the script's mistakes cost only the time to check them, because nothing reaches the DNS filter or other tools. Two weeks gives enough runs to see daily variation, such as busier Mondays. Exports are enabled only after the comparison is accepted, which in the worked example meant the manager's signature. During those two weeks Lena compares each day's report with the manual results, as she did for expiry flagging.

27.5 Rights

The service account can read logs and write to the store; it cannot change firewall rules. Least privilege limits damage if the script misbehaves.

The script needs to read logs and write matches; it has no reason to touch firewall rules. If a bug or a bad input made it misbehave, the worst it could do is write wrong entries to the store, which can be corrected. With firewall rights, the same bug could block legitimate services across the council.

27.6 Documentation

A deployment record: version, date, configuration, validation results, approver.

The record answers the questions that come up when something changes later: which version is running, how it is configured, how it was validated and who approved it. If a fault appears months afterwards, the record shows whether the configuration has drifted since acceptance. The approver's name ties the go-live to a decision by a named person. The record is stored with the change ticket, so operations staff supporting the job out of hours can find it. Each later change to the script or its configuration adds a new entry rather than overwriting the old one, keeping the full history.

27.7 Rollback

Keep the previous version ready; if outputs go wrong, switch back and investigate.

Keeping the previous version ready means switching back takes minutes rather than a rebuild. Investigating after the switch keeps protection running while the cause is found. The deployment record shows exactly which version to return to and how it was configured when it last worked. The rollback steps are written in the deployment record and tested once before go-live, so they are known to work. Who may decide to switch back is named as well, so operations can act out of hours without waiting for Lena.

27.8 Privacy in validation

Validation data comes from real logs, so it contains personal data. Keep it on the managed server, use only what is needed and delete the extract after acceptance.

Resolver logs show which staff machines looked up which names, so they reveal what people were browsing. Keeping the extract on the managed server avoids copies on laptops, and using only the fields the validation needs limits what is exposed. Deleting it after acceptance means the extract does not outlive its purpose.

Change approval

Deployment goes through the normal change process: description, risk, test evidence, rollback and approver. Treating CTI scripts as production software, rather than personal tools, means operations knows they exist and can support them out of hours. It also ensures someone other than the author can restart or disable the job safely.

Worked example

Lena's matching script runs report-only for two weeks. It finds every match the manual process found plus one extra, confirmed real by the supervisor. Runtime is four minutes. The manager signs acceptance and exports are enabled.

Synthetic case. The results are invented.

Errors and limits

Validation sets age; refresh them when data formats change.

Exercises

Synthesis

Deploy with least privilege, validate against known cases, roll out in stages, document acceptance and keep rollback ready.

Continue studying — chapter 27 of 30

← Chapter 26 (draft) · Course chapter index · Chapter 28 (draft) →