Chapter 14 — Quantum and post-quantum security: automation | Technical Quantum and post-quantum security

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 14 published. Map, example, EX040.

flowchart TD
  N0["Learning outcomes"]
  N1["14.1 Routine checks"]
  N2["14.2 Practical activity — Lab K"]
  N3["14.3 After patch windows"]
  N4["14.4 Maintainable scripts"]
  N5["14.6 Hands-on activity"]
  N6["14.7 Automation and scope"]
  N7["14.9 Alert quality"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX040"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["14.1 Routine checks"]
  N2["14.2 Practical activity — Lab K"]
  N3["14.3 After patch windows"]
  N4["14.4 Maintainable scripts"]
  N5["14.6 Hands-on activity"]
  N6["14.7 Automation and scope"]
  N7["14.9 Alert quality"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX040"]

[Learning outcomes]
    |
    v
[14.1 Routine checks]
    |
    v
[14.2 Practical activity — Lab K]
    |
    v
[14.3 After patch windows]
    |
    v
[14.4 Maintainable scripts]
    |
    v
[14.6 Hands-on activity]
    |
    v
[14.7 Automation and scope]
    |
    v
[14.9 Alert quality]
    |
    v
[Worked example] --> [Exercise EX040]

Learning outcomes

You will (1) automate routine checks of negotiation and configuration; (2) use confidence bounds to decide when a rollout stage is safe; (3) schedule checks after patches; and (4) keep scripts maintainable.

14.1 Routine checks

Schedule scripts to: run a client check against each in-scope service daily; parse logs for hybrid share; audit configuration exports for presence and preference. Send results to the dashboard and alert on changes.

At HHG, the nightly job against 14 endpoints is an example. Store results with dates so each run can be compared with the last, and make sure every target is in scope. Raise alerts only for changes that need action. Keep the scripts under version control with their target list, and review both together.

14.2 Practical activity — Lab K

# Lab K (ch 14): canary gate for enabling a hybrid group - promote only if failures stay within budget.
import math
def upper95(k, n):                      # Wilson upper bound
    z = 1.96; p = k / n; d = 1 + z*z/n
    return ((p + z*z/(2*n)) + z*math.sqrt(p*(1-p)/n + z*z/(4*n*n))) / d
baseline, budget = 5e-4, 4e-4            # baseline failure rate, allowed increase
for stage, k, n in [("1% canary", 3, 5000), ("10% canary", 28, 50000), ("10% canary (proxy site)", 96, 50000)]:
    ub = upper95(k, n); ok = ub <= baseline + budget
    print(f"{stage:24s} {k}/{n}  upper95={ub:.5f}  {'PROMOTE' if ok else 'HOLD'}")

Expected output:

1% canary                3/5000  upper95=0.00176  HOLD
10% canary               28/50000  upper95=0.00081  PROMOTE
10% canary (proxy site)  96/50000  upper95=0.00234  HOLD

When a change is rolled out in stages, the decision to move to the next stage should use the upper confidence bound on the failure rate, not just the observed rate. A small sample can look good by luck.

The script computes a Wilson upper bound on the failure rate at each stage and promotes only if it stays within the baseline plus budget. The first stage holds because 5,000 connections are too few to be confident, even though the observed rate is fine. The 10 % stage is promoted. The proxy site holds because its failure rate is clearly above budget, which points to a fault. Change the counts and see when each decision changes.

14.3 After patch windows

Run the full check set automatically after each patch window and compare with the previous run. Differences go to the on-call operator.

At HHG, this comparison found two endpoints that had switched from hybrid to classical after a load-balancer patch. Write the comparison so that a difference names the endpoint, the previous result and the new one, and include a link to the runbook. Keep the full output of each run. Check the output before the next patch window. Record who reviewed each difference.

14.4 Maintainable scripts

Version control, a README, clear inputs and outputs, error messages that say what went wrong, and a small test with known input. Avoid hard-coded passwords; use the approved secret store.

At HHG, the nightly check script is kept in version control with its target list and a test input. Review changes to scripts the same way as configuration changes, and record who maintains each script. Write error messages that say what went wrong and where, so the on-call operator can act without reading the code.

14.5 Worked example

HHG schedules a nightly job that runs openssl s_client checks against 14 staging and production endpoints in scope, extracts the negotiated group, and writes a CSV. A diff with the previous night flags two endpoints that changed from hybrid to classical after a load-balancer patch; the morning operator restores the profile.

The job found a regression that caused no errors, and the morning operator could act on it before guests or partners noticed. The diff made the change obvious without anyone reading 14 sets of output.

14.6 Hands-on activity

Write a script that reads a list of lab hosts, runs a TLS check on each, and prints host and negotiated group.

Read hosts from a file, one per line, and run openssl s_client against each with a timeout. Extract the negotiated group from the output, print host and group, and mark a host as unknown if the check fails. Print the number of hosts read. Test it on your lab hosts only. Add a small test file with expected output, so a colleague can check the script after changing it.

14.7 Automation and scope

Automated checks are still checks: they need to be inside written scope, run at agreed times and stop if a target behaves unexpectedly. Keep the target list in the repository, reviewed like code, so nobody adds an out-of-scope host by accident.

At HHG, the 14 endpoints in the nightly job are in the reviewed target list, each with its scope reference. Review the target list whenever scope changes, and remove hosts that are no longer in scope. Make the job stop and alert if a target responds unexpectedly.

14.8 Common errors and limits

Promoting a stage on a small, lucky sample.
Scripts with embedded secrets.
Unreviewed target lists.

Each error creates risk. Promoting a stage on a small sample can let a fault reach all users, which is why Lab K uses an upper bound. Secrets embedded in scripts leak through version control and logs. An unreviewed target list can add an out-of-scope host, and the job will then test it every night. Check your automation for all three before scheduling it. Fix them first.

14.9 Alert quality

An alert that fires every night for a known, accepted exception teaches operators to ignore alerts. Suppress known exceptions explicitly, with an expiry date that matches the exception record, and review suppressions monthly. Each alert should link to its runbook so the person woken at 03:00 knows what to do first.

At HHG, Summit's interim listener should be suppressed only until its exception expires. Record each suppression with its reason and owner, and remove it when the exception closes. Count alerts that needed no action each month.

Exercises

EX040 — Stage decision. Using Lab K, explain why the first canary is held. Rubric: 3 = upper bound above budget due to small sample.
EX041 — Diff alert. Describe how a nightly diff detects silent fallback. Rubric: 3 = compare negotiated group per host, alert on hybrid→classical.
EX042 — Script hygiene. List four features of a maintainable check script. Rubric: 3 = version control, README, clear errors, test.

Synthesis

Automation runs routine checks, gates rollout stages with confidence bounds, re-checks after patches and stays within scope and maintainable.

Continue studying — chapter 14 of 18

← Chapter 13 (draft) · Course chapter index · Chapter 15 (draft) →