Chapter 10 — Quantum and post-quantum security: quality and troubleshooting
Lesson map
Lesson 10 published. Map, example, EX046.
flowchart TD
N0["Learning outcomes"]
N1["10.1 Data quality"]
N2["10.2 Workflow failures"]
N3["10.3 Practical activity — Lab K"]
N4["10.4 Coordinating troubleshooting"]
N5["10.5 Lessons learned"]
N6["10.7 Practice task"]
N7["10.8 Measuring quality"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX046"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["10.1 Data quality"]
N2["10.2 Workflow failures"]
N3["10.3 Practical activity — Lab K"]
N4["10.4 Coordinating troubleshooting"]
N5["10.5 Lessons learned"]
N6["10.7 Practice task"]
N7["10.8 Measuring quality"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX046"]
[Learning outcomes]
|
v
[10.1 Data quality]
|
v
[10.2 Workflow failures]
|
v
[10.3 Practical activity — Lab K]
|
v
[10.4 Coordinating troubleshooting]
|
v
[10.5 Lessons learned]
|
v
[10.7 Practice task]
|
v
[10.8 Measuring quality]
|
v
[Worked example] --> [Exercise EX046]
Learning outcomes
You will (1) check the quality of the inventory and tracker; (2) detect common workflow failures; (3) interpret rollout gate decisions; (4) coordinate troubleshooting with technical teams; and (5) improve the workflow from lessons learned.
10.1 Data quality
Check regularly: stale evidence (older than the agreed age), items without owners, unknowns without tasks, items marked Verified without evidence, exceptions past expiry. Sample ten items each month and check their evidence yourself.
Record the result of each sample, including items that passed, so trends become visible. At MTI the broker portal regression was found this way. When a sample fails, fix the item, then ask whether the same failure could affect similar items and check those too. Keep the sampling method the same each month so results can be compared over time.
10.2 Workflow failures
Silent fallback: an item verified last quarter has reverted to classical after an update. Stuck items: dependencies with no follow-up. Scope creep: the workflow absorbs unrelated work. Report drift: definitions change between quarters without notice.
Each failure has a different remedy: re-verification for silent fallback, escalation for stuck items, a scope review for creep and a definitions log for drift. Check for all four at the monthly meeting. At MTI, the broker portal regression was a silent fallback, caught by sampling. Record each failure found and the remedy applied.
10.3 Practical activity — Lab K
# Lab K (ch 10): canary gate for enabling a hybrid group - promote only if failures stay within budget.
import math
def upper95(k, n): # Wilson upper bound
z = 1.96; p = k / n; d = 1 + z*z/n
return ((p + z*z/(2*n)) + z*math.sqrt(p*(1-p)/n + z*z/(4*n*n))) / d
baseline, budget = 5e-4, 4e-4 # baseline failure rate, allowed increase
for stage, k, n in [("1% canary", 3, 5000), ("10% canary", 28, 50000), ("10% canary (proxy site)", 96, 50000)]:
ub = upper95(k, n); ok = ub <= baseline + budget
print(f"{stage:24s} {k}/{n} upper95={ub:.5f} {'PROMOTE' if ok else 'HOLD'}")
Expected output:
1% canary 3/5000 upper95=0.00176 HOLD
10% canary 28/50000 upper95=0.00081 PROMOTE
10% canary (proxy site) 96/50000 upper95=0.00234 HOLD
Technical teams roll out changes in stages and promote only when failure rates stay within budget with confidence. As coordinator, read gate results so you can report status accurately: "held for more data" and "held because of a fault" mean different things for the timeline.
Read the three results carefully. The first canary held because 5,000 connections are too few to show that failures are within budget, so more data is needed. The 10 % canary was promoted. The proxy site held because failures were clearly high, which points to a fault that needs technical diagnosis.
10.4 Coordinating troubleshooting
When a rollout causes problems, your role is coordination: make sure the right owners are engaged, the impact is understood, communication goes out, the tracker shows the item as blocked with a reason, and lessons are captured. Leave technical diagnosis to technical staff.
At MTI, when the quote API's hybrid share dropped, the coordinator's job was to engage the platform owner, inform stakeholders and update the tracker, not to diagnose the cause. Keep a short timeline of events and decisions.
10.5 Lessons learned
After each quarter and each significant incident, hold a short review: what went well, what didn't, what to change in the workflow. Record changes to definitions and procedures.
Keep the review short and blame-free, and focus on the workflow rather than on individuals. At MTI, the broker portal regression led to a new rule that every Verified item is re-checked monthly. Each lesson should produce a concrete change with an owner, or a recorded decision not to change. Share the review notes with the team.
10.6 Worked example
A monthly sample finds that the broker portal, marked Verified in Q1, shows 4 % hybrid in Q2 logs. Cause: a platform update restored a default profile. Actions: item moved back to In progress; owner restores profile; a quality rule is added — every Verified item is re-checked automatically each month. Lessons recorded.
The response follows the workflow: the tracker shows the true state, the owner fixes the cause, and a rule prevents a repeat. Reporting the regression openly protects the credibility of the other Verified figures.
10.7 Practice task
Design a monthly quality checklist of eight checks for MTI's tracker.
Include checks for stale evidence, items without owners, unknowns without tasks, Verified items without negotiation evidence and exceptions past expiry. Add checks of your own, such as the monthly re-check of Verified items introduced after the broker portal regression. For each check, say what data it uses, who runs it and what happens when it fails. Keep the checklist to one page so it is used, and review it after each quarter.
10.8 Measuring quality
Report a quality measure alongside progress: for example, the share of sampled items whose evidence matched their tracker state. A falling quality score is an early warning that progress figures may be overstated.
At MTI the score would have fallen in the month the broker portal regression was found. Report the score with the sample size, since a sample of ten items gives only a rough figure. If the score falls, look at which kinds of items failed before drawing conclusions, and say in the report how the figures may be affected.
10.9 Common errors and limits
- Trusting Verified forever.
- Coordinators doing technical diagnosis.
- Definitions changed silently.
Each error lets the numbers drift from reality. MTI's broker portal showed that Verified items can revert after an update. A coordinator who diagnoses technical faults delays the people who can fix them and may misreport the cause. Definitions changed silently make one quarter's figures incomparable with the next, so trends become meaningless. Add a check for each to your monthly quality checklist. Fix any you find. Record the fixes.
Exercises
- EX046 — Quality checks. List five data-quality checks. Rubric: 3 = stale, ownerless, untasked unknowns, unevidenced Verified, expired exceptions.
- EX047 — Gate reading. Report Lab K's three results in plain language for the sponsor. Rubric: 3 = wait for data; proceed; fault at one site under investigation.
- EX048 — Regression. What tracker change and rule follow the broker-portal regression? Rubric: 3 = back to In progress; monthly re-check.
- EX049 — Coordination. List four coordinator actions during a rollout problem. Rubric: 3 = owners, impact, communication, tracker.
- EX050 — Quality measure. Define a quality measure and its calculation. Rubric: 3 = matched/sampled share.
Synthesis
Quality checks, accurate reading of gate decisions, coordinated troubleshooting and lessons learned keep the workflow's numbers honest.
Continue studying — chapter 10 of 12
← Chapter 09 (draft) · Course chapter index · Chapter 11 (draft) →