OpenCyber AcademyLesson map
Lesson 20 published. Map, example, EX058.
flowchart TD
N0["Learning outcomes"]
N1["20.1 Common breakages"]
N2["20.2 Method"]
N3["20.3 Example one"]
N4["20.4 Example two"]
N5["20.5 Rolling back"]
N6["20.6 Learning"]
N7["A shared guide"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX058"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["20.1 Common breakages"]
N2["20.2 Method"]
N3["20.3 Example one"]
N4["20.4 Example two"]
N5["20.5 Rolling back"]
N6["20.6 Learning"]
N7["A shared guide"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX058"]
[Learning outcomes]
|
v
[20.1 Common breakages]
|
v
[20.2 Method]
|
v
[20.3 Example one]
|
v
[20.4 Example two]
|
v
[20.5 Rolling back]
|
v
[20.6 Learning]
|
v
[A shared guide]
|
v
[Worked example] --> [Exercise EX058]
Learning outcomes
You will (1) diagnose failures caused by security controls; (2) restore service safely; (3) learn from incidents.
20.1 Common breakages
New controls can break workloads: Restricted rejects a pod, a network policy blocks a needed connection, or a narrowed role stops the CI bot. These breakages are expected, because each control removes something a workload may have relied on without anyone noticing. A container that always ran as root may write to folders only root can use; a workload that reached any service will fail once a default-deny policy is applied. Planning for breakage, rather than being surprised by it, is what lets enforcement continue without damaging trust.
20.2 Method
Priya follows a method: read the error, check recent changes, reproduce in staging, fix the smallest thing, record it. The error message usually names the rule that blocked the request, such as a Pod Security violation or a denied connection. Recent changes narrow the cause, because a working service that breaks after an enforcement change usually breaks because of it. Reproducing in staging lets the fix be tested safely, and recording it means the next team with the same symptom finds the answer in the guide.
20.3 Example one
After enforcing Restricted, the cron job fails to start because it writes to a root-owned folder. The fix is a writable volume, not relaxing the policy. The cron job had written temporary files into a folder inside the image owned by root, which worked only while the container ran as root. Under Restricted, the container runs as an ordinary user and the write fails, so the job exits at startup. Mounting a small writable volume for those files keeps the job working while it continues to run as non-root.
20.4 Example two
A network policy blocks reports from reaching the storage service. Adding the specific egress rule fixes it. The default-deny policy for the client namespace allowed the flows on Priya's list, but the reports workload's connection to the storage service had been recorded as a flow without a matching egress rule. Adding a rule that allows only that destination fixes the failure without opening other traffic. Priya also corrects the link between the flow list and the policies so the omission does not recur.
20.5 Rolling back
If a fix takes too long, the change is rolled back and retried later, keeping the client service up. Rolling back is planned before each change, so the previous configuration can be restored from the repository in minutes. A rollback is recorded with its reason, and the change returns to the plan with what was learned. Keeping the client's service up during a failed change matters more than finishing enforcement that day, because a lost booking morning damages trust in later security work.
20.6 Learning
Each breakage is added to a short guide for teams. Each entry in the guide records the symptom as the team saw it, the cause found and the fix applied, written in the words a developer would search for. The cron folder problem, the missing reports egress rule and the CI bot's config map permission each become entries. A team that meets the same error later can resolve it without waiting for the platform team. The guide lives in the configuration repository, so its history is reviewed like code.
A shared guide
The troubleshooting guide lists symptoms, likely causes and fixes, such as "pod rejected: runs as root" and its solution. Teams consult it before asking for help, and Priya updates it whenever a new case appears. After a month, questions to the platform team about security controls drop noticeably, and teams start contributing their own entries.
Change windows
Enforcement changes are made during agreed windows so clients know when breakage is possible.
Post-change review
After each enforcement change, the team checks error rates for a day to catch slower failures, such as a nightly job.
Escalation
If a breakage affects a client's live service for more than fifteen minutes, the on-call engineer escalates to Kwame.
Worked example
The CI bot fails after its role is narrowed. Priya reads the error: it needs to read one config map. She adds that permission only, rather than restoring cluster-admin.
Synthetic case. Failures are synthetic.
Errors and limits
Relaxing a control to fix a breakage is tempting; find the specific need.
Exercises
- EX058 — Method. List the troubleshooting steps.
Show marking criteria
Rubric: 3 = error, changes, reproduce, smallest fix, record; 2 = four; 1 = two; 0 = none. - EX059 — Cron fix. How was the cron failure fixed?
Show marking criteria
Rubric: 3 = writable volume; 2 = partial; 1 = vague; 0 = none. - EX060 — CI bot. What did the CI bot actually need?
Show marking criteria
Rubric: 3 = read one config map; 2 = partial; 1 = vague; 0 = none.
Synthesis
Fix the specific need instead of weakening the control.
Continue studying — chapter 20 of 30
← Chapter 19 (draft) · Course chapter index · Chapter 21 (draft) →