Chapter 13 — Cloud and container security: troubleshooting
Lesson map
Lesson 13 published. Map, example, EX049.
flowchart TD
N0["Learning outcomes"]
N1["13.1 After the block"]
N2["13.2 Method"]
N3["13.3 Finding"]
N4["13.4 Another case"]
N5["13.5 Rollback rule"]
N6["13.6 Guide"]
N7["Communicating breakages"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX049"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["13.1 After the block"]
N2["13.2 Method"]
N3["13.3 Finding"]
N4["13.4 Another case"]
N5["13.5 Rollback rule"]
N6["13.6 Guide"]
N7["Communicating breakages"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX049"]
[Learning outcomes]
|
v
[13.1 After the block]
|
v
[13.2 Method]
|
v
[13.3 Finding]
|
v
[13.4 Another case]
|
v
[13.5 Rollback rule]
|
v
[13.6 Guide]
|
v
[Communicating breakages]
|
v
[Worked example] --> [Exercise EX049]
Learning outcomes
You will (1) diagnose failures after security changes; (2) fix the specific cause; (3) keep services running.
13.1 After the block
Once the account-level public block is on, a marketing tool fails to load a brochure from the manifests account. Tomás reads the error: access denied. The tool had been reading the brochure through a public link to the main account, which the block now refuses. Access denied is exactly what the block should produce, so the error confirms the control works. The question is whether the brochure belonged there, not whether the block should be relaxed. Tomás records the time, the tool and the exact error text before changing anything.
13.2 Method
Read the error, list recent changes, reproduce in staging, find the smallest correct fix, record it. Applied to the brochure, the recent change was the account-level block, staging reproduced the same access-denied error, and the smallest fix was moving one file rather than altering any setting. Recording the case means the next person who sees the error finds the answer quickly. Each step is short, but skipping one tends to produce the broad fix that reopens the risk. The same method applies to the CI deploy failure that follows.
13.3 Finding
The brochure was stored in the wrong bucket. Moving it to the assets account fixes the problem without weakening the block. The brochure is public marketing material, so it belongs with the website assets that are meant to be public. Moving it took one file copy and a change to the tool's link. The block stayed on for every other bucket, and the fix was recorded in the troubleshooting guide with a link to the change. The marketing team confirms the tool loads the brochure from its new location.
13.4 Another case
The CI deploy pipeline fails after its permissions are narrowed: it needs to clear a cache path in site-assets. A single extra permission solves it. The failure appeared because the narrowed CI deploy identity could upload to site-assets but not delete cached files there. Granting delete on that one path, not on the whole bucket, restores the deployment. Tomás checks the permission in staging first and records why it is needed, so a later review does not remove it again by mistake.
13.5 Rollback rule
If a fix is not found within an hour during business hours, the change is reversed and retried later. The rule balances keeping a control in place against keeping Fairway's services running. An hour is long enough for most fixes like the brochure, which took twenty minutes, but short enough that a broken service does not stay broken all afternoon. A reversed change is not abandoned; it is retried after the cause is found in staging. Both March breakages were fixed well within the hour, so no rollback was needed.
13.6 Guide
Each case goes into a short troubleshooting guide. The guide entries for March cover the brochure, which was in the wrong account, and the CI deploy cache path, which needed one extra permission. Each entry gives the symptom someone would see, the cause and the fix, with a link to the change record. A colleague who meets access denied after the next control change can check the guide before asking for a rollback. Entries are added within a day of each breakage while the details are fresh.
Communicating breakages
When a security change breaks something, Tomás posts in the team channel what broke, why, and the fix. Being open about breakages builds trust; teams learn that security changes are handled carefully and that reporting problems gets them fixed quickly.
He also records how long each breakage lasted. Both cases in March were fixed within thirty minutes.
Testing before changes
Most breakages could have been found in staging if staging had matched production. Tomás adds the marketing tool to staging tests.
After action
Each breakage gets a short note in the guide within a day.
Metrics
Breakage counts are tracked monthly.
Worked example
A colleague suggests turning the block off "just for today". Tomás explains the risk and finds the misplaced brochure within twenty minutes.
Synthetic case. Synthetic.
Errors and limits
Disabling a control to fix a symptom often reintroduces the original risk.
Exercises
- EX049 — Brochure fix. What fixed the brochure error? Rubric: 3 = moving it to assets account; 2 = partial; 1 = vague; 0 = none.
- EX050 — CI need. What did CI deploy need? Rubric: 3 = clear a cache path in site-assets; 2 = partial; 1 = vague; 0 = none.
- EX051 — Rollback rule. State the rollback rule. Rubric: 3 = reverse if no fix in an hour; 2 = partial; 1 = vague; 0 = none.
- EX052 — Colleague suggestion. Why not turn the block off for a day? Rubric: 3 = reintroduces the risk; 2 = partial; 1 = vague; 0 = none.
Synthesis
Fix the specific cause and keep controls on.
Continue studying — chapter 13 of 18
← Chapter 12 (draft) · Course chapter index · Chapter 14 (draft) →