Chapter 37 — Application and software supply chain security: operations
Lesson map
Lesson 37 published. Map, example, EX109.
flowchart TD
N0["Learning outcomes"]
N1["37.1 Monitoring"]
N2["37.2 Alerts"]
N3["37.3 Patching"]
N4["37.4 Configuration"]
N5["37.5 On-call"]
N6["37.6 Working with operations"]
N7["Incident handover"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX109"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["37.1 Monitoring"]
N2["37.2 Alerts"]
N3["37.3 Patching"]
N4["37.4 Configuration"]
N5["37.5 On-call"]
N6["37.6 Working with operations"]
N7["Incident handover"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX109"]
[Learning outcomes]
|
v
[37.1 Monitoring]
|
v
[37.2 Alerts]
|
v
[37.3 Patching]
|
v
[37.4 Configuration]
|
v
[37.5 On-call]
|
v
[37.6 Working with operations]
|
v
[Incident handover]
|
v
[Worked example] --> [Exercise EX109]
Learning outcomes
You will (1) operate the application securely in production; (2) monitor for security events from the app and pipeline; (3) manage patching and configuration; (4) run on-call for security issues.
37.1 Monitoring
ticket-web logs authentication events, authorisation failures, input validation rejections and admin actions; the pipeline logs builds, signing and deployments. OWASP Top 10:2021 A09 highlights logging and monitoring failures.
The application logs show misuse of ticket-web itself, while pipeline logs show attempts to change what runs. Together they cover both ways an attacker might reach riders. Each log entry has a time, an account and an action, which lets investigators reconstruct events. Logs are sent to a central store that the application and pipeline accounts can write to but not edit or delete, so an intruder cannot erase evidence of their own actions.
37.2 Alerts
Alerts fire on unusual patterns: many authorisation failures from one account, a deployment not from the approved pipeline, or a signature verification failure.
Each alert matches a pattern that is rare in normal operation but common in attacks. Many authorisation failures suggest someone probing for other riders' tickets; a deployment not from the pipeline suggests someone bypassing controls; a verification failure may mean tampering. Alerts are tuned so that each one is worth a person's attention, because an on-call engineer woken often by noise starts to ignore real problems.
37.3 Patching
Servers rebuild weekly from updated base images; the application follows the remediation targets.
Weekly rebuilds mean server patches arrive steadily without anyone needing to remember them. Application fixes follow the same remediation targets as other findings: seven days for critical, thirty for high. Because servers are rebuilt rather than patched in place, every server matches its image exactly, and no server keeps an old library that a manual patch missed. The image used by each server is recorded, so the team can list every server still running an image with a known flaw.
37.4 Configuration
Configuration is stored as code and compared nightly with running systems; drift is reported.
Drift happens when someone changes a running system by hand, often to fix an urgent problem, and the change never reaches the code. The nightly comparison finds such changes within a day. Each one is either added to the code through review or reverted, so the stored configuration remains a true description of production. Repeated drift in one area suggests the pipeline is missing a needed feature.
37.5 On-call
A rota covers security alerts outside office hours, with playbooks.
Attacks do not keep office hours, and a deployment from outside the pipeline at night needs attention before the morning. The rota spreads the load so no one person is always on call, and each shift starts with a short handover. Playbooks give the first steps for each alert type, such as how to check a deployment's origin or freeze releases, so the engineer acts consistently even when tired. The rota is paid and limited in length to avoid burnout.
37.6 Working with operations
Security events are shared with Coastline's operations centre.
The operations centre watches network and server health for all Coastline services, and it may see the same event from another angle, such as unusual traffic during an alert on authorisation failures. Sharing events lets both teams join up the picture quickly. Coastline agrees which events go to which team, so nothing falls between them and nobody receives everything. A joint review each quarter checks whether the split still works and whether any events were missed or duplicated.
Incident handover
When an alert needs more than a quick check, the on-call engineer opens an incident record with the timeline, actions and owner, and hands over to the day team with a short summary. Records are kept so patterns across incidents can be studied later. Kwame notices that three of the year's alerts came from manual changes, which supports removing direct server access entirely and making the pipeline the only route to production.
Reviewing alerts
Every month the team reviews which alerts fired, which were real and which rules need tuning. Rules that never fire are tested to check they still work.
Access reviews
Production access lists are reviewed every quarter and unused access removed.
Worked example
An alert fires: a deployment to a staging server not from the pipeline. Investigation shows a developer copied files manually to debug. The server is rebuilt from the pipeline and the developer's direct access to staging servers is removed.
Synthetic case. Events are synthetic.
Errors and limits
Logs can contain personal data; collect what is needed and protect it.
Exercises
- EX109 — Logged events. List three app events logged for security. Rubric: 3 = authentication, authorisation failures, validation rejections, admin actions (three); 2 = two; 1 = one; 0 = none.
- EX110 — Drift. What is configuration drift and how is it found? Rubric: 3 = running differs from code; nightly comparison; 2 = partial; 1 = vague; 0 = none.
- EX111 — Manual deploy. What happened with the manual deployment alert? Rubric: 3 = developer copied files; rebuilt; access removed; 2 = partial; 1 = vague; 0 = none.
Synthesis
Operate with logs, alerts, drift detection and on-call aligned to the pipeline.
Continue studying — chapter 37 of 50
← Chapter 36 (draft) · Course chapter index · Chapter 38 (draft) →