Chapter 11 — Maturity and continuous improvement
Lesson map
Lesson 11 published. Map, example, EX051.
flowchart TD
N0["Learning outcomes"]
N1["11.1 SOC-CMM"]
N2["11.2 Continuous monitoring"]
N3["11.3 Research"]
N4["11.4 Year-two plan"]
N5["11.5 Lessons loop"]
N6["Improvement backlog"]
N7["External views"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX051"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["11.1 SOC-CMM"]
N2["11.2 Continuous monitoring"]
N3["11.3 Research"]
N4["11.4 Year-two plan"]
N5["11.5 Lessons loop"]
N6["Improvement backlog"]
N7["External views"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX051"]
[Learning outcomes]
|
v
[11.1 SOC-CMM]
|
v
[11.2 Continuous monitoring]
|
v
[11.3 Research]
|
v
[11.4 Year-two plan]
|
v
[11.5 Lessons loop]
|
v
[Improvement backlog]
|
v
[External views]
|
v
[Worked example] --> [Exercise EX051]
Learning outcomes
You will (1) assess SOC capability with SOC-CMM; (2) apply continuous monitoring ideas from SP 800-137; (3) plan year-two improvements; (4) learn from research on SOC practice.
11.1 SOC-CMM
The SOC Capability Maturity Model (van Os, 2016) assesses business, people, process, technology and services. Priya's self-assessment: process improving, people thin (one analyst), technology reliant on the provider. The assessment shows where Fenwick's SOC is strong and where it depends on too few people. With one analyst, holidays and illness leave daytime triage uncovered, which is why the year-two plan adds a second analyst. Reliance on the provider is reasonable for a small company but needs the exit terms and ownership of logs and rules that the agreement provides. Priya repeats the self-assessment each year, and the people score is the first she expects to change once the second analyst starts.
11.2 Continuous monitoring
NIST SP 800-137 (Dempsey et al., 2011) frames monitoring as ongoing awareness feeding risk decisions, not a one-time project. For Fenwick, this means the SOC's figures feed decisions about risk, such as funding ERP log collection or MFA for vendor access, rather than being collected and filed. The monthly report and board page are the routine that keeps monitoring connected to decisions. Priya uses the publication to explain to the board why the SOC needs ongoing funding rather than a one-time project budget.
11.3 Research
Kokulu et al. (2019) interviewed SOC staff and found common issues between analysts and managers, such as disagreement on how to measure success; Priya uses this to agree metrics with Tom rather than imposing them. Agreeing metrics with Tom means they reflect what he sees in daily triage, such as the time wasted on benign macro alerts. He suggested measuring precision per rule rather than for all alerts together, which made tuning easier to target. Priya and Tom review the metrics each quarter and change them if they stop being useful.
11.4 Year-two plan
A second analyst, ERP and factory network monitoring, a quarterly rule review and a purple-team style validation with the provider using authorised test actions only. Each item addresses a gap found during the first year: a second analyst for the thin people score, ERP and factory network monitoring for remaining coverage gaps, quarterly rule reviews to keep rules working as systems change, and validation with the provider to test detections against realistic actions. Priya costs each item for the board, so the second analyst and the monitoring work can be compared with the risks they reduce.
11.5 Lessons loop
Each incident and test produces actions with owners and dates, tracked to closure. The loop makes sure that lessons become changes. After the March replay, legal's contract check became a playbook step; after the 26-minute test, the provider added a backup analyst. Each action is tracked in a register with its owner and date, and Priya reviews open actions at each monthly meeting. Actions not closed by their date are reported to the board with a reason, so lessons cannot be quietly dropped. The register goes into the board pack each month, so directors can see overdue actions such as the remaining tabletop finding.
Improvement backlog
All improvement ideas go into one backlog with priority, owner and date. The backlog is reviewed monthly with the provider. Items come from incidents, exercises, tests, analyst suggestions and the maturity assessment. Keeping a single list stops good ideas being lost in emails and lets Priya show the board how many actions were closed each quarter.
External views
Priya joins a regional manufacturers' security group, where members compare practices and share warnings. A member's report of phishing pages copying a common VPN login screen led Fenwick to warn staff a week before a similar email arrived. Outside views catch things an internal team, busy with daily work, would not notice.
Worked example
Tom suggests that his rule documentation be peer-reviewed by the provider. The provider finds two rules that would not fire on the new VPN product. Both are fixed before the old VPN is retired.
Synthetic case. Assessments are invented.
Errors and limits
Maturity scores are self-reported; external review helps.
Exercises
- EX051 — CMM domains. Name the five SOC-CMM domains. Rubric: 3 = business, people, process, technology, services; 2 = four; 1 = two; 0 = none.
- EX052 — Self-assessment. Summarise Fenwick's self-assessment. Rubric: 3 = process improving; people thin; tech provider-reliant; 2 = partial; 1 = vague; 0 = none.
- EX053 — Kokulu finding. How did Priya use Kokulu et al.? Rubric: 3 = agree metrics with Tom; avoid manager-analyst mismatch; 2 = partial; 1 = vague; 0 = none.
- EX054 — Year two. List three year-two items. Rubric: 3 = second analyst, ERP/factory monitoring, rule review, validation (three); 2 = two; 1 = one; 0 = none.
- EX055 — Peer review gain. What did provider peer review find? Rubric: 3 = two rules failing on new VPN; fixed; 2 = partial; 1 = vague; 0 = none.
Synthesis
Assess honestly, monitor continuously and close improvement actions.
Continue studying — chapter 11 of 12
← Chapter 10 (draft) · Course chapter index · Chapter 12 (draft) →