Chapter 5 — Security governance, risk and compliance: tools and interpretation

Lesson map

Lesson 05 published. Map, example, EX021.

flowchart TD
  N0["Learning outcomes"]
  N1["5.1 Self-assessment"]
  N2["5.2 The comparison"]
  N3["Lab — Self-assessment versus independent review"]
  N4["5.3 Interpreting"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX021"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["5.1 Self-assessment"]
  N2["5.2 The comparison"]
  N3["Lab — Self-assessment versus independent review"]
  N4["5.3 Interpreting"]
  N5["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX021"]

[Learning outcomes]
    |
    v
[5.1 Self-assessment]
    |
    v
[5.2 The comparison]
    |
    v
[Lab — Self-assessment versus independent review]
    |
    v
[5.3 Interpreting]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX021]

Learning outcomes

You will (1) use control self-assessment results; (2) compare them with independent review; (3) interpret gaps carefully.

5.1 Self-assessment

Marston asks control owners to rate their controls each quarter from zero, not in place, to four, fully effective and evidenced. Lena learns that self-assessment is cheap and engages owners, but tends to be optimistic. To test this, she compares owners' ratings with an independent review by internal audit for four control domains: access management, change management, backups and vendor management. Both used the same rating scale and definitions. Using identical definitions means any gap between the two sets of ratings reflects judgement, not different scales.

5.2 The comparison

The lab computes average ratings. Access management self-rates at three point two five against two point five from review, a gap of three quarters of a point. Change management self-rates two point six seven against one point six seven, a gap of one. Backups self-rate four against three point five. Vendor management self-rates two point six seven against one point three three, the largest gap at one point three three. Every domain rates itself higher than the review. The pattern is consistent, but the size of each gap is what directs Lena's attention in the next section.

Lab — Self-assessment versus independent review

"""Control self-assessment: mean rating (0-4) per domain and the gap to independent review."""
self_r = {"access": [3, 3, 4, 3], "change": [3, 2, 3], "backup": [4, 4], "vendor": [3, 3, 2]}
review = {"access": [2, 3, 3, 2], "change": [2, 1, 2], "backup": [4, 3], "vendor": [1, 2, 1]}
for d in self_r:
    s = sum(self_r[d])/len(self_r[d]); r = sum(review[d])/len(review[d])
    print(f"{d}: self {s:.2f} review {r:.2f} gap {s-r:+.2f}")

Run with python3 lab_csa.py. Verified output:

access: self 3.25 review 2.50 gap +0.75
change: self 2.67 review 1.67 gap +1.00
backup: self 4.00 review 3.50 gap +0.50
vendor: self 2.67 review 1.33 gap +1.33

5.3 Interpreting

Lena does not conclude that owners are dishonest. She asks why gaps differ. Vendor management owners rated controls as working because contracts include security clauses, while reviewers found that nobody checks supplier compliance. Backups have the smallest gap because restore tests produce clear evidence. She recommends tightening rating definitions to require evidence and focusing independent review on domains with the largest gaps, starting with vendor management. Requiring evidence for a rating of four would, for example, stop vendor management claiming effectiveness from contract clauses alone.

Worked example

Lena compares self-ratings with independent review, finds all domains overrated with vendor management worst at 1.33 and explains why gaps differ.

Synthetic case. Ratings are synthetic.

Errors and limits

Small samples of controls. Reviewers can be strict.

Exercises

• EX021 — Self-assessment bias. What is the typical bias of self-assessment?
  Show marking criteria

  Rubric: 3 = optimism;; 2 = optimistic;; 1 = pessimistic;; 0 = none.
• EX022 — Largest gap. Which domain had the largest gap?
  Show marking criteria

  Rubric: 3 = vendor management, +1.33;; 2 = vendor;; 1 = backup;; 0 = none.
• EX023 — Smallest gap. Why did backups have the smallest gap?
  Show marking criteria

  Rubric: 3 = restore tests produce clear evidence;; 2 = evidence;; 1 = vague;; 0 = none.
• EX024 — Vendor cause. Why did vendor owners overrate their controls?
  Show marking criteria

  Rubric: 3 = contracts had clauses but nobody checked compliance;; 2 = assumption;; 1 = vague;; 0 = none.
• EX025 — Recommendation. What did Lena recommend?
  Show marking criteria

  Rubric: 3 = evidence-based rating definitions and review focus on largest gaps;; 2 = two;; 1 = vague;; 0 = none.

Synthesis

Self-assessment engages owners, but independent review shows where optimism hides weakness.

Continue studying — chapter 05 of 12

← Chapter 04 (draft) · Course chapter index · Chapter 06 (draft) →