Chapter 38 — Security governance, risk and compliance: validation
Lesson map
Lesson 38 published. Map, example, EX112.
flowchart TD
N0["Learning outcomes"]
N1["38.1 Verification and validation"]
N2["38.2 Sample sizes"]
N3["Lab — Exact upper bounds for session-record samples"]
N4["38.3 Conclusions"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX112"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["38.1 Verification and validation"]
N2["38.2 Sample sizes"]
N3["Lab — Exact upper bounds for session-record samples"]
N4["38.3 Conclusions"]
N5["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX112"]
[Learning outcomes]
|
v
[38.1 Verification and validation]
|
v
[38.2 Sample sizes]
|
v
[Lab — Exact upper bounds for session-record samples]
|
v
[38.3 Conclusions]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX112]
Learning outcomes
You will (1) distinguish verification from validation; (2) use attribute sampling to validate control operation; (3) choose sample sizes that support conclusions.
38.1 Verification and validation
Verification asks whether a control was built as specified; validation asks whether it achieves its purpose in real operation over time. Nadia verified the broker in the laboratory earlier. Now she must validate that the control operated effectively through a year, in the way internal auditors and service auditors examine operating effectiveness. AICPA's trust services criteria, used in SOC 2 reports, distinguish design from operating effectiveness in the same spirit, though her work is internal and not an attestation. The broker's lab tests verified design; a year of operation shows whether approval and recording actually happened in practice.
38.2 Sample sizes
Checking every vendor session is impractical, so Nadia samples session records and counts deviations, such as a session without approval. The lab computes exact upper ninety-five percent bounds. Twenty-five clean records give a bound of about eleven percent. Forty clean records give about seven percent. Forty records with one deviation give about eleven percent. Fifty-nine clean records give exactly five percent. If Ardwell's tolerable deviation rate is five percent, she needs at least fifty-nine records with no deviations. Prof. Brenner notes that one deviation in forty raises the bound to about eleven percent, so a single exception changes the conclusion sharply.
Lab — Exact upper bounds for session-record samples
"""Exact upper 95% bound on deviation rate for attribute samples (stdlib)."""
from math import comb
def cdf(k, n, p): return sum(comb(n, i)*p**i*(1-p)**(n-i) for i in range(k+1))
def upper(k, n):
lo, hi = 0.0, 1.0
for _ in range(60):
m = (lo+hi)/2
lo, hi = (m, hi) if cdf(k, n, m) > 0.05 else (lo, m)
return hi
for n, k in [(25, 0), (40, 0), (40, 1), (59, 0)]:
print(f"n={n} deviations={k}: upper bound {upper(k, n):.3f}")
Run with python3 lab_sample.py. Verified output:
n=25 deviations=0: upper bound 0.113
n=40 deviations=0: upper bound 0.072
n=40 deviations=1: upper bound 0.113
n=59 deviations=0: upper bound 0.050
38.3 Conclusions
Nadia samples fifty-nine sessions at random across the year and finds no deviations, so she concludes with ninety-five percent confidence that the deviation rate is at most five percent. She notes the conclusion covers approval only, not whether vendors did the right work. She also checks emergency sessions separately, because they follow a different approval path and their problems could be diluted in an overall sample. Prof. Brenner reminds the class that sampling supports bounded statements, never certainty, and that each stratum with its own path deserves its own sample.
Worked example
Nadia separates verification from validation, finds fifty-nine clean records needed for a five percent bound and validates approvals.
Synthetic case. Session data are synthetic; AICPA TSC is real; this is not an attestation.
Errors and limits
Random samples miss rare patterns. Validation covers only tested attributes.
Exercises
- EX112 — Validation meaning. What does validation ask that verification does not? Rubric: 3 = whether the control achieves its purpose in operation over time;; 2 = operation;; 1 = vague;; 0 = none.
- EX113 — Minimum sample. How many clean records give a 5% upper bound? Rubric: 3 = fifty-nine;; 2 = 59;; 1 = 25;; 0 = none.
- EX114 — Separate stratum. Why check emergency sessions separately? Rubric: 3 = they follow a different path and could hide problems;; 2 = strata;; 1 = vague;; 0 = none.
Synthesis
Validation uses sized samples to make bounded claims about controls in real operation.
Continue studying — chapter 38 of 50
← Chapter 37 (draft) · Course chapter index · Chapter 39 (draft) →