<a id="ch-29"></a>

## Chapter 29 — Identity and access management: measurement

<!-- open-cyber-academy:lesson-figures:start ch-29 -->
### Lesson map

Lesson 29 published. Map, example, EX085.

```mermaid
flowchart TD
  N0["Learning outcomes"]
  N1["29.1 What to measure"]
  N2["Lab — Leaver removal times"]
  N3["29.2 Reading the result"]
  N4["29.3 Investigating the outlier"]
  N5["29.4 Honest reporting"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX085"]
```

### Logical drawing

```mermaid
flowchart LR
  N0["Learning outcomes"]
  N1["29.1 What to measure"]
  N2["Lab — Leaver removal times"]
  N3["29.2 Reading the result"]
  N4["29.3 Investigating the outlier"]
  N5["29.4 Honest reporting"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX085"]
```

```text
[Learning outcomes]
    |
    v
[29.1 What to measure]
    |
    v
[Lab — Leaver removal times]
    |
    v
[29.2 Reading the result]
    |
    v
[29.3 Investigating the outlier]
    |
    v
[29.4 Honest reporting]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX085]
```
<!-- open-cyber-academy:lesson-figures:end ch-29 -->


### Learning outcomes

You will (1) measure lifecycle and authentication outcomes; (2) compare with baseline using robust summaries; (3) investigate outliers.

### 29.1 What to measure

Measurement follows the success measures agreed in chapter 20: share of applications on single sign-on, share of staff using phishing-resistant authentication, time from HR leaver date to access removal, accounts with no matching employee and password tickets per hundred staff. Each is collected the same way as the baseline, so differences reflect real change rather than a new counting method. Sana documents every query and script used, and stores results with dates, so the comparison can be repeated by the council after her project ends.

### Lab — Leaver removal times

```python
"""Measure synthetic leaver-removal times (hours) before and after automation: median and 90th percentile (stdlib only)."""
import statistics as st
before = [312, 456, 98, 720, 504, 240, 600, 168, 432, 288, 552, 384]
after = [1.5, 2.0, 0.8, 3.5, 1.2, 26.0, 2.2, 1.0, 1.8, 2.6, 0.9, 1.4]
def p90(v): s = sorted(v); return s[int(0.9*(len(s)-1))]
for n, v in [("before", before), ("after", after)]:
    print(f"{n:6} n={len(v)} median_h={st.median(v):.1f} p90_h={p90(v):.1f} over_4h={sum(x > 4 for x in v)}")
```

Run with `python3 lab_review.py`. Verified output:

```text
before n=12 median_h=408.0 p90_h=552.0 over_4h=12
after  n=12 median_h=1.6 p90_h=2.6 over_4h=1
```

### 29.2 Reading the result

In the synthetic sample of twelve leavers, the median time from HR end date to removal falls from 408 hours, about seventeen days, to 1.6 hours. The 90th percentile falls from 552 hours to 2.6 hours. Before automation, all twelve exceeded the four-hour requirement; after, only one did. Sana reports medians and percentiles rather than means because one slow case would distort an average heavily. She also notes the small sample and continues collecting, since twelve leavers cannot show how often rare delays occur.

### 29.3 Investigating the outlier

The one leaver taking twenty-six hours is investigated rather than ignored. Logs show that HR entered the leaver on a Friday evening, the connector to one legacy application failed on a locked account record, and the retry succeeded only on Saturday night. The failure was logged and alerted, but nobody was on call. Sana records two actions: fix the connector's handling of locked records and agree weekend alert handling for leaver failures. Outliers often reveal exactly the weaknesses that averages hide.

### 29.4 Honest reporting

Results go to directors with baseline, target and current value for each measure, plus a short explanation of uncertainty and the outlier. Sana avoids presenting the improvement as complete: some applications remain outside connectors, and accounts in those still depend on manual removal. Honest reporting builds trust, and when a later measurement looks worse, directors understand why instead of losing confidence. It also keeps attention on the remaining work rather than on celebrating the parts that went well. The same format is used every quarter.

### Worked example

Median leaver removal falls from 408 to 1.6 hours and the 90th percentile from 552 to 2.6 hours; one 26-hour outlier reveals a connector bug and a weekend alerting gap.

> **Synthetic case.** Leavers, times and logs are synthetic.

### Errors and limits

Reporting a mean removal time lets one slow case distort the picture in either direction.

### Exercises

- **EX085 — Median change.** How did the median removal time change in the lab? *Rubric:* 3 = 408 to 1.6 hours;; 2 = large fall;; 1 = vague;; 0 = none.
- **EX086 — Over-requirement count.** How many leavers exceeded four hours after automation? *Rubric:* 3 = one of twelve;; 2 = one;; 1 = vague;; 0 = none.
- **EX087 — Outlier causes.** What two actions came from the outlier? *Rubric:* 3 = fix locked-record handling; agree weekend alert handling;; 2 = one;; 1 = vague;; 0 = none.

### Synthesis

Consistent measurement, robust summaries and investigated outliers show real improvement and the remaining gaps.

### Continue studying — chapter 29 of 50

[← Chapter 28 (draft)](#ch-28) · [Course chapter index](../SUBJECTS.md) · [Chapter 30 (draft) →](#ch-30)

## Worked answers

<a id="answer-ex085"></a>
**EX085.** From 408 hours to 1.6 hours.

<a id="answer-ex086"></a>
**EX086.** One of twelve.

<a id="answer-ex087"></a>
**EX087.** Fix the connector's locked-record handling and agree weekend handling of leaver alerts.