Chapter 6 — Data collection and analysis | Master's Systems, endpoints and hardware security

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 6 — Data collection and analysis

Lesson map

Lesson 06 published. Map, example, EX033.

flowchart TD
  N0["Learning outcomes"]
  N1["6.1 Main data"]
  N2["Lab — competing explanations"]
  N3["6.2 Reading results"]
  N4["6.3 Stratification"]
  N5["6.4 Settings pattern"]
  N6["6.5 Conclusion"]
  N7["Effect size and significance"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX033"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["6.1 Main data"]
  N2["Lab — competing explanations"]
  N3["6.2 Reading results"]
  N4["6.3 Stratification"]
  N5["6.4 Settings pattern"]
  N6["6.5 Conclusion"]
  N7["Effect size and significance"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX033"]

[Learning outcomes]
    |
    v
[6.1 Main data]
    |
    v
[Lab — competing explanations]
    |
    v
[6.2 Reading results]
    |
    v
[6.3 Stratification]
    |
    v
[6.4 Settings pattern]
    |
    v
[6.5 Conclusion]
    |
    v
[Effect size and significance]
    |
    v
[Worked example] --> [Exercise EX033]

Learning outcomes

You will (1) analyse drift across explanations; (2) interpret risk ratios and tests; (3) check overlap with stratification.

6.1 Main data

Six months for 600 laptops: 138 drifted (23%), consistent with Halvorsen's reports.

The match with Halvorsen's own quarterly reports, about a quarter of laptops, is a useful check: if Chloé's definition had produced a very different rate, either her coding or Halvorsen's reports would need investigating before any comparison. The 138 drifted laptops are split 66 among the 120 admin devices and 72 among the 480 others. Before any comparison, Chloé also checks the excluded replaced devices and the never-compliant laptops flagged at the first check, and reports both counts beside the 600, so the denominator for every later percentage is visible.

Lab — competing explanations

"""Compare drift rates across competing explanations with risk ratios and chi-square (stdlib only)."""
def test(name, a_drift, a_n, b_drift, b_n):
    p1, p2 = a_drift/a_n, b_drift/b_n
    tot = a_drift + b_drift; n = a_n + b_n
    chi = 0
    for d, m in ((a_drift, a_n), (b_drift, b_n)):
        for obs, exp in ((d, m*tot/n), (m-d, m*(n-tot)/n)): chi += (obs-exp)**2/exp
    print(f"{name}: {a_drift}/{a_n} ({p1*100:.1f}%) vs {b_drift}/{b_n} ({p2*100:.1f}%), RR {p1/p2:.2f}, chi2 {chi:.1f}")
test("local admin vs none", 66, 120, 72, 480)
test("remote vs office", 58, 200, 80, 400)
test("engineering dept vs other", 41, 150, 97, 450)

Run with python3 lab_explain.py. Verified output:

local admin vs none: 66/120 (55.0%) vs 72/480 (15.0%), RR 3.67, chi2 86.7
remote vs office: 58/200 (29.0%) vs 80/400 (20.0%), RR 1.45, chi2 6.1
engineering dept vs other: 41/150 (27.3%) vs 97/450 (21.6%), RR 1.27, chi2 2.1

6.2 Reading results

Local admin devices drifted 55.0% versus 15.0% (risk ratio 3.67, chi-square 86.7). Remote devices drifted 29.0% versus 20.0% (RR 1.45, chi-square 6.1). Engineering drifted 27.3% versus 21.6% (RR 1.27, chi-square 2.1, below the 3.84 threshold for p < 0.05 with one degree of freedom).

The three results line up with the predictions differently. The admin comparison is large and clear: 66 of 120 admin laptops drifted against 72 of 480 others. The remote comparison is smaller but passes the threshold. The engineering comparison does not, so Chloé reports it as no clear association rather than a weak effect, as the proposal committed her to do.

6.3 Stratification

Within non-admin devices, remote and office drift rates are close; within admin devices, both are high. This suggests the remote effect partly reflects admin rights.

This is the overlap Chloé planned for. The remote result in the main comparison, a risk ratio equal to 1.45, is not wrong, but most of it is explained by where the admin devices are: many admin users work remotely. Stratifying is what lets her say that remote working adds little once admin rights are considered, instead of recommending changes to remote policy refreshes that would not address the main cause.

6.4 Settings pattern

Admin devices drifted on many settings, matching Explanation A's prediction.

This is the second test planned in Chapter 4, separate from the risk ratios: Explanation A predicted many settings changing together on admin devices, which the seminar sample already hinted at when the laptop with two extra administrators also had SMB1 on and a 15-minute screen lock. The comparison lab reports drift per group, not per setting, so the settings pattern rests on the per-device records Chloé kept for this purpose. A reader who wants to check it needs those records, which is why the archive includes them in pseudonymised form alongside the extract.

6.5 Conclusion

Local admin rights are the strongest association; remote working adds little once admin rights are considered; department shows no clear association.

Each part of the conclusion carries its own evidence: a risk ratio equal to 3.67 and chi-square equal to 86.7 for admin rights; a remote effect equal to 1.45 that largely disappears within admin strata; and an engineering chi-square equal to 2.1, below the threshold. The wording stays associational, as in Chloé's research question, because the design observed existing records and changed nothing. That is why Halvorsen's follow-up targets the 120 admin devices rather than remote policy refreshes or engineering software.

Effect size and significance

A chi-square test says whether a difference is unlikely by chance; the risk ratio says how large it is. The admin difference is both large and clear. The remote difference is smaller and, after stratification, mostly explained by admin rights. Chloé reports both measures so readers can judge importance, not only significance.

She also notes that with 600 devices, even small differences can reach significance, so effect sizes matter more for decisions.

The finance installer

The fourth explanation is confirmed for one setting: finance laptops show screen lock drift after the installer runs. Halvorsen asks the vendor for a fix.

Robustness

Excluding devices with any missing field changes the admin risk ratio only slightly, supporting the main finding.

Reporting

Results go to Sven in a short memo.

Worked example

Sven asks what to do. Chloé suggests removing standing local admin rights and providing a request process for temporary elevation, then re-measuring.

Synthetic case. Synthetic.

Errors and limits

Association does not prove that removing admin rights will reduce drift; re-measure after change.

Exercises

EX033 — Overall drift. What overall drift rate was observed? Rubric: 3 = 138/600 (23%); 2 = partial; 1 = vague; 0 = none.
EX034 — Admin result. Report the admin comparison. Rubric: 3 = 55% vs 15%, RR 3.67, chi2 86.7; 2 = two; 1 = one; 0 = none.
EX035 — Department result. Why is the department result not significant? Rubric: 3 = chi2 2.1 below 3.84; 2 = partial; 1 = vague; 0 = none.
EX036 — Stratified finding. What did stratification show about remote work? Rubric: 3 = effect largely reflects admin rights; 2 = partial; 1 = vague; 0 = none.
EX037 — Recommendation. What did Chloé recommend? Rubric: 3 = remove standing admin, temporary elevation, re-measure; 2 = two; 1 = one; 0 = none.
EX038 — Caution. What caution applies? Rubric: 3 = association not proof; 2 = partial; 1 = vague; 0 = none.

Synthesis

Compare explanations directly and check overlap before concluding.

Continue studying — chapter 06 of 08

← Chapter 05 (draft) · Course chapter index · Chapter 07 (draft) →

Chapter 6 — Data collection and analysis