◇ OpenCyber Academy
Master's · Chapter 3 · English

Chapter 3 — Study design, ethics and reproducibility

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 3 — Study design, ethics and reproducibility

Lesson map

Lesson 03 published. Map, example, EX015.

flowchart TD
  N0["Learning outcomes"]
  N1["3.1 Choosing an observational design"]
  N2["3.2 Ethics and approval"]
  N3["3.3 Keyed pseudonyms"]
  N4["Lab — Keyed pseudonyms"]
  N5["3.4 Reading the pseudonyms"]
  N6["3.5 Reproducibility plan"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX015"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["3.1 Choosing an observational design"]
  N2["3.2 Ethics and approval"]
  N3["3.3 Keyed pseudonyms"]
  N4["Lab — Keyed pseudonyms"]
  N5["3.4 Reading the pseudonyms"]
  N6["3.5 Reproducibility plan"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX015"]

[Learning outcomes]
    |
    v
[3.1 Choosing an observational design]
    |
    v
[3.2 Ethics and approval]
    |
    v
[3.3 Keyed pseudonyms]
    |
    v
[Lab — Keyed pseudonyms]
    |
    v
[3.4 Reading the pseudonyms]
    |
    v
[3.5 Reproducibility plan]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX015]

Learning outcomes

You will (1) choose an observational design suited to existing certificate records; (2) pseudonymise hostnames with a keyed function; (3) plan for a study others can reproduce.

3.1 Choosing an observational design

Felipe cannot randomise certificates to renewal modes; Briarwood decided those years ago. He chooses a retrospective cohort design: every certificate active during the past two years is a member, its renewal mode is the exposure, and expiry incidents from the ticketing system are the outcome. He measures exposure time in certificate-years, so a certificate that existed for six months contributes half a year. He accepts the main weakness openly: services chosen for automation may differ from manual ones in age, team or importance, so differences may reflect those traits, not renewal mode.

3.2 Ethics and approval

The study touches no patient data, but the ethics committee still reviews it because hostnames and tickets can reveal Briarwood's internal architecture and staff names. Felipe's protocol says ticket text is read only on Briarwood premises, staff names are removed before export, and the exported dataset contains only pseudonymised host identifiers, renewal mode, dates and incident counts. Any currently expired or soon-expiring certificate he notices is reported to the Briarwood infrastructure lead on the same day, because a study of outages must not knowingly let one happen.

3.3 Keyed pseudonyms

Felipe needs stable pseudonyms: the same hostname must map to the same identifier across files, but outsiders must not be able to reverse it. A plain hash fails, because anyone can hash a guessed hostname such as portal.briarwood.example and compare. A keyed hash, HMAC from RFC 2104 with SHA-256, fixes this: only holders of the secret key can compute the mapping. Briarwood generates and keeps the key; Felipe receives only pseudonymised data. The lab uses a fixed demonstration key purely so the output is reproducible, and shows that a fresh random key from the secrets module produces unrelated pseudonyms.

Lab — Keyed pseudonyms

"""Keyed pseudonyms for hostnames with HMAC-SHA256 (RFC 2104); synthetic names, demo key only."""
import hmac, hashlib, secrets
HOSTS = ["portal.briarwood.example", "lab-results.briarwood.example", "portal.briarwood.example"]

def pseud(key, host):
    return "h-" + hmac.new(key, host.encode(), hashlib.sha256).hexdigest()[:10]

def main():
    fixed = b"study-demo-key-not-for-production"  # fixed so output is reproducible
    for h in HOSTS:
        print(f"{h:<33} -> {pseud(fixed, h)}")
    plain = "h-" + hashlib.sha256(HOSTS[0].encode()).hexdigest()[:10]
    print(f"unkeyed SHA-256 of first host -> {plain} (anyone can recompute)")
    fresh = secrets.token_bytes(32)
    print(f"fresh random key gives same pseudonym as demo key: {pseud(fresh, HOSTS[0]) == pseud(fixed, HOSTS[0])}")

if __name__ == "__main__":
    main()

Run with python3 lab_pseudo.py. Verified output:

portal.briarwood.example          -> h-6f292159ff
lab-results.briarwood.example     -> h-915f39cfa2
portal.briarwood.example          -> h-6f292159ff
unkeyed SHA-256 of first host -> h-80926435c3 (anyone can recompute)
fresh random key gives same pseudonym as demo key: False

3.4 Reading the pseudonyms

Both appearances of portal.briarwood.example map to h-6f292159ff, so records join correctly, while lab-results maps to h-915f39cfa2. The unkeyed SHA-256 of the first host yields h-80926435c3, which any outsider who guesses the name can recompute; that is why plain hashing is not pseudonymisation. With a fresh 32-byte key the pseudonym differs, confirming that the mapping depends on the secret. Felipe notes that truncating to ten hexadecimal characters is acceptable for 1,400 hosts, giving about a trillion possible values, but must be checked for collisions before joining tables.

3.5 Reproducibility plan

Felipe's plan lists what another researcher would need: the data dictionary, extraction queries run against Briarwood's inventory and ticketing systems, the analysis scripts with random seeds, the pseudonymised dataset and a log of every manual coding decision. Briarwood cannot publish the extraction queries, so he documents them in prose detailed enough to adapt elsewhere. He fixes the software environment to the Python standard library to avoid dependency drift. Dr. Solberg asks for a dry run: a classmate repeats the analysis from the package alone and reproduces every table.

Worked example

Felipe registers a retrospective cohort design measured in certificate-years, obtains ethics approval with same-day reporting of live expiries, and exports data with HMAC-SHA256 pseudonyms whose key stays at Briarwood.

Synthetic note. The hosts and demonstration key are invented; HMAC (RFC 2104) is real, and real keys must come from a managed secret store.

Errors and limits

Observational cohorts cannot rule out confounding; they can only measure and adjust for what was recorded. Pseudonyms protect against outsiders, not against anyone who obtains the key.

Exercises

Synthesis

An honest observational design, keyed pseudonyms held by the partner and a tested reproducibility package make a field study both ethical and checkable.

Continue studying — chapter 03 of 08

← Chapter 02 (draft) · Course chapter index · Chapter 04 (draft) →