Chapter 5 — Proposal and pilot
Lesson map
Lesson 05 published. Map, example, EX027.
flowchart TD
N0["Learning outcomes"]
N1["5.1 The proposal"]
N2["5.2 Pilot extraction"]
N3["Lab — bucket classification"]
N4["5.3 Lab result"]
N5["5.4 Revising measures"]
N6["5.5 CIS and CCM"]
N7["Proposal feedback"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX027"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["5.1 The proposal"]
N2["5.2 Pilot extraction"]
N3["Lab — bucket classification"]
N4["5.3 Lab result"]
N5["5.4 Revising measures"]
N6["5.5 CIS and CCM"]
N7["Proposal feedback"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX027"]
[Learning outcomes]
|
v
[5.1 The proposal]
|
v
[5.2 Pilot extraction]
|
v
[Lab — bucket classification]
|
v
[5.3 Lab result]
|
v
[5.4 Revising measures]
|
v
[5.5 CIS and CCM]
|
v
[Proposal feedback]
|
v
[Worked example] --> [Exercise EX027]
Learning outcomes
You will (1) write the research proposal; (2) pilot the data extraction on a small sample; (3) check measures work; (4) revise before the main collection.
5.1 The proposal
Twelve pages: background, question, explanations and predictions, design, data, ethics, analysis, risks and timeline. Each prediction maps to an analysis. Mapping each prediction to an analysis lets Dr Tanaka check that nothing in the proposal is untestable. The risks part names the main threats: missing origin labels, an incomplete register of intended public resources and the possibility that Larkspur cannot export a full year. The timeline leaves time to rerun the analysis after examination corrections. Her sign-off came before any pilot data existed.
5.2 Pilot extraction
Lucas extracts one month of synthetic-format records from a test account and classifies current bucket settings to check that the classification works. The pilot uses a test account so that no production records are involved before the data agreement is signed. It checks that Lucas's scripts read the export format, that origin labels are present, and that bucket settings can be classified consistently. Problems found here can still change the method without biasing results. No customer data sit in the test account.
Lab — bucket classification
"""Classify synthetic storage bucket settings against a simple baseline (stdlib only)."""
buckets = [ # name, public, encrypted, access_logging, origin
("orders-archive", False, True, True, "iac"), ("marketing-assets", True, True, False, "console"),
("tmp-export-0412", True, False, False, "console"), ("ml-features", False, True, False, "iac"),
("receipts", False, True, True, "iac"), ("partner-drop", True, True, True, "iac"),
]
for n, pub, enc, log, origin in buckets:
issues = [x for x, bad in (("public", pub), ("unencrypted", not enc), ("no access logs", not log)) if bad]
print(f"{n:17} {origin:8} {', '.join(issues) or 'ok'}")
print("public buckets:", sum(b[1] for b in buckets), "of", len(buckets))
Run with python3 lab_buckets.py. Verified output:
orders-archive iac ok
marketing-assets console public, no access logs
tmp-export-0412 console public, unencrypted, no access logs
ml-features iac no access logs
receipts iac ok
partner-drop iac public
public buckets: 3 of 6
5.3 Lab result
Of six synthetic buckets, three are public: marketing-assets (console; also no access logs), tmp-export-0412 (console; also unencrypted and no access logs) and partner-drop (IaC; public by design for a partner). orders-archive and receipts are ok; ml-features lacks access logs. The pilot shows that "public" needs a field for intended public access: partner-drop's exposure was approved. The other findings, missing access logs and an unencrypted bucket, are outside the study's question, but Lucas reports them to Larkspur through the agreed channel. The pilot's main lesson is about the measure: without an intended-public field, the study would have counted approved exposures as errors.
5.4 Revising measures
Exposure is now "unintended public access", with intended public resources recorded in an approved register. This change is made before the main data are seen and is noted in the pre-registration log. The register comes from Larkspur's existing approvals for public resources, such as partner-drop, so Lucas does not decide himself what was intended. The mobile-app labelling rule found in the pilot is logged the same way. Both changes are listed in the dissertation's method chapter with their dates, so an examiner can see that they preceded the main analysis.
5.5 CIS and CCM
The baseline checks align with CIS Benchmark recommendations for storage and with CSA CCM v4 control domains for data security, which helps relate findings to frameworks Larkspur already uses. Mapping findings to these frameworks lets Larkspur file them where its auditors already look, and lets other organisations compare their own checks. Lucas uses the frameworks as vocabulary, not as evidence that a control works; whether account-level blocking reduces exposure is a question for the follow-up measurement, not for the benchmark.
Proposal feedback
The proposal reviewers ask for a power estimate. Using the pilot's rough rates, Lucas estimates that a year of records gives enough changes to detect a difference of about three percentage points between origins, and states that smaller differences may be missed.
Worked example
The pilot also shows that console changes made through a provider's mobile app are logged with a different origin label. Lucas maps both to "console" and records the rule.
Synthetic case. Buckets and labels are synthetic.
Errors and limits
Changing measures after seeing main data is a source of bias; change them only after pilots, and log why.
Exercises
- EX027 — Lab public. Which buckets were public? Rubric: 3 = marketing-assets, tmp-export-0412, partner-drop; 2 = two; 1 = one; 0 = none.
- EX028 — tmp-export. What issues did tmp-export-0412 have? Rubric: 3 = public, unencrypted, no logs; 2 = two; 1 = one; 0 = none.
- EX029 — Revised measure. How was exposure redefined? Rubric: 3 = unintended public access; 2 = partial; 1 = vague; 0 = none.
- EX030 — Mobile app. What labelling issue was found? Rubric: 3 = mobile app console origin label; 2 = partial; 1 = vague; 0 = none.
- EX031 — Framework links. Which frameworks do checks align with? Rubric: 3 = CIS Benchmarks, CSA CCM v4; 2 = partial; 1 = vague; 0 = none.
- EX032 — When to change measures. When may measures change? Rubric: 3 = after pilots, before main data, logged; 2 = partial; 1 = vague; 0 = none.
Synthesis
Pilots refine measures before they can bias results.
Continue studying — chapter 05 of 08
← Chapter 04 (draft) · Course chapter index · Chapter 06 (draft) →