Chapter 6 — Privacy and data protection: controlled evaluation | Postgraduate Privacy and data protection

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 06 published. Map, example, EX026.

flowchart TD
  N0["Learning outcomes"]
  N1["6.1 Running the test"]
  N2["6.2 Results"]
  N3["6.3 Interpretation"]
  N4["6.4 Communicating results"]
  N5["Lab — Consent banner A/B test"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
  N6 --> EX["Worked example"]
  EX --> Q["Exercise EX026"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["6.1 Running the test"]
  N2["6.2 Results"]
  N3["6.3 Interpretation"]
  N4["6.4 Communicating results"]
  N5["Lab — Consent banner A/B test"]
  N6["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX026"]

[Learning outcomes]
    |
    v
[6.1 Running the test]
    |
    v
[6.2 Results]
    |
    v
[6.3 Interpretation]
    |
    v
[6.4 Communicating results]
    |
    v
[Lab — Consent banner A/B test]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX026]

Learning outcomes

You will (1) run an A/B test ethically; (2) estimate a difference in proportions; (3) interpret behavioural outcomes for compliance.

6.1 Running the test

Corvina randomly shows banner A or B to 8,000 new app users, 4,000 each, over two weeks. Priscila checks that the test itself is lawful: banner A is shown only during the test and both banners record choices correctly. She also limits analysis to aggregate choices. The test is approved by the data protection officer. She notes that testing a less compliant design should be brief and justified by the need to measure the improvement. The test runs for two weeks with random assignment.

6.2 Results

The lab computes results. Banner A has an acceptance rate of 79.0%; banner B, 57.0%. The difference is 22.0 percentage points, with a 95% interval from 20.0 to 24.0 points and a z statistic of 21.1. The difference is far beyond chance. Over ninety days, withdrawals per thousand consents are 41 under A and 12 under B. Customers who accepted under A were more than three times as likely to change their minds later. The confidence interval for the acceptance difference, 20.0 to 24.0 points, excludes zero, so the gap is not a chance result of this sample.

6.3 Interpretation

Priscila interprets the results. Banner A pushed about a fifth of users into accepting when an equal choice would not have. Higher withdrawals under A suggest that many of those consents did not reflect real preferences. From a compliance view, consents under A are questionable as freely given. From a business view, B produces fewer but more durable consents. She recommends adopting B immediately. The extra consents under A carry a legal and reputational cost that the acceptance rate alone hides from view.

6.4 Communicating results

Marketing leaders are disappointed by lower acceptance. Priscila presents the evidence: B is compliant, A probably is not, and B's consents are more stable. She also estimates that marketing reach falls less than acceptance suggests, because fewer people withdraw. Dr. Ward notes that controlled evaluation turns a values debate into an evidence-based decision, which is easier for leaders to accept even when the result is unwelcome. Her one-page summary shows the two acceptance rates, the two withdrawal rates and a single recommendation, with the confidence interval in a footnote for technical readers.

Lab — Consent banner A/B test

# Consent banner A/B test: A has "Accept" prominent; B gives "Reject all" equal prominence. Two-proportion z-test.
import math
na, ka, nb, kb = 4000, 3160, 4000, 2280
pa, pb = ka / na, kb / nb; p = (ka + kb) / (na + nb)
z = (pa - pb) / math.sqrt(p * (1 - p) * (1 / na + 1 / nb))
se = math.sqrt(pa * (1 - pa) / na + pb * (1 - pb) / nb)
print(f"A accept rate {pa:.1%}; B accept rate {pb:.1%}; difference {pa-pb:.1%} (95% CI {pa-pb-1.96*se:.1%} to {pa-pb+1.96*se:.1%}); z = {z:.1f}")
wa, wb = 130, 27  # synthetic 90-day follow-up: withdrawals among consenting users
print(f"withdrawals within 90 days per 1,000 consents: A {1000*wa/ka:.0f} vs B {1000*wb/kb:.0f}")

Run with python3 lab_ab.py. Verified output:

A accept rate 79.0%; B accept rate 57.0%; difference 22.0% (95% CI 20.0% to 24.0%); z = 21.1
withdrawals within 90 days per 1,000 consents: A 41 vs B 12

Worked example

Priscila finds acceptance of 79.0% under banner A versus 57.0% under equal buttons (difference 22.0 points, 95% CI 20.0 to 24.0) and withdrawals of 41 versus 12 per 1,000, and recommends banner B.

Synthetic case. Users and choices are synthetic.

Errors and limits

Two weeks may not capture seasonal behaviour; repeat later.

Exercises

EX026 — Acceptance difference. What was the difference in acceptance between A and B?
Show marking criteria
Rubric: 3 = 22.0 points (95% CI 20.0 to 24.0);; 2 = approximately;; 1 = wrong;; 0 = none.
EX027 — Withdrawal ratio. How did withdrawals compare?
Show marking criteria
Rubric: 3 = 41 vs 12 per 1,000, more than three times higher under A;; 2 = numbers only;; 1 = vague;; 0 = none.
EX028 — Test ethics. Why must showing banner A during the test be brief?
Show marking criteria
Rubric: 3 = it is likely less compliant, justified only to measure improvement;; 2 = mentions compliance;; 1 = unlimited;; 0 = none.
EX029 — Durable consent. What business benefit does banner B offer?
Show marking criteria
Rubric: 3 = fewer but more durable consents;; 2 = mentions stability;; 1 = vague;; 0 = none.
EX030 — Freely given doubt. Why are consents under A questionable?
Show marking criteria
Rubric: 3 = design pushed users, and higher withdrawals suggest choices did not reflect preferences;; 2 = mentions nudging;; 1 = valid;; 0 = none.

Synthesis

Chapter 6 evaluates a design. Chapter 7 integrates privacy into the organisation.

Continue studying — chapter 06 of 12

← Chapter 05 (draft) · Course chapter index · Chapter 07 (draft) →