Chapter 7 — Privacy and data protection: data and telemetry
Lesson map
Lesson 07 published. Map, example, EX025.
flowchart TD
N0["Learning outcomes"]
N1["7.1 Telemetry everywhere"]
N2["7.2 The log lab"]
N3["7.3 Fixing at the source"]
N4["7.4 Retention for logs"]
N5["Lab — Personal data in booking-server logs"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5
N5 --> EX["Worked example"]
EX --> Q["Exercise EX025"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["7.1 Telemetry everywhere"]
N2["7.2 The log lab"]
N3["7.3 Fixing at the source"]
N4["7.4 Retention for logs"]
N5["Lab — Personal data in booking-server logs"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX025"]
[Learning outcomes]
|
v
[7.1 Telemetry everywhere]
|
v
[7.2 The log lab]
|
v
[7.3 Fixing at the source]
|
v
[7.4 Retention for logs]
|
v
[Lab — Personal data in booking-server logs]
|
v
[Worked example] --> [Exercise EX025]
Learning outcomes
You will (1) find personal data in logs and telemetry; (2) mask it safely; (3) set log retention.
7.1 Telemetry everywhere
Marlow's systems produce logs, error reports and usage statistics. The booking server logs each request, the SMS gateway logs delivery results and the website sends performance data to a monitoring service. Leila learns that telemetry is often designed by developers for debugging, with little thought about privacy. Logs are copied to monitoring tools, shared with vendors during support cases and kept longer than the data they describe. A log line that includes an email address can therefore travel further than the booking record itself.
7.2 The log lab
Leila analyses a synthetic sample of booking-server logs. The lab shows that 95 of 400 lines, 23.8%, contain personal data: 75 include patient email addresses and 20 include phone numbers from failed text messages. Each of those lines links a person to a dental appointment. After a masking step that replaces emails with a placeholder and hides phone digits, no lines with personal data remain. The booking reference still allows support staff to trace a problem through the practice-management system when they are authorised to do so.
7.3 Fixing at the source
Masking after the fact is useful, but Leila argues that the better fix is to stop logging personal data in the first place. She asks the developer to log booking references instead of emails and to record only the last three digits of phone numbers for failed messages. The developer agrees and changes the logging calls in the next release. Leila reruns the analysis on synthetic traffic from the new version and confirms that no new lines contain email addresses or full phone numbers.
7.4 Retention for logs
Logs are kept for ninety days in the monitoring service by default, and old logs in the booking server were never deleted. Leila proposes thirty days for detailed logs, which covers most troubleshooting, and twelve months for aggregated statistics without personal data. She documents the decision with the data protection officer, linking it to storage limitation in Article 5(1)(e). The IT contractor configures automatic deletion, and Leila adds a quarterly check that the oldest log file is younger than the limit. The first check passes.
Lab — Personal data in booking-server logs
"""Find and mask personal data in synthetic booking-server log lines (stdlib only)."""
import random, re
random.seed(7)
names = ["ana", "ben", "chloe", "dev", "erin", "farid", "gita", "hugo"]
lines = []
for i in range(400):
kind = random.random()
if kind < 0.15:
lines.append(f"INFO booking created for {random.choice(names)}@example.org slot={i}")
elif kind < 0.22:
lines.append(f"WARN sms failed to +44 7700 900{random.randint(100,999)} ref={i}")
else:
lines.append(f"INFO booking created ref=B{i:05d} clinic={random.randint(1,8)}")
EMAIL = re.compile(r"[\w.]+@[\w.]+"); PHONE = re.compile(r"\+44 \d{4} \d{6}")
hit = [l for l in lines if EMAIL.search(l) or PHONE.search(l)]
masked = [PHONE.sub("+44 **** ******", EMAIL.sub("<email>", l)) for l in lines]
left = sum(1 for l in masked if EMAIL.search(l) or PHONE.search(l))
print(f"log lines {len(lines)}; with personal data {len(hit)} ({100*len(hit)/len(lines):.1f}%)")
print(f"emails {sum(1 for l in lines if EMAIL.search(l))}; phones {sum(1 for l in lines if PHONE.search(l))}; after masking {left}")
Run with python3 lab_logs.py. Verified output:
log lines 400; with personal data 95 (23.8%)
emails 75; phones 20; after masking 0
Worked example
Leila finds personal data in 95 of 400 synthetic log lines (23.8%), masks them to zero, has the developer stop logging emails and full phone numbers, and sets thirty-day log retention.
Synthetic case. Logs are synthetic; GDPR Article 5(1)(e) is real.
Errors and limits
Regular expressions miss unusual formats; review samples by hand as well.
Exercises
- EX025 — Log share. What share of log lines held personal data? Rubric: 3 = 95 of 400, 23.8%;; 2 = approximate;; 1 = wrong;; 0 = none.
- EX026 — Source fix. What did the developer change? Rubric: 3 = log booking references, partial phone digits;; 2 = mentions references;; 1 = vague;; 0 = none.
- EX027 — Why logs travel. Why do logs spread further than records? Rubric: 3 = copied to monitoring tools and shared in support cases;; 2 = mentions copies;; 1 = vague;; 0 = none.
- EX028 — Log retention. What retention did Leila propose for detailed logs? Rubric: 3 = thirty days;; 2 = partly correct;; 1 = ninety;; 0 = none.
Continue studying — lesson 07 (chapter 07 of 18)
← Chapter 06 · Course chapter index · Chapter 08 →
Continue to the next authored lesson.