Chapter 13 — Data systems
Lesson map
Lesson 13 published. Map, example, EX037.
flowchart TD
N0["Learning outcomes"]
N1["13.1 Data model"]
N2["13.2 Encryption at rest"]
N3["13.3 Access control"]
N4["13.4 Retention"]
N5["13.5 Backups"]
N6["13.6 Data in logs"]
N7["Restore tests"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX037"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["13.1 Data model"]
N2["13.2 Encryption at rest"]
N3["13.3 Access control"]
N4["13.4 Retention"]
N5["13.5 Backups"]
N6["13.6 Data in logs"]
N7["Restore tests"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX037"]
[Learning outcomes]
|
v
[13.1 Data model]
|
v
[13.2 Encryption at rest]
|
v
[13.3 Access control]
|
v
[13.4 Retention]
|
v
[13.5 Backups]
|
v
[13.6 Data in logs]
|
v
[Restore tests]
|
v
[Worked example] --> [Exercise EX037]
Learning outcomes
You will (1) design storage for meter readings with security in mind; (2) apply encryption, access control and retention; (3) separate personal data; (4) plan backups and restores.
13.1 Data model
meterhub stores readings (meter ID, timestamp, value) in a time-series database and customer details (name, address) in a separate customer system. Linking requires a lookup the API controls, reducing what a database leak reveals.
If the readings database leaked alone, an attacker would see meter IDs and values but not names or addresses. Linking them requires the customer system, which holds different credentials and is accessed only through the API's controlled lookup. Separation does not make readings harmless, but it raises the effort needed to identify households.
13.2 Encryption at rest
Cloud storage and databases encrypt data at rest by default with provider-managed keys. Solent uses customer-managed keys for readings so it can control and audit key use, as CSA CCM v4's cryptography domain describes.
Provider-managed keys protect against stolen disks but are used automatically, so their use reveals little. Customer-managed keys let Solent set who can use them and see every use in audit logs. Disabling the key makes the data unreadable, which is a strong control in an emergency but also a risk if done by mistake.
13.3 Access control
Only the ingest service writes readings; the API reads them; analysts get aggregated views without meter IDs. Database credentials are short-lived and issued per service.
Each service gets the narrowest access that its job requires: ingest can write but not read customer details; the API can read but not write. Aggregated views for analysts remove meter IDs, so analysis does not expose individuals. Short-lived credentials limit how long a stolen credential stays useful. Each grant is recorded with its owner and reviewed twice a year.
13.4 Retention
Fine-grained readings are kept 13 months for billing disputes, then aggregated to daily values. Retention is enforced by lifecycle rules, not manual deletion.
Thirteen months covers a full billing year plus time for disputes. After that, daily aggregates still support long-term planning while revealing much less about daily household routines. Lifecycle rules apply automatically and consistently, whereas manual deletion is easily forgotten or done unevenly. Elena checks that aggregation runs before deletion, so no month is lost if a job fails, and that failures raise an alert.
13.5 Backups
Daily snapshots are copied to a separate account so that a compromised production account cannot delete them. Restores are tested quarterly.
An attacker who gains control of the production account might try to delete backups before encrypting or destroying data. Copies in a separate account, with different credentials, survive that. Quarterly restore tests prove the copies are usable and show how long recovery takes, which matters for planning. The backup account allows writes from production but no deletes, and its own administrators use separate credentials and MFA.
13.6 Data in logs
Logs must not contain readings linked to addresses; Elena adds a check for meter IDs in application logs.
Developers often log request details while debugging and forget to remove them. A log line containing a meter ID and an address would recreate the link the data model was designed to prevent. Elena's check searches log samples for patterns matching meter IDs and raises an issue for the owning team. Fixes go into the application's own logger, so identifiers are redacted before any line is written, rather than into downstream filters that could fail silently.
Restore tests
A quarterly restore test brings back one day of synthetic readings from the separate backup account into a test database and compares counts and checksums with the source. The first test reveals that restores need a permission nobody on call held; the runbook and access are fixed.
Worked example
The log check finds that the API logs full query parameters, including meter IDs, at debug level, and debug logging was left on in one environment. Debug logging is turned off and the logger is changed to redact meter IDs at every level.
Synthetic case. Data and findings are synthetic.
Errors and limits
Encryption at rest does not protect against an authorised service misusing its access.
Exercises
- EX037 — Separation. Why separate readings from customer details? Rubric: 3 = leak reveals less; 2 = partial; 1 = vague; 0 = none.
- EX038 — Backup protection. How are backups protected from a compromised account? Rubric: 3 = copied to separate account; 2 = partial; 1 = vague; 0 = none.
- EX039 — Log finding. What did the log check find? Rubric: 3 = meter IDs in debug logs; 2 = partial; 1 = vague; 0 = none.
Synthesis
Design data storage so each part reveals and allows as little as possible.
Continue studying — chapter 13 of 50
← Chapter 12 (draft) · Course chapter index · Chapter 14 (draft) →