Chapter 17 — Change management and documentation
Lesson map
Lesson 17 published. Map, example, EX065.
flowchart TD
N0["Learning outcomes"]
N1["17.1 Why change control is security"]
N2["17.2 A change request"]
N3["17.3 Rollback"]
N4["17.4 Keeping documents current"]
N5["17.5 Reviewers"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX065"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["17.1 Why change control is security"]
N2["17.2 A change request"]
N3["17.3 Rollback"]
N4["17.4 Keeping documents current"]
N5["17.5 Reviewers"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX065"]
[Learning outcomes]
|
v
[17.1 Why change control is security]
|
v
[17.2 A change request]
|
v
[17.3 Rollback]
|
v
[17.4 Keeping documents current]
|
v
[17.5 Reviewers]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX065]
Learning outcomes
You will (1) explain why every change on the line needs a ticket; (2) write a change request with a rollback plan; and (3) keep drawings and records in step with reality.
17.1 Why change control is security
Most problems on plants come from changes nobody recorded: a firewall rule added in a hurry, a program edited during a breakdown, a camera bolted on by a contractor. Change control means every change is requested, reviewed, approved, carried out and recorded. It also makes it easy to notice an unrecorded change, which may be the first sign of an attack.
At HPFL several problems in this book started as unrecorded changes: the edited wrapper program in Chapter 11, the temporary firewall rule in Chapter 12 and the remote desktop port in Chapter 16. Each would have been visible sooner with a ticket.
17.2 A change request
It states what will change, why, which devices, who does it, when, the risk to the process, how success is checked, and how to undo it. Emergency changes during a breakdown are allowed but get recorded afterwards within one shift.
At HPFL Tomás's request for the VIS-CAM update covered all of these in a few lines. Keep the form short so that people actually use it. The approver checks that the success test and the rollback are realistic before agreeing.
17.3 Rollback
Before the change, save the current program or configuration and its hash. If the check fails, restore it. A change with no rollback is a gamble.
At HPFL the rollback for VIS-CAM was to reinstall firmware 1.0.7 from a saved file with its hash recorded. Test that the saved file is complete before starting the change. If rollback itself fails, stop and call the controls engineer. At HPFL the hashes of saved files were recorded in the change ticket, so a restored file could be checked before loading. Without that, a rollback could load a corrupted or edited copy without anyone noticing.
17.4 Keeping documents current
Update the asset register, network drawing and rule table in the same ticket. Mark drawings with a revision number and date.
At HPFL the network drawing was still at revision 3 while the IoT VLAN appeared only in revision 4 drafts, which the capstone walk-through found. Updating documents in the same ticket prevents that gap from opening. When a drawing changes, mark the old revision as superseded and keep it, so past decisions can still be understood. Check revision numbers during each walk-through; a drawing whose revision is behind reality is itself a finding, as HPFL learned.
17.5 Reviewers
The person who makes a change should not be the only person who approves it. At HPFL the controls engineer reviews program changes, the network owner reviews firewall and switch changes, and the supervisor confirms the production window. A second pair of eyes catches typing errors in addresses and values before they reach the line.
Record the reviewer's name in the ticket. If the only person who can review is unavailable, wait for the next window unless the change is an emergency.
Worked example (synthetic HPFL)
The vision camera supplier needs firmware 1.1.0 on VIS-CAM. Tomás writes: "Change 512 — update VIS-CAM 1.0.7 to 1.1.0 for reject-accuracy fix; by quality technician; Tuesday 06:00 planned stop; risk: rejects stop working; check: 20 test packs including 5 bad seals rejected; rollback: reinstall 1.0.7 from saved file, hash recorded." The update passes the check and the register shows 1.1.0.
Synthetic case — HPFL. The change and supplier fix are invented.
Errors and limits
Change control can become a paper exercise; keep forms short and enforce the key fields. Emergency changes must not become the normal route.
Exercises
- EX065 — Fields of a request. List six fields of a change request. Rubric: 3 = what, why, devices, who, when, risk, check, rollback (any six); 2 = four; 1 = two; 0 = none.
- EX066 — Breakdown edit. The engineer edits PLC-FILL during a 2 a.m. breakdown. What must happen afterwards? Rubric: 3 = recorded as emergency change within shift, new backup and hash, register updated; 2 = record only; 1 = vague; 0 = none.
- EX067 — Failed check. VIS-CAM's test lets 2 of 5 bad seals through. What happens next? Rubric: 3 = rollback to 1.0.7, verify, record failure, contact supplier; 2 = rollback; 1 = run anyway; 0 = none.
- EX068 — Unrecorded change. Why is an unrecorded change worth reporting even if it looks harmless? Rubric: 3 = may be unauthorised/attack sign; breaks baseline and backups; 2 = one; 1 = ignore; 0 = none.
Synthesis
Requested, reviewed, reversible and recorded changes keep the line known; unrecorded ones are warnings.
Continue studying — chapter 17 of 18
← Chapter 16 (draft) · Course chapter index · Chapter 18 (draft) →