◇ OpenCyber Academy
Technical · Chapter 16 · English

Chapter 16 — Recovery and returning to production

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Chapter 16 — Recovery and returning to production

Lesson map

Lesson 16 published. Map, example, EX061.

flowchart TD
  N0["Learning outcomes"]
  N1["16.1 Order of recovery"]
  N2["16.2 Verification before restart"]
  N3["16.3 Watching afterwards"]
  N4["16.4 Lessons learned"]
  N5["16.5 Record keeping"]
  N6["16.6 Communicating the restart"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
  N7 --> EX["Worked example"]
  EX --> Q["Exercise EX061"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["16.1 Order of recovery"]
  N2["16.2 Verification before restart"]
  N3["16.3 Watching afterwards"]
  N4["16.4 Lessons learned"]
  N5["16.5 Record keeping"]
  N6["16.6 Communicating the restart"]
  N7["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX061"]

[Learning outcomes]
    |
    v
[16.1 Order of recovery]
    |
    v
[16.2 Verification before restart]
    |
    v
[16.3 Watching afterwards]
    |
    v
[16.4 Lessons learned]
    |
    v
[16.5 Record keeping]
    |
    v
[16.6 Communicating the restart]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX061]

Learning outcomes

You will (1) describe the steps to restore a line after an incident; (2) verify restored systems before restarting; and (3) explain why lessons learned must change something.

16.1 Order of recovery

Recovery is planned by the incident lead, controls engineer and operations. Usually: confirm the cause is contained, rebuild affected computers from known-good images, restore PLC programs from verified backups (Chapter 11), change passwords that may be exposed, then test.

At HPFL HMI-1 was rebuilt from the vendor's image, both PLC programs were compared with matching backups, and the admin password was changed. Following the order prevents restarting with a problem still present. Each step is recorded with its time and the person who did it.

16.2 Verification before restart

Compare restored PLC programs with verified copies, check HMI screens show correct values against local gauges, confirm firewall rules match the approved table, and run the line empty through a test cycle. Quality signs off before product runs.

At HPFL the test cycle ran empty for 30 minutes and quality signed off at 11:20 before production restarted. Record each check and its result in the incident file. If any check fails, the line does not restart until the cause is understood. The file then shows who approved the restart and on what evidence.

16.3 Watching afterwards

Attackers sometimes return. For days after restart, watch logins, remote sessions and new devices more closely, and report anything unusual immediately.

Agree with the supervisor how long the watch period lasts and who reviews logs each day. At HPFL the lessons meeting found an internet-reachable remote desktop port opened by a contractor, which makes close watching of remote sessions especially important. Report anything unusual to the incident lead, not only to your shift. Note the end date of the watch period in the incident file so it does not quietly lapse.

16.4 Lessons learned

A short meeting within two weeks answers: what happened, what worked, what did not, what changes. Each change has an owner and date. A lessons session that changes nothing has failed.

At HPFL the lessons meeting assigned closing the remote desktop port on HMI-1 and a monthly check to the network owner. Each action is tracked until it is done and shown to be effective, and the meeting notes are kept with the incident file. Invite the operators who were on shift; they often know what worked.

16.5 Record keeping

Keep the timeline, photographs, decisions and test results together. They support insurance, regulators and the next improvement.

Store the records together in one folder or ticket, with access limited to people who need them. Include the first report, its corrections, photographs and the test cycle results. The next walk-through can then check whether lessons were applied. Keep the records for as long as the site's policy requires, and note who may read them. When an insurer or regulator asks a question months later, the answer should come from these records rather than from memory, which fades quickly.

16.6 Communicating the restart

Tell every shift what changed: new passwords, any screens that look different, and which unusual signs to report during the watch period.

At HPFL that meant telling every shift that the admin password had changed and that unexpected remote sessions or new devices should be reported. A short written note left at HMI-1, without passwords, helps people who missed the briefing. Brief the next two shifts in person if possible, and ask them to confirm they understood. Questions raised at these briefings often show where the procedure needs to be clearer, so record them as well.

Worked example (synthetic HPFL)

After the ransom incident, HMI-1 is rebuilt from the vendor's image; both PLC programs are compared with matching backups and found unchanged. The admin password is changed. A test cycle runs empty for 30 minutes, quality signs off at 11:20, and production restarts. The lessons meeting finds HMI-1 had an internet-reachable remote desktop port opened by a contractor; the action — close it and add a monthly check — is assigned to the network owner.

Synthetic case — HPFL. The recovery is invented.

Errors and limits

Restoring from a backup taken after the compromise reloads the problem; verified dates and hashes matter. Pressure to restart quickly is real; the test cycle is non-negotiable.

Exercises

Synthesis

Recover in order, verify before restart, watch afterwards and turn lessons into owned actions.

Continue studying — chapter 16 of 18

← Chapter 15 (draft) · Course chapter index · Chapter 17 (draft) →