Chapter 46 — Engineering capstone II

Chapter 46 — Engineering capstone II

Lesson map

Lesson 46 published. Map, example, EX136.

flowchart TD
  N0["Learning outcomes"]
  N1["46.1 Building"]
  N2["46.2 First trial"]
  N3["46.3 Improvements"]
  N4["46.4 Repeatability"]
  N0 --> N1 --> N2 --> N3 --> N4
  N4 --> EX["Worked example"]
  EX --> Q["Exercise EX136"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["46.1 Building"]
  N2["46.2 First trial"]
  N3["46.3 Improvements"]
  N4["46.4 Repeatability"]
  N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX136"]

[Learning outcomes]
    |
    v
[46.1 Building]
    |
    v
[46.2 First trial]
    |
    v
[46.3 Improvements]
    |
    v
[46.4 Repeatability]
    |
    v
[Worked example] --> [Exercise EX136]

Learning outcomes

You will (1) build and test a recovery capability; (2) measure against an objective; (3) iterate until the objective is met.

46.1 Building

Lindiwe builds the recovery environment on isolated virtual machines and writes scripts to rebuild each dispatch component: directory connection, message queue, database, application servers, certificates and the licence server. Each script is idempotent, producing the same result when run twice, so a partial failure can be retried safely. She tests scripts individually before combining them, keeping a log of problems found and fixed. She stores every script in version control and records each test run with its date and outcome in a project log.

46.2 First trial

The first timed trial takes six hours and forty minutes, well over the four-hour objective. Most time goes to restoring the database from backup and to manual certificate steps. Lindiwe records the timeline minute by minute. Prof. Hale reminds her that a failed first trial is normal and that the timeline shows exactly where to work, which is more useful than a vague sense that recovery is slow. She shares the timeline with the security lead, highlighting the two largest sources of delay.

46.3 Improvements

Lindiwe makes three changes: restoring the database from a more recent snapshot stored in the immutable backup system, automating certificate installation and starting independent components in parallel. The second trial takes four hours and ten minutes. A third change, pre-staging application server images, brings the third trial to three hours and twenty minutes, inside the objective. She records each change and its effect on the timeline. Prof. Hale asks her to chart the three trials side by side, showing how each change shortened a specific part of the timeline.

46.4 Repeatability

One successful trial could be luck. Lindiwe runs two more trials on different days with different team members following the runbook. Both finish within four hours. She notes that the slowest remaining step is database restoration, which grows with data volume, so the objective may be missed as dispatch data grow. She recommends repeating the trial every six months and after major changes. The security lead adds the six-monthly trial to the security calendar and assigns an owner. Lindiwe writes the procedure so new staff can lead it.

Worked example

Lindiwe builds idempotent recovery scripts, measures a first trial at 6 hours 40 minutes, improves to 3 hours 20 minutes and confirms repeatability with different staff.

Synthetic case. Trials are synthetic.

Errors and limits

A single successful recovery trial does not prove recovery is reliable.

Exercises

• EX136 — First trial time. How long did the first trial take? Rubric: 3 = six hours forty minutes;; 2 = partly correct;; 1 = wrong;; 0 = none.
• EX137 — Idempotent meaning. What does idempotent mean for her scripts? Rubric: 3 = same result when run twice;; 2 = partly correct;; 1 = vague;; 0 = none.
• EX138 — Remaining slow step. What remains the slowest step? Rubric: 3 = database restoration;; 2 = partly correct;; 1 = other;; 0 = none.

Continue studying — lesson 46 (chapter 46 of 50)

← Chapter 45 · Course chapter index · Chapter 47 →

Continue to the next authored lesson.