Chapter 49 — Engineering thesis on response performance | Engineering Incident response and cyber resilience

Open, free learning for anyone who wants to protect people and technology. No account is required to read this text or its worked answers.

Lesson map

Lesson 49 published. Map, example, EX097.

flowchart TD
  N0["Learning outcomes and connection"]
  N1["49.1 What an engineering thesis is"]
  N2["49.2 Structure"]
  N3["49.3 Baselines and attribution"]
  N4["49.4 Uncertainty and validity"]
  N5["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5
  N5 --> EX["Worked example"]
  EX --> Q["Exercise EX097"]

Logical drawing

flowchart LR
  N0["Learning outcomes and connection"]
  N1["49.1 What an engineering thesis is"]
  N2["49.2 Structure"]
  N3["49.3 Baselines and attribution"]
  N4["49.4 Uncertainty and validity"]
  N5["Chapter synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> Q["EX097"]

[Learning outcomes and connection]
    |
    v
[49.1 What an engineering thesis is]
    |
    v
[49.2 Structure]
    |
    v
[49.3 Baselines and attribution]
    |
    v
[49.4 Uncertainty and validity]
    |
    v
[Chapter synthesis]
    |
    v
[Worked example] --> [Exercise EX097]

Learning outcomes and connection

After this chapter you can (1) structure an engineering thesis that links design decisions to measured outcomes; (2) report method, baseline, results and uncertainty; (3) discuss generalisability and limits honestly; and (4) write about failures as contributions. It builds on Chapters 29, 41, 42 and 44–48 and prepares Chapter 50.

49.1 What an engineering thesis is

An engineering thesis explains a designed and built system and what was learned by measuring it. Its contribution is a justified design with evidence of how it performs and where it fails — not an original theory (the doctoral task) or a single empirical study of practice (the master's dissertation). Readers should come away able to reuse the design decisions that worked, avoid those that did not, and know the conditions under which the results hold. In the HVSS-style capstone, the reusable decisions include the write-behind outbox (DD-4) and its limit under gradual storage degradation.

49.2 Structure

Problem and context: mission, stakeholders, constraints (Chapter 20).
Requirements and objectives with rationale (Chapter 21).
Design: architecture, key decisions and alternatives considered, each with the reason chosen (Chapters 18, 26, 41).
Implementation: enough to reproduce; link to repository and versions.
Method of evaluation: scenarios, baseline, measures, runs, analysis plan (Chapters 29, 46).
Results: per requirement and objective, with uncertainty; include failures.
Discussion: which design decisions explain which outcomes; threats to validity; generalisability.
Limits and future work.
Appendices: dossier summary (Chapter 48), reproduction package (Chapter 47).

In the DD-4 extract below, results under item 6 compare against Lab 40 design N as baseline, and the limit to abrupt failures belongs under item 8.

49.3 Baselines and attribution

A result means something only relative to a baseline: the system without the design feature (e.g., direct writes vs outbox), a simpler design, or a published reference. Attribute outcomes to decisions cautiously — several changes made between trial iterations confound attribution; ablation (removing one feature at a time) separates them.

The DD-4 extract in the worked example below follows this advice. Its baseline is Lab 40's design N, which loses 15 items per 15-minute outage. It then reports an ablation with fix D11 alone, without the outbox, which lost items in every run, so the improvement cannot be credited to D11 by itself.

49.4 Uncertainty and validity

Report distributions and intervals. Discuss internal validity (could something other than the design explain the result? e.g., synthetic traffic too regular), construct validity (do the measures capture the requirement? e.g., queue latency vs time to informed decision), and external validity (would results hold with real providers, real staff, real attack patterns?). The synthetic and mocked nature of a capstone is a limit to state, not to hide.

In the DD-4 extract, the external-validity threat is named directly: the mock store fails abruptly, while real storage often degrades gradually. The claim about the outbox is therefore limited to abrupt storage failure. In DD-7, precision measured per batch window did not capture the requirement for slow-burn cases, which is a construct-validity limit.

49.5 Worked example (synthetic): results and discussion extract

"Design decision DD-4 (write-behind outbox for evidence) was evaluated against the baseline of direct writes (Lab 40 design N) in recovery trial RT-1 (10 seeds). Direct writes lost a median of 15 items per 15-minute outage; the outbox lost none after fix D11; before the fix, one run in ten lost 3 items due to a health-detection race. We attribute the improvement to DD-4 with D11; an ablation with D11 alone (no outbox) lost items in all runs. Generalisability: the mock store fails abruptly; real storage often degrades gradually (rising latency), which our health check may not detect; we therefore claim the result only for abrupt failure and list gradual degradation as future work."

"Failure as contribution: the scoring design DD-7 (rule-based priority with entity grouping) improved precision in the trial but missed two slow-burn cases that never crossed thresholds within one batch window. We report this as a limit of batch-window grouping and propose rolling-window grouping, with expected latency costs estimated from Chapter 36's model."

49.6 Errors and limits

Describing the system without evaluation.
Results without baseline or uncertainty.
Attributing outcomes to one decision when several changed.
Omitting failures.

The two extracts in the worked example avoid all four errors. DD-4 is reported against a baseline and an ablation, with the one seed in ten that lost three items stated openly. DD-7 is presented with its failure on two slow-burn cases, turned into a limit of batch-window grouping and a proposal for rolling windows. A thesis that reported only DD-7's improved precision would commit the last error.

Exercises — chapter 49

EX097 — Validity threats. Objective: discuss validity. Inputs: the result "ingest p99 4.2 min under 5× load" from the dossier. Steps: write one internal, one construct and one external validity threat with how each could be addressed. Expected evidence: three threats. Rubric: 3 = specific to the result with remedies; 2 = generic; 1 = one; 0 = none.

EX098 — Abstract. Objective: summarise honestly. Inputs: the worked examples of Chapters 46–49. Steps: write a 200-word thesis abstract stating problem, design, method, main results with numbers, one failure and limits. Expected evidence: abstract. Rubric: 3 = numbers, failure and limits present, no overclaiming; 2 = missing one; 1 = promotional; 0 = none.

Chapter synthesis

An engineering thesis links design decisions to measured outcomes against baselines, reports uncertainty and validity threats, attributes carefully and treats failures and limits as part of the contribution.