Chapter 37 — Evidence review seminar

Chapter 37 — Evidence review seminar

Lesson map

Lesson 37 published. Map, example, EX109.

flowchart TD
  N0["Learning outcomes"]
  N1["37.1 The seminar task"]
  N2["37.2 Assessing quality"]
  N3["37.3 Balanced synthesis"]
  N4["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4
  N4 --> EX["Worked example"]
  EX --> Q["Exercise EX109"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["37.1 The seminar task"]
  N2["37.2 Assessing quality"]
  N3["37.3 Balanced synthesis"]
  N4["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX109"]

[Learning outcomes]
    |
    v
[37.1 The seminar task]
    |
    v
[37.2 Assessing quality]
    |
    v
[37.3 Balanced synthesis]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX109]

Learning outcomes

You will (1) critically review evidence about cryptographic practice; (2) assess sources' quality; (3) present a balanced synthesis.

37.1 The seminar task

In the evidence review seminar, each student reviews evidence on a question. Priya's is: "Does automated certificate management reduce expiry outages?" She gathers sources: the ACME specification, RFC 8555, which describes the protocol but not outcomes; peer-reviewed measurement studies of certificate ecosystems; operator reports; and her own Tidewell data. She classifies each by type and strength, noting that a specification explains how something works but not whether it improves results. She records each source in a table with its type, date and author, so peers can follow her reasoning.

37.2 Assessing quality

Priya assesses each source: who produced it, the method, sample, possible conflicts of interest and whether results can be checked. Vendor reports claim large reductions but give no method. Measurement studies offer large samples but measure certificate lifetimes, not outages. Her Tidewell data are accurate but tiny. She learns to weigh evidence by quality rather than by how often a claim is repeated, and to say when evidence is indirect. She asks a classmate to rate the sources independently and compares their ratings.

37.3 Balanced synthesis

Priya's synthesis says that evidence consistently suggests automation shortens certificate lifetimes and reduces manual work, that direct evidence on outage reduction is limited and mostly from operator reports, and that automation introduces silent failure modes requiring monitoring, as Tidewell's own portal case showed. She presents the synthesis with a table of sources and strengths. Peers question her treatment of vendor reports, and she adds a note explaining why she gave them low weight. The seminar leader commends the synthesis as a model.

Worked example

Priya reviews sources on automated certificate management, weighs vendor, measurement and local data by quality, and presents a balanced synthesis noting silent-failure risks.

Synthetic case. Her local data are invented; RFC 8555 is real.

Errors and limits

Evidence that is repeated widely is not necessarily strong. Local data rarely generalise.

Exercises

• EX109 — Specification limit. What can RFC 8555 tell Priya, and what not? Rubric: 3 = how ACME works, not whether it reduces outages;; 2 = partial;; 1 = outcomes;; 0 = none.
• EX110 — Vendor weight. Why did vendor reports receive low weight? Rubric: 3 = they give no method and may have conflicts of interest;; 2 = no method;; 1 = high weight;; 0 = none.
• EX111 — Synthesis caveat. What caveat does the synthesis include about automation? Rubric: 3 = automation introduces silent failure modes requiring monitoring;; 2 = silent failures;; 1 = vague;; 0 = none.

Synthesis

Critical review weighs evidence by quality and directness, producing balanced conclusions with stated caveats.

Continue studying — chapter 37 of 40

← Chapter 36 (draft) · Course chapter index · Chapter 38 (draft) →