Chapter 8 — Independent review and defense
Lesson map
Lesson 08 published. Map, example, EX045.
flowchart TD
N0["Learning outcomes"]
N1["8.1 Independent review"]
N2["8.2 The defense"]
N3["8.3 Handing over"]
N4["8.4 Reflection"]
N0 --> N1 --> N2 --> N3 --> N4
N4 --> EX["Worked example"]
EX --> Q["Exercise EX045"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["8.1 Independent review"]
N2["8.2 The defense"]
N3["8.3 Handing over"]
N4["8.4 Reflection"]
N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX045"]
[Learning outcomes]
|
v
[8.1 Independent review]
|
v
[8.2 The defense]
|
v
[8.3 Handing over]
|
v
[8.4 Reflection]
|
v
[Worked example] --> [Exercise EX045]
Learning outcomes
You will (1) obtain independent review of analysis; (2) defend a master's dissertation; (3) hand findings to the partner organisation.
8.1 Independent review
Before submission, a statistician from another department at Kestrel reviews Tomás's code and data without his help. She reruns the analysis from the archive and obtains the same estimate and interval. She questions whether 2,000 bootstrap resamples are enough; Tomás reruns with 10,000 and the interval changes by less than a tenth of a day. She also asks for a placebo test using a pre-pilot period as if it were the pilot, which shows no effect, supporting the design. The placebo result is reported in an appendix.
8.2 The defense
At the defense, examiners ask why the council should trust a result from two departments. Tomás answers that the result is consistent across months, robust to the checks reported and supported by a plausible mechanism, but that extension should itself be evaluated. An examiner asks about the synthetic reproduction; he explains that the council's data team ran the extraction internally and the synthetic data match its published summaries. The examiners award a pass and praise the treatment of rival explanations. Minor edits follow.
8.3 Handing over
Tomás presents the findings to Arden's data governance board. The board decides to extend stewardship to libraries and parking, while social care waits for a review of its legal duties. It adopts the normalisation rules as a council standard and asks the data team to keep reporting correction time to the last system. Tomás hands over the analysis scripts and a guide so council staff can rerun the evaluation for the extension without him. The guide includes the placebo test and the normalisation rules.
8.4 Reflection
Tomás reflects that the most useful part of his work may be the measurement definitions, which the council will keep using, rather than the single estimate. He notes that governance research must respect the people whose work is studied: the stewards were anxious about being evaluated, and department-level reporting kept their trust. Dr. Solberg closes the supervision by reminding him that good evaluations make it easy for organisations to learn from both success and failure. Tomás later joins Arden's data team.
Worked example
An independent statistician reproduces the estimate, checks resample count and runs a placebo test; Tomás passes the defense, and the council extends stewardship and adopts his definitions.
Synthetic case. The review and board are synthetic.
Errors and limits
Extensions to new departments may not reproduce pilot effects; evaluate them.
Exercises
- EX045 — Reviewer check. What did the reviewer change in the bootstrap? Rubric: 3 = resamples from 2,000 to 10,000;; 2 = partly correct;; 1 = vague;; 0 = none.
- EX046 — Placebo. What did the placebo test show? Rubric: 3 = no effect in a pre-pilot period;; 2 = partly correct;; 1 = vague;; 0 = none.
- EX047 — Examiner concern. What did examiners question? Rubric: 3 = trusting a two-department result;; 2 = partly correct;; 1 = vague;; 0 = none.
- EX048 — Board decision. Which departments does stewardship extend to next? Rubric: 3 = libraries and parking;; 2 = both;; 1 = one;; 0 = none.
- EX049 — Social care. Why does social care wait? Rubric: 3 = review of its legal duties;; 2 = partly correct;; 1 = vague;; 0 = none.
- EX050 — Lasting value. What may be the most useful part of Tomás's work? Rubric: 3 = the measurement definitions;; 2 = partly correct;; 1 = vague;; 0 = none.
Continue studying — lesson 08 (chapter 08 of 08)
← Chapter 07 · Course chapter index · Next course via chapter index →
The chapter index leads to the next course in the study path.