Chapter 37 — Evidence review seminar

Chapter 37 — Evidence review seminar

Lesson map

Lesson 37 published. Map, example, EX109.

flowchart TD
  N0["Learning outcomes"]
  N1["37.1 Presenting evidence"]
  N2["37.2 Challenges"]
  N3["37.3 Improving the evaluation"]
  N4["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4
  N4 --> EX["Worked example"]
  EX --> Q["Exercise EX109"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["37.1 Presenting evidence"]
  N2["37.2 Challenges"]
  N3["37.3 Improving the evaluation"]
  N4["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX109"]

[Learning outcomes]
    |
    v
[37.1 Presenting evidence]
    |
    v
[37.2 Challenges]
    |
    v
[37.3 Improving the evaluation]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX109]

Learning outcomes

You will (1) present evidence to peers; (2) respond to challenge; (3) improve the evaluation.

37.1 Presenting evidence

In a university seminar, Elena presents her capstone evidence to classmates and two lecturers. She shows her measures, data sources, test results and early after-period figures, and explains her limitations. The aim is not to impress but to find weaknesses before her final evaluation. She prepares by checking every number against its source and making sure each chart shows counts as well as percentages, so the audience can judge the evidence themselves. She also brings copies of her evaluation plan, so the audience can check that the analysis follows what was agreed in advance.

37.2 Challenges

A classmate asks whether multi-factor coverage rose simply because twelve unused accounts were disabled, shrinking the base. A lecturer asks whether one quiet month for leavers proves anything. Both are fair. Elena checks: disabling unused accounts raised coverage by about two points, so most of the rise came from enrolment. On leavers, she agrees one month is too little and extends her after period to three full months, as her plan required. She thanks both questioners and writes down each challenge in her project log before responding with evidence.

37.3 Improving the evaluation

The seminar improves her evaluation. She adds a note separating coverage changes due to enrolment from those due to account clean-up, reports leavers over the full three months and adds a short section on alternative explanations, such as a new IT staff member joining. She thanks participants and records changes in her project log. Welcoming challenge, rather than defending every point, is a sign of mature evidence work. Her supervisor reviews the changes and confirms that they follow the original plan rather than altering it after results were seen, which protects the credibility of the final evaluation.

Worked example

Seminar challenges lead Elena to separate enrolment from clean-up effects on coverage (about two points from clean-up) and to report leavers over three full months.

Synthetic case. Seminar participants and figures are synthetic.

Errors and limits

Ignoring changes in a measure's base can credit an intervention with an improvement that came from clean-up.

Exercises

• EX109 — Base challenge. What did the classmate's question concern? Rubric: 3 = coverage rising because unused accounts shrank the base;; 2 = base;; 1 = cost;; 0 = none.
• EX110 — Clean-up effect. How much of the coverage rise came from clean-up? Rubric: 3 = about two points;; 2 = small;; 1 = all;; 0 = none.
• EX111 — Leaver period. How was the leaver evaluation period fixed? Rubric: 3 = extended to three full months;; 2 = longer;; 1 = one month;; 0 = none.

Synthesis

Presenting evidence to challenge and responding openly makes the final evaluation stronger.

Continue studying — chapter 37 of 40

← Chapter 36 (draft) · Course chapter index · Chapter 38 (draft) →