Chapter 33 — Research methods II

Chapter 33 — Research methods II

Lesson map

Lesson 33 published. Map, example, EX097.

flowchart TD
  N0["Learning outcomes"]
  N1["33.1 Evaluation design"]
  N2["33.2 Choosing measures"]
  N3["33.3 Analysis plan"]
  N4["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4
  N4 --> EX["Worked example"]
  EX --> Q["Exercise EX097"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["33.1 Evaluation design"]
  N2["33.2 Choosing measures"]
  N3["33.3 Analysis plan"]
  N4["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX097"]

[Learning outcomes]
    |
    v
[33.1 Evaluation design]
    |
    v
[33.2 Choosing measures]
    |
    v
[33.3 Analysis plan]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX097]

Learning outcomes

You will (1) design a before-and-after evaluation; (2) choose measures and comparison; (3) plan analysis in advance.

33.1 Evaluation design

For her capstone, Elena plans to evaluate whether the new identity processes improved Greenhill's position. She uses a before-and-after design: measure key indicators for three months before changes and three months after. Because there is no separate comparison group at a small organisation, she records other events that could affect results, such as staff turnover or a new system, so she can discuss alternative explanations honestly rather than assuming every improvement came from her changes. The plan is agreed with her supervisor before any change is made.

33.2 Choosing measures

Elena chooses five measures already defined in earlier chapters: multi-factor coverage, leavers disabled within one day, shared accounts, administrator accounts and unused paid licences. Each has a definition, a data source and a target. She avoids adding new measures after seeing results, which would let her pick whichever looked best. She also decides in advance how she will treat missing data, such as a month when an export failed, so that choices are not made to favour a particular outcome.

33.3 Analysis plan

The analysis plan states that for each measure she will report before and after values with counts, the change and whether the target was met. With small numbers, she will not use statistical tests that need large samples; instead she will describe changes carefully and discuss possible alternative explanations. Her supervisor approves the plan before the after period begins. Writing it down first is what makes the evaluation trustworthy, because the method was fixed before the answer was known. The approved plan is stored with her evidence.

Worked example

Elena's plan fixes five measures, three-month periods before and after, a record of other events and a descriptive analysis approved before changes go live.

Synthetic case. Measures, periods and plans are synthetic.

Errors and limits

Choosing measures after seeing results lets an evaluation report whichever looks best.

Exercises

• EX097 — Comparison limit. Why does Elena record other events during the evaluation? Rubric: 3 = no comparison group, so alternatives must be discussed;; 2 = alternatives;; 1 = habit;; 0 = none.
• EX098 — Five measures. List the five evaluation measures. Rubric: 3 = MFA coverage, leavers within a day, shared accounts, admin accounts, unused licences;; 2 = four;; 1 = two;; 0 = none.
• EX099 — Statistical caution. Why avoid tests needing large samples? Rubric: 3 = numbers are small;; 2 = small n;; 1 = cost;; 0 = none.

Synthesis

A planned before-and-after evaluation with fixed measures and honest discussion of alternatives gives credible results.

Continue studying — chapter 33 of 40

← Chapter 32 (draft) · Course chapter index · Chapter 34 (draft) →