## Chapter 24 — Applied research methods ### Lesson map Lesson 24 published. Map, example, EX070. ```mermaid flowchart TD N0["Learning outcomes"] N1["24.1 Question"] N2["24.2 Design"] N3["24.3 Measures"] N4["24.4 Ethics"] N5["24.5 Results"] N6["24.6 Interpretation"] N7["24.7 Reporting"] N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 N7 --> EX["Worked example"] EX --> Q["Exercise EX070"] ``` ### Logical drawing ```mermaid flowchart LR N0["Learning outcomes"] N1["24.1 Question"] N2["24.2 Design"] N3["24.3 Measures"] N4["24.4 Ethics"] N5["24.5 Results"] N6["24.6 Interpretation"] N7["24.7 Reporting"] N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX070"] ``` ```text [Learning outcomes] | v [24.1 Question] | v [24.2 Design] | v [24.3 Measures] | v [24.4 Ethics] | v [24.5 Results] | v [24.6 Interpretation] | v [24.7 Reporting] | v [Worked example] --> [Exercise EX070] ``` ### Learning outcomes You will (1) design a small applied study for an IT setting; (2) collect data ethically; (3) analyse simply and honestly; (4) report for practitioners. ### 24.1 Question "Does forwarding desktop security logs change how quickly the city can scope a lateral-movement incident?" The question is narrow on purpose. It asks about one change, central forwarding, and one outcome, the speed of scoping, in one setting. A broader question such as whether logging improves security could not be answered with the city's resources. A precise question also makes clear what the results can and cannot be used to claim. Jonah writes it down before designing anything. Renata agrees it with him so the result will be useful to the city. ### 24.2 Design Two tabletop-plus-technical exercises using synthetic lateral movement across test desktops: one with logs only on desktops (as before), one with central forwarding. Same team, same scenario variant, scenario order randomised. Using the same team and scenario variant controls for skill and difficulty, and randomising the order addresses the risk that whichever run comes second benefits from practice. With one team and two runs, though, randomisation can only choose which order happened; it cannot average out a learning effect. Jonah notes this design limit before running the study. ### 24.3 Measures Time to identify all affected test desktops; number correctly identified; analyst confidence (1–5). Time and the number of desktops identified are measured from exercise records; confidence is reported by the analysts. The first two are objective and comparable; the third shows how sure people felt, which matters because a fast answer given with low confidence may lead to more checking in a real incident. Each measure is defined before the exercise so it cannot be changed after the results are known. ### 24.4 Ethics Participants are staff; participation is voluntary, results are reported for the team, not individuals. Staff are told the study measures a process, not their performance, and that results will not appear in appraisals. Anyone can decline without giving a reason. The synthetic desktops hold no personal data, so the only personal information involved is who took part, which is kept separately from the results. Results are shared with participants before anyone else, and anyone may ask for their data to be removed until the note is published. ### 24.5 Results Without forwarding: 2 hours 10 minutes, 4 of 5 desktops identified. With forwarding: 25 minutes, 5 of 5. The difference is large: almost two hours faster and one more desktop found. The desktop missed in the first run had rotated its log during the exercise, which is exactly the problem forwarding is meant to solve. Jonah reports the raw figures rather than a percentage improvement, because with two runs a percentage suggests more precision than exists. Confidence ratings rose as well, but they are reported separately as opinion. ### 24.6 Interpretation Forwarding greatly sped scoping in this exercise and found the desktop missed previously. Learning effects from the first run could inflate the second's speed; randomising order across two teams in future would help. The learning effect is a real concern: in the second run the team already knew the scenario's shape. Even so, the missed desktop in the first run is explained by log rotation, not by inexperience, which supports the conclusion independently of speed. Jonah separates these two lines of evidence in his note so readers can weigh them. ### 24.7 Reporting A one-page practitioner note with figures and limits, shared with county peers. The note opens with the question and the main result, then gives the design in two sentences, the figures, and the limits: one team, two runs, a possible learning effect and a synthetic scenario. It ends with what the city decided, to forward desktop logs, and an invitation to peers to repeat the study with their own teams. Peers receive the scenario without network details, as agreed with the city. ### Literature check Before the study, Jonah looks for guidance on centralised logging for investigations. NIST SP 800-86 discusses the value of logs from multiple sources for analysis, and ISO/IEC 27043 includes readiness processes. His study adds local evidence for the city rather than a general result, and he says so in the note's opening paragraph. ### Data handling Exercise records are kept with the scenario files and analysis spreadsheet, hashed and stored in the team share, so the county peers who read the note can request and repeat the study. ### Sharing results The county peers ask for the scenario so they can run it themselves; Jonah shares it with the city's approval, stripped of network details. ### Worked example Jonah notes that in the no-forwarding run, the missed desktop had rotated its log during the exercise's synthetic activity burst — the same retention problem seen in real incidents. > **Synthetic case.** The study is invented. ### Errors and limits One team and two runs give a demonstration, not a general measurement. ### Exercises - **EX070 — Measures.** List the three measures. *Rubric:* 3 = time, correct count, confidence; 2 = two; 1 = one; 0 = none. - **EX071 — Results.** Report both results. *Rubric:* 3 = 2h10/4 of 5 vs 25 min/5 of 5; 2 = partial; 1 = wrong; 0 = none. - **EX072 — Learning effect.** Why might the second run look better regardless? *Rubric:* 3 = learning from first run; randomise across teams; 2 = partial; 1 = vague; 0 = none. ### Synthesis Small applied studies give practitioners useful evidence when designed fairly and reported with limits. ### Continue studying — chapter 24 of 30 [← Chapter 23 (draft)](#ch-23) · [Course chapter index](../SUBJECTS.md) · [Chapter 25 (draft) →](#ch-25) ## Worked answers **EX070.** Time to identify all affected test desktops, how many were correctly identified, and analysts' confidence. **EX071.** Without forwarding: 2 hours 10 minutes and 4 of 5 desktops found. With forwarding: 25 minutes and all 5 found. **EX072.** The team learned the scenario in the first run, which could speed the second; randomising order across two teams would control for it.