Chapter 18 — Research methods I
Lesson map
Lesson 18 published. Map, example, EX052.
flowchart TD
N0["Learning outcomes"]
N1["18.1 Question"]
N2["18.2 Hypothesis"]
N3["18.3 Data"]
N4["18.4 Analysis"]
N5["18.6 Ethics"]
N6["18.7 Report"]
N7["Planning the study"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX052"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["18.1 Question"]
N2["18.2 Hypothesis"]
N3["18.3 Data"]
N4["18.4 Analysis"]
N5["18.6 Ethics"]
N6["18.7 Report"]
N7["Planning the study"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX052"]
[Learning outcomes]
|
v
[18.1 Question]
|
v
[18.2 Hypothesis]
|
v
[18.3 Data]
|
v
[18.4 Analysis]
|
v
[18.6 Ethics]
|
v
[18.7 Report]
|
v
[Planning the study]
|
v
[Worked example] --> [Exercise EX052]
Learning outcomes
You will (1) design a small study of StudyHub's configuration; (2) form a question and hypothesis; (3) collect data ethically; (4) analyse and report.
18.1 Question
Samuel asks: "Are StudyHub images built from the shared base image less likely to contain high-severity vulnerabilities than images built from public bases?"
The question is narrow and answerable with data the team already has: scan reports and the base image each team chose. It also matters to a real decision, since the team was considering whether to require the shared base. A broader question such as "are our images secure?" could not be answered in a month. Dr Jensen approved the question.
18.2 Hypothesis
Images from the shared base, rebuilt weekly, have fewer high-severity findings.
Writing the hypothesis down before looking at the data commits Samuel to a direction of effect. If public-base images turned out to have fewer findings, he would report that rather than rephrasing the question. The weekly rebuild is part of the hypothesis because it is the mechanism by which the shared base stays patched. Samuel also states what result would count against the hypothesis: no difference in the share of images with high findings, or a higher share among shared-base images. Stating it in advance makes the test honest.
18.3 Data
Scan reports for 40 images over one month: 22 from the shared base, 18 from public bases.
Forty images over a month is small but covers every StudyHub image in use. The scanner version stayed the same throughout, so differences cannot come from a tool update. The groups are unequal in size, which Samuel handles by reporting proportions as well as counts. Images were counted once each, using the most recent scan in the month, so frequently rebuilt images did not count more than others. Samuel records which scan was used for each image in his evidence log, with the scanner version and date.
18.4 Analysis
Median high-severity findings: 0 for shared-base images, 3 for public-base images. Of public-base images, 12 of 18 had at least one high finding versus 3 of 22 shared-base images.
In proportions, two thirds of public-base images had at least one high finding, against about one in seven shared-base images. The medians tell a similar story. Samuel reports both counts and proportions so readers can check the arithmetic and see the size of each group. He does not compute a formal test, since the groups were not randomly formed and a p-value would suggest more certainty than the design supports. Instead, he shows the counts in a small table with the two bases as rows and presence or absence of high findings as columns.
18.5 Limits
Images were not randomly assigned to bases; teams using public bases may maintain their images less. Shu, Gu and Enck (2017) also found many vulnerable images on public registries, which is consistent but does not prove cause here.
Samuel states the confounding directly: the difference may come from the base image, from how carefully each team maintains its images, or from both. The study cannot separate them. The recommendation is still reasonable, because moving to the shared base also brings weekly rebuilds, which addresses maintenance as well.
18.6 Ethics
Data are scan reports about images, not people; team names are not reported.
Not naming teams avoids turning the study into a league table, which could make teams defensive about adopting the shared base. The data contain no personal information, so no ethics review was needed beyond Dr Jensen's approval of the plan. Scan reports can still reveal which services are vulnerable, so the raw data stay in the team's restricted repository. Only aggregate figures appear in the report and slides, and Samuel's local copies are deleted after the analysis, as his access diary records.
18.7 Report
A three-page report with method, results, limits and recommendation: move all images to the shared base.
The recommendation follows from the result and from the practical fact that the shared base is already maintained. The report's limits paragraph sits beside the main finding, so readers who stop early still see that the evidence is observational. It names the decision the team must make, the evidence for it and what would make the recommendation wrong, such as the shared base itself falling behind on patches. Tom read it in one sitting, which was the aim.
Planning the study
Before collecting data, Samuel writes a one-page plan: question, hypothesis, data source, measures, analysis and limits. Dr Jensen approves it. Writing the plan first prevents changing the question after seeing the results. He also decides in advance to report all images, including the ones that do not fit the hypothesis, and to describe how scan tool versions were kept the same through the month.
Sharing results
Results are presented to the cloud team in ten minutes, with the limits shown on the same slide as the main finding.
Worked example
Tom accepts the recommendation and adds a pipeline check that warns when an image uses a non-approved base. Within two months, 34 of 40 images use the shared base.
Synthetic case. Data are synthetic.
Errors and limits
A correlation in observational data suggests but does not prove that the base causes the difference.
Exercises
- EX052 — Hypothesis. State the hypothesis. Rubric: 3 = shared base images have fewer high findings; 2 = partial; 1 = vague; 0 = none.
- EX053 — Results. What were the shares with high findings? Rubric: 3 = 12/18 public vs 3/22 shared; 2 = partial; 1 = vague; 0 = none.
- EX054 — Limit. Why can't the study prove cause? Rubric: 3 = not randomly assigned; 2 = partial; 1 = vague; 0 = none.
Synthesis
Small, careful studies guide practical decisions.
Continue studying — chapter 18 of 40
← Chapter 17 (draft) · Course chapter index · Chapter 19 (draft) →