Chapter 42 — Application and software supply chain security: applied research
Lesson map
Lesson 42 published. Map, example, EX124.
flowchart TD
N0["Learning outcomes"]
N1["42.1 The question"]
N2["42.2 Design"]
N3["42.3 Results"]
N4["42.4 Interpretation"]
N5["42.5 Limitations"]
N6["42.6 Reporting"]
N7["Ethics of the study"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX124"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["42.1 The question"]
N2["42.2 Design"]
N3["42.3 Results"]
N4["42.4 Interpretation"]
N5["42.5 Limitations"]
N6["42.6 Reporting"]
N7["Ethics of the study"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX124"]
[Learning outcomes]
|
v
[42.1 The question]
|
v
[42.2 Design]
|
v
[42.3 Results]
|
v
[42.4 Interpretation]
|
v
[42.5 Limitations]
|
v
[42.6 Reporting]
|
v
[Ethics of the study]
|
v
[Worked example] --> [Exercise EX124]
Learning outcomes
You will (1) formulate an applied research question from engineering practice; (2) design a small study with synthetic or approved data; (3) analyse and interpret results; (4) report limitations.
42.1 The question
Kwame asks: does requiring a written justification for new dependencies reduce the number added without lowering developer satisfaction? This connects to findings that dependency growth spreads risk (Zimmermann et al., 2019; Decan et al., 2018).
A good applied question is narrow enough to answer with available data and matters to the people who will use the answer. Kwame's question has two outcomes, so the study can show whether a benefit comes with a hidden cost to developers. Linking it to published research shows why the answer matters beyond Coastline.
42.2 Design
A before-and-after comparison over two six-month periods at Coastline, with dependency additions counted from the repository and satisfaction measured by a short anonymous survey. The study uses only repository metadata and anonymous survey answers, with Coastline's approval.
Comparing two periods is the simplest design possible inside one company. Counting additions from repository history gives objective data that cannot be skewed by memory. The survey captures how developers feel, which the counts cannot. Approval from Coastline and the use of metadata only keep the study within what developers agreed to.
42.3 Results
New direct dependencies fell from 34 to 19. Survey satisfaction with the dependency process stayed similar (3.8 to 3.7 on a 5-point scale). Justifications most often cited "needed for a new feature" and "replacing an abandoned package".
The fall from 34 to 19 is a drop of about 44%, while the satisfaction score barely changed. The common justifications show that most additions had a clear purpose, which suggests the step filtered out casual additions rather than blocking needed ones. Only direct dependencies are counted, since transitive ones follow from them.
42.4 Interpretation
The justification step seems to reduce casual additions without much frustration. But other changes happened at the same time, such as the new checklist and fewer new features in the second period.
Kwame states both explanations openly. The justification step could explain the fall, but so could fewer new features, which would reduce the need for dependencies regardless of the process. Without a comparison group, the data cannot separate these causes, so Kwame describes the result as consistent with a benefit rather than proof of one.
42.5 Limitations
One company, two periods, no control group; survey response rate 60%. The findings are suggestive, not conclusive.
Each limit weakens the conclusion in a specific way. A single company may not represent others; two periods may reflect seasonal work patterns; with no control group, other changes cannot be ruled out; and the forty percent who did not answer may feel differently from those who did. Stating these lets readers judge how far to trust the findings and suggests how a better study could be designed, such as the staged rollout Kwame proposes in the worked example.
42.6 Reporting
Kwame writes a short paper for a student conference and a summary for Coastline's engineering blog, using synthetic examples.
Different readers need different forms. The conference paper describes methods and limits in full so other researchers can judge or repeat the work. The blog summary tells engineers what was tried and what seemed to help, in plain language. Both use synthetic examples, so no real code, names or internal details from Coastline appear. Coastline reviews both before publication to check nothing confidential slipped through.
Ethics of the study
Developers were told about the survey's purpose, that it was anonymous and optional, and that results would not be used to judge individuals. Kwame reports only totals, so no developer's dependency choices can be identified.
Data handling
Survey data are deleted after analysis.
Worked example
Ingrid asks whether to roll the justification step out to other Coastline apps. Kwame suggests doing so in stages, measuring the same outcomes, which would turn the rollout into a stronger study.
Synthetic case. Study data are synthetic.
Errors and limits
Before-and-after studies are vulnerable to time trends; staged rollouts improve evidence.
Exercises
- EX124 — Research question. State Kwame's research question. Rubric: 3 = justification reduces additions without lowering satisfaction; 2 = partial; 1 = vague; 0 = none.
- EX125 — Results. Give the two main results. Rubric: 3 = 34 to 19 additions; satisfaction 3.8 to 3.7; 2 = one; 1 = vague; 0 = none.
- EX126 — Staged rollout. Why roll out in stages? Rubric: 3 = stronger evidence; compare groups; 2 = partial; 1 = vague; 0 = none.
Synthesis
Applied research turns engineering changes into evidence, with honest limits.
Continue studying — chapter 42 of 50
← Chapter 41 (draft) · Course chapter index · Chapter 43 (draft) →