Chapter 8 — Independent review and defense
Lesson map
Lesson 08 published. Map, example, EX045.
flowchart TD
N0["Learning outcomes"]
N1["8.1 Independent review"]
N2["8.2 Defense"]
N3["8.3 Questions"]
N4["8.4 Examiner advice"]
N5["8.5 Dissemination"]
N6["8.6 Next steps"]
N7["Lessons"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX045"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["8.1 Independent review"]
N2["8.2 Defense"]
N3["8.3 Questions"]
N4["8.4 Examiner advice"]
N5["8.5 Dissemination"]
N6["8.6 Next steps"]
N7["Lessons"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX045"]
[Learning outcomes]
|
v
[8.1 Independent review]
|
v
[8.2 Defense]
|
v
[8.3 Questions]
|
v
[8.4 Examiner advice]
|
v
[8.5 Dissemination]
|
v
[8.6 Next steps]
|
v
[Lessons]
|
v
[Worked example] --> [Exercise EX045]
Learning outcomes
You will (1) obtain independent review of findings; (2) defend methods and claims orally; (3) respond to examiner questions with evidence; (4) plan dissemination.
8.1 Independent review
A security engineer from another company reviews the method and synthetic dataset, re-running the scripts. She confirms the numbers and suggests reporting the routes count over time.
Re-running the scripts on the synthetic dataset tests the method and code, not Tallyline's figures. The reviewer's suggestion, reporting the routes count over time, addresses a reader's obvious question: whether the matrix kept up with the app. Sofia adds it, and the examiners later ask for the same information. Her re-run used only the published scripts and the CC0 dataset, which is the same route any later reader would take.
8.2 Defense
Twenty minutes plus questions, with two examiners.
Sofia's talk follows the dissertation's order: question, design, competing explanations, results with uncertainty, limitations. She keeps the 0.14 ratio on the same slide as the framework-separated result, so the examiners never see one without the other. Sofia rehearses the talk with Dr Lee using the questions she expects most: the credibility of the 0.14 ratio with twenty findings, the framework change in release 8, and whether the matrix could miss tests. The route completeness check is ready on a backup slide, and the defence is held after Tallyline's legal team has approved the final text.
8.3 Questions
"Is the 0.14 ratio credible with 20 findings?" Sofia answers that it is imprecise, but direction and size are consistent with other evidence. "Would mutation of the matrix catch missing tests?" She explains the route completeness check.
Both answers rest on earlier work: the imprecision is acknowledged in the results chapter, and the route completeness check came from the pilot, where the matrix had missed the export endpoint. The examiners' questions test whether those choices were deliberate, and the dissertation shows that they were.
8.4 Examiner advice
Report how many routes and roles exist over time, so readers see the matrix scale.
The advice matches the independent reviewer's suggestion. Showing how many routes and roles the matrix covers at each release lets readers judge whether the approach would scale to their own apps, and whether the fall in findings happened while the matrix was growing or stable. Sofia adds a table to the results chapter with the number of routes and roles at each of the twelve releases, drawn from the framework's route list that the completeness check already produces. No new data collection is needed, because the check has been running since the pilot.
8.5 Dissemination
A conference short paper, a Tallyline engineering blog post with synthetic examples, and an OWASP community talk.
Each output reaches a different audience: researchers through the short paper, Tallyline's developers and customers through the blog, and practitioners through the OWASP talk. All three use the synthetic examples and the CC0 dataset, so none needs Tallyline's internal data. The blog post and the talk carry the same hedged wording as the dissertation: findings fell after the tests, and part of the fall is explained by the framework default. Using the CC0 synthetic dataset in all three means readers can reproduce the examples without any Tallyline data.
8.6 Next steps
Tallyline extends the matrix approach to its mobile API.
The mobile API serves the same customer, support and finance roles, so the matrix can be extended rather than rebuilt. The route completeness check will need a mobile equivalent, because the API's endpoints are listed differently from the web framework's routes. Mobile clients often call endpoints the web app never uses, so the first step is listing every mobile endpoint and adding it to the matrix with an intended decision for each role. The same three roles mean existing decisions carry over where endpoints are shared. Sofia's scripts can then compare mobile findings before and after, as the dissertation did for the web app.
Lessons
Sofia notes that early ethics and data agreements saved time later.
Preparation
Sofia rehearses twice with classmates, who ask the hardest questions they can think of. She prepares a backup slide for each limitation and for the framework-separated analysis, which examiners did ask about. Preparation lets her answer calmly and briefly, leaving more time for the discussion.
After the defense
Corrections are made within two weeks, and the final dissertation is deposited in the university repository with its synthetic dataset and scripts. Sofia also writes a short note for the OWASP community describing the role-matrix approach and its limits, so other teams can try it and report back. She plans to apply for a doctoral position to test the approach across several companies.
Worked example
The examiners award a distinction. One asks Sofia to share her synthetic dataset with a course on secure development, which she does under CC0.
Synthetic case. The defense and reviews are synthetic.
Errors and limits
A single defense cannot test generalisation; replication by others is needed.
Exercises
- EX045 — Reviewer role. What did the independent reviewer do? Rubric: 3 = re-ran scripts; confirmed; suggested routes count; 2 = partial; 1 = vague; 0 = none.
- EX046 — Ratio answer. How did Sofia defend the 0.14 ratio? Rubric: 3 = imprecise but consistent with other evidence; 2 = partial; 1 = vague; 0 = none.
- EX047 — Missing tests. How are missing tests caught? Rubric: 3 = route completeness check; 2 = partial; 1 = vague; 0 = none.
- EX048 — Dissemination. List three dissemination outputs. Rubric: 3 = short paper, blog, OWASP talk; 2 = two; 1 = one; 0 = none.
- EX049 — Next step. What did Tallyline do next? Rubric: 3 = extend to mobile API; 2 = partial; 1 = vague; 0 = none.
- EX050 — Dataset sharing. How was the dataset shared? Rubric: 3 = CC0 for a course; 2 = partial; 1 = vague; 0 = none.
Synthesis
Independent review and an honest defense turn a dissertation into shared knowledge.
Continue studying — chapter 08 of 08
← Chapter 07 (draft) · Course chapter index · Worked results ↓ · Next course via chapter index →