Chapter 5 — Proposal and pilot
Lesson map
Lesson 05 published. Map, example, EX027.
flowchart TD
N0["Learning outcomes"]
N1["5.1 Proposal"]
N2["5.2 Role matrix"]
N3["Lab — matrix check"]
N4["5.3 Lab result"]
N5["5.4 Pilot lessons"]
N6["5.5 Developer feedback"]
N7["Timeline"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX027"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["5.1 Proposal"]
N2["5.2 Role matrix"]
N3["Lab — matrix check"]
N4["5.3 Lab result"]
N5["5.4 Pilot lessons"]
N6["5.5 Developer feedback"]
N7["Timeline"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX027"]
[Learning outcomes]
|
v
[5.1 Proposal]
|
v
[5.2 Role matrix]
|
v
[Lab — matrix check]
|
v
[5.3 Lab result]
|
v
[5.4 Pilot lessons]
|
v
[5.5 Developer feedback]
|
v
[Timeline]
|
v
[Worked example] --> [Exercise EX027]
Learning outcomes
You will (1) write a dissertation proposal; (2) pilot the role-matrix test approach on one feature; (3) measure matrix mismatches in a pilot; (4) refine the method before the full study.
5.1 Proposal
Background, question, design, ethics, alternative explanations, timeline and risks, in ten pages approved by Dr Lee.
Ten pages forces each part to be specific. The alternative-explanations part lists pentest effort, the framework change, the senior developer and tester focus, each with the data that will check it. The timeline fixes when the before and after periods begin. The risks part names the main threat to the design: a small number of findings. Dr Lee's approval also fixes the analysis before data collection, which is what later lets the examiners see that the framework separation was planned, not added after results arrived.
5.2 Role matrix
For each role (customer, support, finance) and each endpoint, the matrix states whether access is allowed. Tests call each endpoint as each role in a test environment and compare the result with the matrix. This follows the WSTG's authorisation testing guidance, applied to Tallyline's own system with permission.
Three roles keep Tallyline's matrix small enough to review by hand, and the six decisions in the lab are a small part of it. The matrix's value comes from being complete: a role or endpoint missing from it is never tested, which is why Sofia added the route completeness check after the pilot.
Lab — matrix check
"""Compare a synthetic app's actual access decisions with the intended role matrix."""
intended = {("customer", "GET /invoices/own"): True, ("customer", "GET /invoices/other"): False,
("customer", "POST /refunds"): False, ("support", "GET /invoices/other"): True,
("support", "POST /refunds"): False, ("finance", "POST /refunds"): True}
actual = {("customer", "GET /invoices/own"): True, ("customer", "GET /invoices/other"): True,
("customer", "POST /refunds"): False, ("support", "GET /invoices/other"): True,
("support", "POST /refunds"): True, ("finance", "POST /refunds"): True}
bad = [(k, intended[k], actual[k]) for k in intended if intended[k] != actual[k]]
for (role, ep), want, got in bad:
print(f"MISMATCH {role:8} {ep:22} intended={want} actual={got}")
print(f"{len(bad)} of {len(intended)} decisions wrong")
Run with python3 lab_authz.py. Verified output:
MISMATCH customer GET /invoices/other intended=False actual=True
MISMATCH support POST /refunds intended=False actual=True
2 of 6 decisions wrong
5.3 Lab result
Two of six decisions are wrong: a customer can read another business's invoice (intended no), and support can issue refunds (intended no). Both match the kinds of flaw that had recurred.
The two wrong decisions mirror the history that started the study: the March flaw let a customer view another business's invoice, and a similar flaw reappeared in refunds. A test suite that encodes the intended matrix would have failed on both before release, which is the mechanism Sofia's study puts to the test.
5.4 Pilot lessons
The matrix initially missed the "export" endpoint; Sofia adds a check that lists all routes from the framework and flags any not in the matrix, addressing the completeness critique from chapter 2.
The completeness check later found three more unmapped routes, all internal health checks, which were added as "deny for all user roles". The two pilot mismatches were fixed within a day. Each addition to the matrix is recorded, which is why the examiners later asked for route and role counts over time.
5.5 Developer feedback
Developers found failing tests clear because each names the role, endpoint and intended decision, avoiding the usability problems Johnson et al. (2013) reported for static tools.
The clarity matters because tests that developers distrust get disabled. A failing test saying "support, POST /refunds, intended deny, got allow" tells the developer exactly what to fix and where, without anyone needing to interpret a scanner's output. That is part of why developers later said the tests caught problems before review. Because each test is a single call compared against one matrix cell, a developer can fix a failure without knowing how the rest of the suite works.
Timeline
The full study runs for six months after the pilot, covering six releases. A risk register lists what could delay it, such as release schedule changes or a pentest supplier change, with fallback plans.
Approval
Dr Lee and Tallyline's engineering manager both sign the proposal, which records what each party has agreed to provide.
Test environment
Tests run against a seeded test database with synthetic businesses only.
Worked example
The two pilot mismatches are fixed within a day. The completeness check finds three more unmapped routes, all internal health checks, which are added as "deny for all user roles".
Synthetic case. Pilot data are synthetic.
Errors and limits
A test environment may differ from production configuration.
Exercises
- EX027 — Proposal sections. List five proposal sections. Rubric: 3 = background, question, design, ethics, alternatives, timeline, risks (five); 2 = three; 1 = one; 0 = none.
- EX028 — Matrix meaning. What does the role matrix state? Rubric: 3 = per role and endpoint, allowed or not; 2 = partial; 1 = vague; 0 = none.
- EX029 — Lab mismatches. What mismatches did the lab find? Rubric: 3 = customer reads other invoice; support issues refunds; 2 = one; 1 = vague; 0 = none.
- EX030 — Completeness fix. How was matrix completeness addressed? Rubric: 3 = list framework routes; flag unmapped; 2 = partial; 1 = vague; 0 = none.
- EX031 — Clear failures. Why did developers find failures clear? Rubric: 3 = named role, endpoint, intended decision; 2 = partial; 1 = vague; 0 = none.
- EX032 — Health checks. How were the unmapped health checks handled? Rubric: 3 = deny for all user roles; 2 = partial; 1 = vague; 0 = none.
Synthesis
Pilot the intervention, find its blind spots and fix them before the main study.
Continue studying — chapter 05 of 08
← Chapter 04 (draft) · Course chapter index · Chapter 06 (draft) →