Chapter 34 — Comparative project
Lesson map
Lesson 34 published. Map, example, EX100.
flowchart TD
N0["Learning outcomes"]
N1["34.1 The project"]
N2["34.2 Criteria"]
N3["34.3 Evidence"]
N4["34.4 Findings"]
N5["34.5 Balanced conclusions"]
N6["34.6 Fairness"]
N7["34.7 Recommendations"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX100"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["34.1 The project"]
N2["34.2 Criteria"]
N3["34.3 Evidence"]
N4["34.4 Findings"]
N5["34.5 Balanced conclusions"]
N6["34.6 Fairness"]
N7["34.7 Recommendations"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX100"]
[Learning outcomes]
|
v
[34.1 The project]
|
v
[34.2 Criteria]
|
v
[34.3 Evidence]
|
v
[34.4 Findings]
|
v
[34.5 Balanced conclusions]
|
v
[34.6 Fairness]
|
v
[34.7 Recommendations]
|
v
[Worked example] --> [Exercise EX100]
Learning outcomes
You will (1) compare SOC practices across organisations or approaches; (2) choose fair criteria; (3) gather comparable evidence; (4) draw balanced conclusions.
34.1 The project
Lucas and two classmates compare how three universities' SOCs handle phishing reports: Riverton (internal student-staffed), a partner university using a managed provider, and a third with a regional shared SOC.
Each of the three models is common among universities, which made the comparison useful to others. The team chose phishing reports because all three SOCs handle them and because Riverton's campaign gave them a detailed understanding of what good handling should achieve. Contacts at the other two came through Dr. Kim.
34.2 Criteria
They compare time from report to acknowledgement, whether reporters get feedback, automation of mailbox searches, and staffing model, using the same definitions for each.
Agreeing definitions mattered: one SOC counted acknowledgement as an automatic reply, another as a person reading the report. The team chose a person's first action, and each SOC recalculated its figure on that basis. Without the shared definition, the fastest SOC might simply have been the one with an automatic reply. Feedback was defined as a personal reply telling the reporter what happened, so an automatic acknowledgement did not count. Automation was judged by whether a reported message could be searched and removed from all mailboxes without manual steps.
34.3 Evidence
Each SOC shares anonymised figures for one term and a description of its process. No personal data is exchanged.
Each SOC provided counts and medians rather than individual reports, so no reporter or recipient could be identified. Process descriptions were checked against a short questionnaire the team wrote, which asked the same questions of each SOC in the same order. That kept answers comparable. The team also asked each SOC to describe one recent phishing report from start to finish, which brought the process descriptions to life and revealed steps the questionnaire had missed, such as manual approval before a purge at the regional SOC.
34.4 Findings
The managed provider acknowledges fastest (median 6 minutes) but rarely gives reporters feedback; Riverton is slower (median 22 minutes in staffed hours) but replies to every reporter; the regional SOC automates mailbox purges best.
Riverton's slower median reflects staffed hours: reports that arrive overnight wait until the morning, while the managed provider staffs nights. The regional SOC's purges remove reported phishing from every mailbox within minutes, which Riverton did by hand during the campaign. Each strength has a cost the team recorded alongside it.
34.5 Balanced conclusions
No model is best on all criteria. Feedback to reporters, which Riverton does well, encourages reporting; speed and automation, which others do well, reduce exposure.
The interviews in Lucas's study connect these: students said they did not know how to report or assumed IT already knew. Personal feedback addresses both by teaching reporters that reports matter. Speed and purges limit how many people click. Both matter, so a balanced conclusion recommends keeping one and adding the other. The team avoided ranking the three overall.
34.6 Fairness
Differences in size and threat exposure limit comparison; the project states these.
The managed provider serves a university twice Riverton's size, and the regional SOC serves several institutions, so their volumes and staff differ. The team reported medians and percentages rather than raw counts, and stated the differences in size beside each figure so readers could judge comparability themselves. Threat exposure also differs: the regional SOC's members include a medical school that receives targeted phishing more often. The team could not adjust for that, so they described it and advised readers to compare processes more than raw speeds.
34.7 Recommendations
Riverton adopts automated purges of reported phishing from all mailboxes, with feedback kept.
Riverton tested automated purges on a simulated phishing message in a test mailbox group before enabling them. The change keeps the personal reply to each reporter, which the team found was Riverton's distinctive strength. The purge removes copies that recipients have not yet opened, reducing exposure in the next campaign. Feedback to reporters remains a person's reply, written by the analyst who handled the report. The team also suggested that Riverton ask the managed provider's university how it handles feedback at night, since that was the one gap all three shared.
Process of the project
The team agreed definitions with all three SOCs before collecting figures, met them online twice and sent each a draft of its own section for factual correction. This avoided misreporting another organisation's practice and built goodwill for future sharing. Each SOC approved its section before the report was submitted.
Limits of anonymised figures
Anonymised medians hide spread: a median of 6 minutes may include some reports waiting hours. The team asks each SOC for 90th percentiles too.
Worked example
The team's comparison table lists each criterion with values and sources. Their conclusion: "Riverton should keep personal feedback to reporters and adopt automated purges; a managed provider might improve night-time speed but would need a feedback agreement."
Synthetic case. Universities and figures are invented.
Errors and limits
Three cases cannot establish general rules.
Exercises
- EX100 — Comparison criteria. List the four comparison criteria. Rubric: 3 = acknowledgement time, reporter feedback, purge automation, staffing; 2 = three; 1 = one; 0 = none.
- EX101 — Trade-off finding. What trade-off did the comparison reveal? Rubric: 3 = speed vs feedback; no model best everywhere; 2 = partial; 1 = one best; 0 = none.
- EX102 — Adopted change. What did Riverton adopt from the comparison? Rubric: 3 = automated purges, keeping feedback; 2 = partial; 1 = vague; 0 = none.
Synthesis
Fair comparisons use shared criteria and comparable evidence, and lead to balanced, specific improvements.
Continue studying — chapter 34 of 40
← Chapter 33 (draft) · Course chapter index · Chapter 35 (draft) →