Chapter 10 — Research reading and critique
Lesson map
Lesson 10 published. Map, example, EX046.
flowchart TD
N0["Learning outcomes"]
N1["10.1 Reading with a purpose"]
N2["10.2 Assessing methods"]
N3["10.3 Classic critiques"]
N4["10.4 Synthesising findings"]
N5["10.5 Critique as a professional skill"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX046"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["10.1 Reading with a purpose"]
N2["10.2 Assessing methods"]
N3["10.3 Classic critiques"]
N4["10.4 Synthesising findings"]
N5["10.5 Critique as a professional skill"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX046"]
[Learning outcomes]
|
v
[10.1 Reading with a purpose]
|
v
[10.2 Assessing methods]
|
v
[10.3 Classic critiques]
|
v
[10.4 Synthesising findings]
|
v
[10.5 Critique as a professional skill]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX046]
Learning outcomes
You will (1) read research critically; (2) assess claims and methods; (3) relate findings to practice.
10.1 Reading with a purpose
Marta reads research to inform decisions, not to collect references. For route validation, she asks what is known about how often invalid results come from errors rather than attacks, and how enforcement affects reachability. For ingress filtering, she asks what is known about adoption and spoofed traffic. She reads abstracts first, then methods and results of the most relevant papers, and keeps notes on each paper's question, data, method, findings and limits. Purposeful reading is faster and produces notes she can use directly.
10.2 Assessing methods
A paper's conclusions are only as good as its methods. Marta checks where the data came from, whether the sample represents the networks she cares about, whether the measures match the claims, and whether alternative explanations were considered. Measurement studies of the internet often observe from a limited set of vantage points, which may miss behaviour elsewhere. She notes such limits in her summaries rather than discarding useful papers, since every study has limits and the question is whether they matter for her decision.
10.3 Classic critiques
Some older papers remain useful because they explain persistent problems. Sommer and Paxson's 2010 paper on machine learning for network intrusion detection explains why detection research often fails to transfer into operations: real traffic is diverse, errors are costly and evaluation datasets are unrealistic. Axelsson's 2000 paper on the base-rate fallacy shows why false alarms dominate when attacks are rare. Marta uses these to temper claims from vendors about automated detection of routing anomalies, asking how such tools perform on realistic, rare-event data.
10.4 Synthesising findings
Marta writes a short synthesis for each question, combining findings across papers and noting agreement, disagreement and gaps. Where research is thin, she says so and relies more on practitioner experience, clearly labelled. She avoids citing a paper for a claim it does not make, checking each citation against the paper itself. A good synthesis tells a reader what is reasonably known, what is uncertain and what that means for the decision at hand, which is more useful than a long list of summaries.
10.5 Critique as a professional skill
Critiquing research is not about finding fault; it is about judging how much weight evidence can bear. Marta practises in a seminar, where students critique each other's summaries. She learns to state criticisms specifically, such as "the sample includes only large networks, so results may not apply to small providers", and to suggest how a limit could be addressed. These habits apply equally to vendor reports and internal analyses, which deserve the same scrutiny as academic papers. She also learns to accept critique of her own work calmly.
Worked example
Marta's synthesis on route validation errors combines four sources, notes one gap and labels practitioner experience separately.
Synthetic case. Seminar and notes are synthetic; cited papers are real.
Errors and limits
Citing a paper for a claim it does not make misleads readers and damages credibility.
Exercises
- EX046 — Reading notes. List four items Marta records for each paper. Rubric: 3 = question, data, method, findings or limits;; 2 = three;; 1 = fewer;; 0 = none.
- EX047 — Vantage point limit. Why do internet measurement studies have vantage point limits? Rubric: 3 = observe from few locations, may miss behaviour elsewhere;; 2 = limited view;; 1 = vague;; 0 = none.
- EX048 — Base-rate relevance. How does Axelsson's paper apply to routing anomaly tools? Rubric: 3 = rare events make false alarms dominate;; 2 = false alarms;; 1 = not relevant;; 0 = none.
- EX049 — Specific criticism. Rewrite "the study is weak" as a specific criticism. Rubric: 3 = names the limit and its effect on applicability;; 2 = specific;; 1 = still vague;; 0 = none.
- EX050 — Thin research response. What should a synthesis do where research is thin? Rubric: 3 = say so, rely on labelled practitioner experience;; 2 = state it;; 1 = invent;; 0 = none.
Synthesis
Purposeful, method-aware reading and honest synthesis turn research into usable evidence.
Continue studying — chapter 10 of 12
← Chapter 09 (draft) · Course chapter index · Chapter 11 (draft) →