Chapter 10 — Research reading and critique

Lesson map

Lesson 10 published. Map, example, EX046.

flowchart TD
  N0["Learning outcomes"]
  N1["10.1 Reading critically"]
  N2["10.2 Evaluating methods and claims"]
  N3["10.3 Implications for practice"]
  N4["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4
  N4 --> EX["Worked example"]
  EX --> Q["Exercise EX046"]

Logical drawing

flowchart LR
  N0["Learning outcomes"]
  N1["10.1 Reading critically"]
  N2["10.2 Evaluating methods and claims"]
  N3["10.3 Implications for practice"]
  N4["Synthesis"]
  N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX046"]

[Learning outcomes]
    |
    v
[10.1 Reading critically]
    |
    v
[10.2 Evaluating methods and claims]
    |
    v
[10.3 Implications for practice]
    |
    v
[Synthesis]
    |
    v
[Worked example] --> [Exercise EX046]

Learning outcomes

You will (1) read research critically; (2) evaluate methods and claims; (3) identify implications for practice.

10.1 Reading critically

Nadia reads Perez and Livshits' 2021 study, which examined whether weaknesses reported by analysis tools in deployed contracts were actually exploited, and found that only a small fraction were. She identifies the question, data, method, results and the authors' stated limitations. She asks how exploitation was measured, what might be missed, and whether the contracts studied resemble the pilot's settlement contract. Critical reading is careful, not cynical: its aim is to understand how far a study's conclusions can be trusted and applied, using the evidence the authors present.

10.2 Evaluating methods and claims

The study's strength is its use of large-scale on-chain data rather than assumptions. Its limitations include the difficulty of detecting all exploitation from public data and the focus on public chains. Nadia also reads Durieux and colleagues' 2020 evaluation of analysis tools and Tsankov and colleagues' 2018 work on property-based analysis. Together they suggest that tool findings need contextual triage. She writes a critique noting that the claim is well supported for public contracts in the period studied, and less certain for private settlement contracts with different users and incentives.

10.3 Implications for practice

Nadia draws implications for the pilot. Tool output should be triaged by reviewers who understand the contract's context, not treated as a list of urgent defects. Equally, unexploited does not mean unexploitable, so findings in privileged functions should still be fixed. She proposes a triage rule: any finding affecting privileged roles or settlement amounts is fixed before deployment; others are assessed and documented. Professor Tanaka praises the balance between using evidence and avoiding over-generalisation. The rule is adopted by the joint operations group for future changes.

Worked example

Nadia critiques Perez and Livshits, combines it with Durieux and Tsankov, and proposes fixing all findings touching privileged roles or settlement amounts before deployment.

Synthetic case. The critique and rule are synthetic; Perez and Livshits (2021), Durieux et al. (2020) and Tsankov et al. (2018) are real sources.

Errors and limits

Reading unexploited as unexploitable leads to ignoring findings that still matter.

Exercises

• EX046 — Study finding. What did Perez and Livshits find?
  Show marking criteria

  Rubric: 3 = only a small fraction of reported weaknesses were exploited;; 2 = few exploited;; 1 = all exploited;; 0 = none.
• EX047 — Study strength. What is the study's main strength?
  Show marking criteria

  Rubric: 3 = large-scale on-chain data;; 2 = real data;; 1 = interviews;; 0 = none.
• EX048 — Study limitation. Name one limitation Nadia notes.
  Show marking criteria

  Rubric: 3 = hard to detect all exploitation from public data;; 2 = public chains only;; 1 = vague;; 0 = none.
• EX049 — Triage rule. What does the triage rule require?
  Show marking criteria

  Rubric: 3 = fix privileged-role and settlement findings before deployment;; 2 = fix critical ones;; 1 = ignore tools;; 0 = none.
• EX050 — Over-generalisation caution. Why fix findings even if unexploited elsewhere?
  Show marking criteria

  Rubric: 3 = unexploited does not mean unexploitable;; 2 = still possible;; 1 = tidiness;; 0 = none.

Synthesis

Critical reading turns research into balanced, context-aware practice rules.

Continue studying — chapter 10 of 12

← Chapter 09 (draft) · Course chapter index · Chapter 11 (draft) →