OpenCyber AcademyLearning outcomes
You will (1) move from a topic to a researchable question with a clear unit of analysis; (2) run a systematic, documented literature search across security and machine-learning venues and preprint servers; (3) extract evidence into a comparison matrix; (4) judge source quality, including preprints and vendor material; and (5) write a short, honest state-of-the-art section.
1.1 From topic to question
"Privacy of AI" is a topic. "Does deduplicating training records reduce membership-inference advantage against a fine-tuned text classifier, and at what accuracy cost?" is a question. A researchable question names a system (fine-tuned classifier), an intervention or factor (deduplication), an outcome (membership-inference advantage; accuracy) and implicitly a comparison (with versus without). The acronym PICO from evidence-based medicine (population, intervention, comparison, outcome) is a useful template. Check feasibility early: can you obtain or synthesise data, do you have compute, can the work be done in the time, and is it ethical and authorised?
1.2 A documented search
AI-security literature is spread across security conferences (IEEE S&P, USENIX Security, ACM CCS, NDSS), machine-learning conferences (NeurIPS, ICML, ICLR) and arXiv preprints, plus frameworks from NIST, OWASP and MITRE. Write a search protocol: databases (e.g., DBLP, Google Scholar, arXiv, ACM Digital Library), search strings, date range, inclusion and exclusion criteria, and the date you searched. Follow citations forwards and backwards from seed papers — for membership inference, Shokri et al. (arXiv:1610.05820) is the obvious seed. Log counts at each stage (found, screened, included) so a reader could repeat the search. This is a light version of systematic-review practice; it protects you from citing only what you already knew.
1.3 Extracting evidence
Build a matrix with one row per paper and columns for: threat model (attacker knowledge and access), model type and size, data, metric, main result, defence tested, adaptive evaluation, code availability and limitations. For privacy work, record whether the paper reports true-positive rate at low false-positive rate or only average accuracy/AUC; the former matters because an attack that confidently identifies a few individuals is a real harm even if its average performance looks modest. The matrix will reveal disagreements and gaps that become your contribution.
1.4 Judging sources
Rank evidence by how checkable it is. Peer-reviewed papers with released code are strongest; preprints may be excellent but are unreviewed; vendor blogs and marketing are claims to verify, not evidence. Standards and frameworks (NIST AI 100-1, AI 100-2 E2025, ISO/IEC 42001) define vocabulary and expectations; they are not experimental results. When a preprint later appears in a venue, cite the final version if it differs. Never cite a paper you have not opened, and never invent a DOI: if you cannot find an identifier, cite the title, authors, year and URL you actually used.
1.5 Worked example (synthetic Harbourline case)
Leo's protocol searched DBLP, arXiv and Google Scholar on 2 February with strings combining "membership inference", "fine-tuning", "deduplication" and "memorization", 2016–2026, including empirical studies on classifiers or language models and excluding papers without a stated threat model. Of 212 hits, 61 survived title screening, 24 abstract screening and 17 full-text reading. The matrix showed that most studies reported AUC, few reported TPR at 0.1% FPR, and only three tested deduplication directly — all on generative language models, none on fine-tuned classifiers. Leo's state-of-the-art paragraph said so plainly and cited Carlini et al. (arXiv:2012.07805) for verbatim extraction as the motivating harm.
The membership-inference lab shows the core idea Leo will study: when a model effectively memorises its training records (k = 1), membership is detectable; when it generalises (larger k), the signal fades.
"""Lab M: membership inference against an overfit 1-nearest-neighbour 'model'
that returns a confidence score. Synthetic records only; teaching model."""
import random, math
rng = random.Random(5)
def rec():
y = rng.random() < 0.5
return [rng.gauss(0.6 if y else -0.6, 1.0) for _ in range(4)], int(y)
members = [rec() for _ in range(150)]; nonmembers = [rec() for _ in range(150)]
def confidence(x, y, k):
# k-NN vote share for the true label, computed over the training set
d = sorted((sum((a-b)**2 for a, b in zip(x, m)), my) for m, my in members)
return sum(1 for _, my in d[:k] if my == y)/k
def auc(pos, neg):
s = sum((p > n) + 0.5*(p == n) for p in pos for n in neg)
return s/(len(pos)*len(neg))
for k in (1, 5, 25):
pos = [confidence(x, y, k) for x, y in members]
neg = [confidence(x, y, k) for x, y in nonmembers]
best = max(sum(p >= t for p in pos)/len(pos) - sum(n >= t for n in neg)/len(neg)
for t in sorted(set(pos + neg)))
print(f"k={k:2d} attack_auc={auc(pos, neg):.3f} best_advantage={best:.3f}")
Expected output:
k= 1 attack_auc=0.600 best_advantage=0.200
k= 5 attack_auc=0.521 best_advantage=0.073
k=25 attack_auc=0.520 best_advantage=0.060
1.6 Practice activities
- Write your topic, then three candidate questions in PICO form; pick one and justify feasibility. 2. Write a search protocol and run it, logging counts. 3. Build a ten-row evidence matrix. For the questions, name the system, the intervention or factor, the comparison and the outcome, as Leo's question does. Judge feasibility on data, compute, time and ethics, and drop any question that fails one of them. For the search, record databases, strings, dates and counts at each stage, so someone else could repeat it. For the matrix, include whether each paper reports TPR at low FPR.
1.7 Errors and limits
Five errors recur, and each weakens the dissertation from the start:
- Questions too broad to answer in a dissertation; a broad question cannot be answered in the time.
- Searches that stop at the first page of a search engine; a search that stops early misses work such as the three deduplication studies Leo found.
- Treating a vendor blog as evidence; a vendor blog is a claim to verify.
- Citing papers you have not read, or fabricating identifiers; uncited or invented identifiers damage trust in every other citation.
- Reporting only averages for privacy attacks; averages hide the few individuals an attack identifies confidently, which is the harm Leo studies.
Exercises
- EX001 — PICO. Write a PICO question about prompt-injection defences. Rubric: 3 = all four elements specific and feasible; 2 = one vague; 1 = two missing; 0 = topic only.
- EX002 — Search protocol. List six elements of a documented search protocol. Rubric: 3 = sources, strings, dates, criteria, search date, counts; 2 = four; 1 = two; 0 = none.
- EX003 — Matrix columns. Name six matrix columns for an AI-security literature review and why one matters. Rubric: 3 = six relevant plus justification; 2 = six; 1 = three; 0 = none.
- EX004 — Low-FPR. Why report TPR at low FPR for membership inference? Rubric: 3 = harm comes from confident identification of some individuals; averages hide it; 2 = partial; 1 = vague; 0 = wrong.
- EX005 — Source ranking. Rank: vendor blog, arXiv preprint with code, peer-reviewed paper without code, NIST framework — and explain the framework's role. Rubric: 3 = sensible ranking plus framework as vocabulary not results; 2 = ranking only; 1 = vague; 0 = none.
- EX006 — Lab reading. What AUC does the MIA lab report at k = 1 and at k = 5? Rubric: 3 = 0.600 and 0.521 with interpretation; 2 = numbers only; 1 = one number; 0 = wrong.
- EX007 — Missing identifier. You cannot find a DOI for a workshop paper. How do you cite it? Rubric: 3 = authors, title, venue/year, URL actually used, no invented DOI; 2 = partial; 1 = vague; 0 = invents one.
Synthesis
A good dissertation starts with a narrow, feasible question and a search that someone else could repeat. The evidence matrix turns reading into a map of gaps, and honest sourcing keeps the map trustworthy.