Chapter 12 — Practice portfolio and final review
Lesson map
Lesson 12 published. Map, example, EX056.
flowchart TD
N0["Learning outcomes"]
N1["12.1 Portfolio"]
N2["12.2 Principles to keep"]
N3["12.3 Readiness and gaps"]
N4["12.4 Continuing development"]
N5["12.5 Ethics"]
N6["12.5b Using the portfolio"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX056"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["12.1 Portfolio"]
N2["12.2 Principles to keep"]
N3["12.3 Readiness and gaps"]
N4["12.4 Continuing development"]
N5["12.5 Ethics"]
N6["12.5b Using the portfolio"]
N7["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX056"]
[Learning outcomes]
|
v
[12.1 Portfolio]
|
v
[12.2 Principles to keep]
|
v
[12.3 Readiness and gaps]
|
v
[12.4 Continuing development]
|
v
[12.5 Ethics]
|
v
[12.5b Using the portfolio]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX056]
Learning outcomes
You will (1) assemble a practice portfolio; (2) review the course's main principles; (3) assess your readiness and gaps; (4) plan continued development; and (5) commit to ethical practice.
12.1 Portfolio
Include: a context and data-flow analysis; a test plan and results; a finding report; a control verification record; a case analysis; a communication sample; the lab report. Remove confidential details and use synthetic examples where needed. For each item, add a short note on what you learnt.
Sofia's portfolio has seven items, each with a note. Ask your supervisor to review items for confidential details before you share them outside Meridian. Keep each item short enough to discuss in a few minutes, and order them to tell the story of one assessment from context to communication.
12.2 Principles to keep
Start from context and affected people. Work only with authorisation. Assume models can be steered; enforce consequential limits outside them. Measure by effects, with intervals and base rates. Verify controls in both directions and after every change. Communicate honestly to each audience.
Each principle appears in Sofia's work at Meridian: the stakeholder map, the authorisation, the callback tool rule, the intervals, the verification and the reports. Use them as a checklist when starting new work. Revisit them when work goes wrong.
12.3 Readiness and gaps
Rate yourself on: mapping a service; designing tests; configuring and verifying controls; interpreting results; writing findings; incident analysis. Choose two gaps to work on next.
Base each rating on evidence from your portfolio, as Sofia did when she identified statistics and automation as gaps. Ask your supervisor to check the ratings, and write down how you will close each chosen gap and by when. Rate yourself on each skill in this book, from context work to communication. Choose two gaps to close in the next six months.
12.4 Continuing development
Follow OWASP, NIST and MITRE ATLAS updates; practise in authorised labs and capture-the-flag events with AI tracks; read one research paper a month — for example, Greshake et al. (arXiv:2302.12173) on indirect injection; join a professional community.
"""Lab R: token-bucket limits against unbounded consumption (synthetic).
Simulates 60 seconds of requests from a normal client and a flooding client
against a model endpoint with a per-client token bucket."""
import random
rng = random.Random(4)
def simulate(rate_per_s, burst, reqs):
tokens, last, ok = burst, 0.0, 0
for t in reqs:
tokens = min(burst, tokens + (t - last) * rate_per_s); last = t
if tokens >= 1: tokens -= 1; ok += 1
return ok
normal = sorted(rng.uniform(0, 60) for _ in range(90))
flood = sorted(rng.uniform(0, 60) for _ in range(3000))
for rate, burst in ((5, 20), (2, 10), (1, 5)):
print(f"limit={rate}/s burst={burst:2d} normal_served={simulate(rate, burst, normal):3d}/90 "
f"flood_served={simulate(rate, burst, flood):4d}/3000")
Expected output:
limit=5/s burst=20 normal_served= 90/90 flood_served= 319/3000
limit=2/s burst=10 normal_served= 90/90 flood_served= 129/3000
limit=1/s burst= 5 normal_served= 62/90 flood_served= 64/3000
Sofia's plan includes a statistics short course and automating her ClaimChat suite with the engineering team. Choose activities that close the gaps you identified, and set measurable goals, such as completing a course or automating a suite by a date. Review progress with your supervisor every quarter, and update the plan when your responsibilities change.
12.5 Ethics
Authorised testing only; protect personal data; report honestly; coordinate disclosure; consider people affected by AI decisions.
Each commitment has an example in this book: Sofia's written authorisation, her stop when she saw a real customer's name, her refusal to change a rating under pressure and her recommendations for FraudFlag claimants. Keep these in mind when deadlines press. Seek permission, protect the people affected, report honestly and stop when something goes wrong. These commitments matter most when there is pressure to cut corners, and they apply to every system you test.
12.5b Using the portfolio
Bring the portfolio to performance reviews and job interviews, and use it with your supervisor to agree on the next responsibilities.
Update it as you complete new work, and remove items that no longer reflect your skills. Before each review, choose the items most relevant to the responsibilities you want next. Sofia's seven items support her supervisor's stretch goal for her: leading the next quarterly evaluation with support. Items that show judgement, such as a case analysis, often say more than a list of tools used.
12.6 Worked example (synthetic Meridian case)
Sofia's portfolio has seven items. Her self-assessment shows strengths in mapping and communication (from her claims background) and gaps in statistics and automation. Her plan: complete a statistics short course; automate her ClaimChat suite in the pipeline with the engineering team; present the FraudFlag case at an internal forum. Her supervisor adds a stretch goal: lead the next quarterly evaluation with support.
The plan has measurable goals and a stretch goal agreed with the supervisor. Sofia's supervisor agreed her gap ratings.
12.7 Errors and limits
- Portfolios with confidential data.
- Plans without measurable goals.
- Self-assessments without evidence.
Each error weakens the portfolio or plan. Confidential data in a portfolio breaches trust and may break the law. Plans without measurable goals cannot be checked at the next review. Self-assessments without evidence are opinions; Sofia's rested on her portfolio items. Check your portfolio and plan for all three before your review. Fix any you find before the review. Sofia's supervisor checked her portfolio for confidential details, and her plan names dates for each goal.
Exercises
- EX056 — Portfolio. List your seven portfolio items and one learning note each. Rubric: 3 = seven with notes; 2 = five; 1 = three; 0 = fewer.
- EX057 — Principles. State the six principles in your own words. Rubric: 3 = six; 2 = four or five; 1 = two or three; 0 = fewer.
- EX058 — Gaps. Rate yourself on six skills with evidence and choose two gaps. Rubric: 3 = six ratings, evidence, two gaps; 2 = partial; 1 = vague; 0 = none.
- EX059 — Plan. Write a six-month development plan. Rubric: 3 = specific, measurable, includes ethics; 2 = two; 1 = vague; 0 = none.
- EX060 — Rate limit. Using the lab, explain the trade-off at 1/s. Rubric: 3 = flood cut to 64 but normal users blocked (62/90); 2 = partial; 1 = vague; 0 = wrong.
Synthesis
The portfolio shows what you can do; the principles guide what you do next; ethics keeps practice trustworthy.
Continue studying — chapter 12 of 12
← Chapter 11 (draft) · Course chapter index · Worked results ↓ · Next course via chapter index →