Chapter 7 — Networks and protocols
Lesson map
Lesson 07 published. Map, example, EX019.
flowchart TD
N0["Learning outcomes"]
N1["7.1 The path of a request"]
N2["7.2 What TLS does and does not do"]
N3["7.3 API authentication and rate limiting"]
N4["7.4 Network-level AI threats"]
N5["7.6b Timeouts and retries on model calls"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6
N6 --> EX["Worked example"]
EX --> Q["Exercise EX019"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["7.1 The path of a request"]
N2["7.2 What TLS does and does not do"]
N3["7.3 API authentication and rate limiting"]
N4["7.4 Network-level AI threats"]
N5["7.6b Timeouts and retries on model calls"]
N6["Synthesis"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> Q["EX019"]
[Learning outcomes]
|
v
[7.1 The path of a request]
|
v
[7.2 What TLS does and does not do]
|
v
[7.3 API authentication and rate limiting]
|
v
[7.4 Network-level AI threats]
|
v
[7.6b Timeouts and retries on model calls]
|
v
[Synthesis]
|
v
[Worked example] --> [Exercise EX019]
Learning outcomes
You will (1) explain how a chatbot request travels across the internet (DNS, TCP, TLS, HTTP); (2) describe what TLS protects and what it does not; (3) explain API authentication and rate limiting; and (4) relate network controls to AI threats such as extraction and resource exhaustion.
7.1 The path of a request
The browser looks up the chatbot's domain (DNS), connects (TCP), sets up an encrypted, authenticated channel (TLS) and sends an HTTP request containing the message. The application server may in turn call a model API over another TLS connection.
Each step leaves a record that can help an investigation: the DNS query, the TLS handshake at the load balancer, the HTTP request at the application server and the outbound call to the model provider. For Ask Lumen, a single patient message therefore crosses two encrypted connections, and the application server is the one place where the plain text is visible to LCH. That is where input checks, logging and policy decisions belong.
7.2 What TLS does and does not do
TLS protects data in transit from eavesdropping and tampering and authenticates the server. It does not protect against a malicious message from an authenticated user, against prompt injection inside content, or against data misuse once it arrives.
For Ask Lumen, TLS ensures that a patient's message reaches LCH's server unread and unchanged and that the patient is talking to the real LCH site. It does nothing about a patient who types an instruction designed to manipulate the model, or a partner document carrying hidden text. Those problems sit inside the protected channel and need controls in the application.
7.3 API authentication and rate limiting
APIs authenticate callers with keys or tokens. Rate limits and quotas bound how much each caller can use. For AI APIs, costs scale with tokens, so limits should count tokens, not just requests. OWASP LLM10 (unbounded consumption) covers attacks that exhaust resources or budgets.
"""Lab R: token-bucket limits against unbounded consumption (synthetic).
Simulates 60 seconds of requests from a normal client and a flooding client
against a model endpoint with a per-client token bucket."""
import random
rng = random.Random(4)
def simulate(rate_per_s, burst, reqs):
tokens, last, ok = burst, 0.0, 0
for t in reqs:
tokens = min(burst, tokens + (t - last) * rate_per_s); last = t
if tokens >= 1: tokens -= 1; ok += 1
return ok
normal = sorted(rng.uniform(0, 60) for _ in range(90))
flood = sorted(rng.uniform(0, 60) for _ in range(3000))
for rate, burst in ((5, 20), (2, 10), (1, 5)):
print(f"limit={rate}/s burst={burst:2d} normal_served={simulate(rate, burst, normal):3d}/90 "
f"flood_served={simulate(rate, burst, flood):4d}/3000")
Expected output:
limit=5/s burst=20 normal_served= 90/90 flood_served= 319/3000
limit=2/s burst=10 normal_served= 90/90 flood_served= 129/3000
limit=1/s burst= 5 normal_served= 62/90 flood_served= 64/3000
Lab R shows the trade-off. At 5 requests per second with a burst of 20, every normal request is served and the flood gets 319 of 3,000. At 1 per second with a burst of 5, the flood drops to 64, but the normal client loses 28 of its 90 requests. A limit must be set from measured normal use, not chosen only to stop attacks.
7.4 Network-level AI threats
Systematic querying can copy a model (Tramèr et al., arXiv:1609.02943). Floods of long prompts can exhaust capacity. Network controls — authentication, quotas, monitoring, segmentation and egress filtering — reduce these risks and limit what a compromised component can reach.
Extraction needs many queries, so per-client quotas and monitoring for unusual query patterns raise its cost. Egress filtering matters if a component is compromised: an inference server that can reach only the model provider cannot send data elsewhere. Segmentation keeps the chatbot's servers apart from the scheduling database except through the booking tool's narrow interface.
7.5 Worked example (synthetic LCH case)
Ask Lumen's public chat endpoint had no per-session limits. A synthetic load test sending long prompts drove model costs up tenfold in an hour. LCH added per-session token quotas, a maximum message length and an alert when spending exceeds 150% of the daily median.
Counting tokens rather than requests was the key change: a few very long prompts had cost more than thousands of normal messages. The alert threshold of 150% of the daily median catches abuse within hours while ignoring the normal rise on busy mornings, and the maximum message length stops a single prompt from consuming a large share of the budget.
7.6 Errors and limits
- Believing TLS solves application-level attacks.
- Request limits that ignore token cost.
- API keys embedded in browser code.
TLS protects the channel, not the content, so prompt injection and abuse by logged-in users pass straight through it. Request-only limits let an attacker send fewer but far longer prompts, as the LCH load test showed. API keys in browser code can be read by anyone who opens the page, letting them call the model provider directly and bypass LCH's quotas entirely; keys belong on the server.
7.6b Timeouts and retries on model calls
Ask Lumen calls the model provider over the network, and slow responses tempt engineers to retry aggressively. Each retry resends the patient's question and costs money; unbounded retries during an outage can multiply traffic and exhaust the rate limit for every clinic. LCH caps retries at two with a growing delay and shows patients a phone number when the model is unavailable.
A growing delay between retries lets a struggling provider recover instead of being flooded. The phone number keeps patients served while the chatbot is down.
Exercises
- EX019 — TLS. Name two things TLS protects and two it does not. Rubric: 3 = confidentiality/integrity in transit, server authentication vs malicious content, injection/misuse; 2 = three; 1 = one or two; 0 = none.
- EX020 — Lab reading. In the lab, why does the 1/s, burst 5 limit hurt normal users? Rubric: 3 = only 62/90 served because natural bursts exceed the bucket; 2 = number only; 1 = vague; 0 = wrong.
- EX021 — Keys. Why must model API keys not appear in browser code? Rubric: 3 = anyone can read and misuse them, causing cost and abuse; use server-side calls; 2 = partial; 1 = vague; 0 = wrong.
Synthesis
Networks deliver AI services; TLS, authentication, token-aware quotas and monitoring protect the delivery but not the content, which later chapters address.
Continue studying — chapter 07 of 40
← Chapter 06 (draft) · Course chapter index · Chapter 08 (draft) →