Chapter 4 — Scripting and data formats
Lesson map
Lesson 04 published. Map, example, EX013.
flowchart TD
N0["Learning outcomes"]
N1["4.1 JSON lines"]
N2["4.2 Filtering"]
N3["Lab — filtering endpoint events"]
N4["4.3 Reading the result"]
N5["4.4 Keeping originals"]
N6["4.5 Output clarity"]
N7["4.6 Malformed lines"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7
N7 --> EX["Worked example"]
EX --> Q["Exercise EX013"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["4.1 JSON lines"]
N2["4.2 Filtering"]
N3["Lab — filtering endpoint events"]
N4["4.3 Reading the result"]
N5["4.4 Keeping originals"]
N6["4.5 Output clarity"]
N7["4.6 Malformed lines"]
N0 --> N1 --> N2 --> N3 --> N4 --> N5 --> N6 --> N7 --> Q["EX013"]
[Learning outcomes]
|
v
[4.1 JSON lines]
|
v
[4.2 Filtering]
|
v
[Lab — filtering endpoint events]
|
v
[4.3 Reading the result]
|
v
[4.4 Keeping originals]
|
v
[4.5 Output clarity]
|
v
[4.6 Malformed lines]
|
v
[Worked example] --> [Exercise EX013]
Learning outcomes
You will (1) read JSON-lines event exports; (2) filter events by host and process; (3) produce clear, checkable outputs; (4) keep original exports intact.
4.1 JSON lines
EDR exports often use JSON lines: one JSON object per line with fields such as time, host, process, action and detail. Python's json module turns each line into a dictionary.
Each line can be read and checked on its own, so a single damaged line does not stop the rest of the export from loading. Sana keeps the field names exactly as the export gives them, such as t, host, proc, act and detail, so her scripts match the console's documentation. Times in the export arrive in UTC, marked with the Z suffix. The EDR server stamps each event with its own synchronised UTC time when it receives it from the agent. WARD4-PC2's slow clock therefore does not affect these times, and they go into the ticket without correction.
4.2 Filtering
Investigations narrow to one host and process. Sana filters WARD4-PC2 events for wscript.exe, the Windows script host that ran right after the rota shortcut was opened.
Filtering first by host and then by process keeps the result small enough to read line by line. The filter returns the two wscript.exe events on WARD4-PC2 and leaves out ADMIN-PC1's ordinary minutes.docx event. Sana records the filter values with the output, so another operator can repeat the same narrowing on the same export.
Lab — filtering endpoint events
"""Filter synthetic endpoint events (JSON lines) for one host and process."""
import json
lines = [
'{"t":"2026-09-14T10:02:11Z","host":"WARD4-PC2","proc":"explorer.exe","act":"usb_mount","detail":"E:"}',
'{"t":"2026-09-14T10:02:40Z","host":"WARD4-PC2","proc":"explorer.exe","act":"open","detail":"E:/Rota_Sept.lnk"}',
'{"t":"2026-09-14T10:02:41Z","host":"WARD4-PC2","proc":"wscript.exe","act":"start","detail":"E:/.sys/run.vbs"}',
'{"t":"2026-09-14T10:02:44Z","host":"WARD4-PC2","proc":"wscript.exe","act":"file_write","detail":"C:/Users/Public/svch.exe"}',
'{"t":"2026-09-14T10:05:00Z","host":"ADMIN-PC1","proc":"outlook.exe","act":"open","detail":"minutes.docx"}']
ev = [json.loads(l) for l in lines]
for e in ev:
if e["host"] == "WARD4-PC2" and e["proc"] == "wscript.exe":
print(e["t"], e["act"], e["detail"])
print("events for WARD4-PC2:", sum(e["host"] == "WARD4-PC2" for e in ev))
Run with python3 lab_json.py. Verified output:
2026-09-14T10:02:41Z start E:/.sys/run.vbs
2026-09-14T10:02:44Z file_write C:/Users/Public/svch.exe
events for WARD4-PC2: 4
4.3 Reading the result
Of five synthetic events, four are from WARD4-PC2. Filtering for wscript.exe returns two: at 10:02:41 it started a script from a hidden folder on the USB stick (E:/.sys/run.vbs), and at 10:02:44 it wrote C:/Users/Public/svch.exe. The sequence — USB mount 10:02:11, shortcut opened 10:02:40, script 10:02:41, executable written 10:02:44 — shows a shortcut on the stick launching a hidden script that dropped a program.
The four-step sequence takes thirty-three seconds from mount to the executable being written, which leaves little room for anything a person might have done in between. The lab prints the count of four events for WARD4-PC2, so a reader can check that nothing was dropped by the filter. These EDR times, stamped by the synchronised EDR server, are later used to check the corrected PC-local times from the file system and event logs.
4.4 Keeping originals
The original export is hashed and stored; the filtered output is derived and labelled with the filter used.
The SHA-256 of the original export is recorded in the ticket before any filtering is done. Every derived file names the original's hash and the filter used, so anyone can recreate it from the evidence store. If a derived file is lost or changed, nothing is lost, because the original can be filtered again. Derived files are stored in a separate folder from the originals, which the evidence store keeps read-only.
4.5 Output clarity
Print times in UTC with the Z suffix, and print counts so a reviewer can check nothing was dropped.
Mixing local and UTC times caused confusion early in the case, when ward staff reported times in local time and the exports used UTC. Sana writes every time in her notes with its zone, and prints counts such as 'events for WARD4-PC2: 4' beside each result. A reviewer comparing the four printed events with the four WARD4-PC2 lines in the export then knows the filter kept everything for that host.
4.6 Malformed lines
Lines that fail to parse are counted and listed.
A line that fails to parse is printed with its line number and kept in a separate list, so it can be examined rather than silently dropped. In the WARD4-PC2 export no line failed, and the script says so explicitly with a zero count. A silent skip could hide exactly the event that matters, such as a truncated record from 10:02. The count of malformed lines appears in the output even when it is zero, so a reader knows the check ran.
4.7 Versioning
The script is kept in the team repository; ticket notes cite its version.
The note in the ticket reads 'filter script v1.0', which ties the result to one reviewed version of the code. If the script is later fixed or extended, older notes still point to the version that produced them. Owen can retrieve that exact version from the repository and rerun it on the hashed export to confirm the two events. Version numbers change only after Owen reviews the new code and its test on synthetic data.
Combining filters
Filters can combine host, process and time window. Sana keeps each filter as a named, saved query so it can be rerun exactly by a colleague.
Worked example
Sana's ticket note: "EDR export (SHA-256 recorded) filtered for host WARD4-PC2, process wscript.exe: 2 events. Script E:/.sys/run.vbs started 10:02:41Z; wrote C:/Users/Public/svch.exe 10:02:44Z. Filter script v1.0."
Synthetic case. Events come from the lab script.
Errors and limits
EDR records what its sensors observe; actions outside its view, or before installation, are missing.
Exercises
- EX013 — Lab events. What did the wscript.exe filter return? Rubric: 3 = 10:02:41 start run.vbs; 10:02:44 write svch.exe; 2 = one; 1 = wrong; 0 = none.
- EX014 — Sequence. Summarise the four-step sequence on WARD4-PC2. Rubric: 3 = USB mount, shortcut opened, script ran, executable written (with times); 2 = three; 1 = one; 0 = none.
- EX015 — Derived output. How is filtered output distinguished from the original? Rubric: 3 = labelled derived with filter; original hashed and kept; 2 = partial; 1 = overwrite; 0 = none.
- EX016 — EDR limit. What can EDR miss? Rubric: 3 = actions outside sensors or before install; 2 = partial; 1 = vague; 0 = none.
Synthesis
Parse structured exports carefully, filter transparently and keep originals; sequences emerge from well-filtered events.
Continue studying — chapter 04 of 18
← Chapter 03 (draft) · Course chapter index · Chapter 05 (draft) →