OpenCyber AcademyLesson map
Lesson 06 published. Map, example, EX016.
flowchart TD
N0["Learning outcomes"]
N1["6.1 Why logs matter"]
N2["6.2 Extracting fields"]
N3["6.3 Time and clocks"]
N4["6.4 Keeping logs"]
N0 --> N1 --> N2 --> N3 --> N4
N4 --> EX["Worked example"]
EX --> Q["Exercise EX016"]
Logical drawing
flowchart LR
N0["Learning outcomes"]
N1["6.1 Why logs matter"]
N2["6.2 Extracting fields"]
N3["6.3 Time and clocks"]
N4["6.4 Keeping logs"]
N0 --> N1 --> N2 --> N3 --> N4 --> Q["EX016"]
[Learning outcomes]
|
v
[6.1 Why logs matter]
|
v
[6.2 Extracting fields]
|
v
[6.3 Time and clocks]
|
v
[6.4 Keeping logs]
|
v
[Worked example] --> [Exercise EX016]
Learning outcomes
You will (1) read common log formats; (2) extract fields with patterns; (3) understand time zones and clocks in logs.
6.1 Why logs matter
Logs record what systems did: logins, failures, changes and errors. Jamal learns that logs are how technologists find out what happened after a problem. Riverbend's systems produce logs in different formats: the booking system exports CSV, servers write text lines and the firewall uses its own format. Ms. Grant explains that being able to read any format is a core technologist skill. He collects one example line from each format and annotates the fields in his notebook, which helps him recognise each format quickly later. The IT manager confirms that these are the logs he checks most often when something goes wrong at a centre.
6.2 Extracting fields
Jamal learns regular expressions with Python's re module to extract fields from text lines, such as the time, username and source address from a login record. He practises on synthetic logs and checks his pattern against unusual lines, such as usernames with dots or addresses in different formats. He learns that a pattern which works on the first hundred lines may fail on the thousandth, so testing with varied data matters. He keeps a test file of unusual lines and runs every new pattern against it before use.
6.3 Time and clocks
Log times only help if clocks are correct. Jamal finds that one centre's camera recorder clock is eleven minutes slow, which would confuse any investigation comparing footage with till records. He also learns that some systems log in local time and others in universal time. He recommends synchronising all clocks to a reliable time source and recording time zones explicitly. Ms. Grant notes that mismatched clocks have undermined many investigations. The IT manager configures all recorders and servers to use the same time source, and Jamal checks a week later that every clock matches.
6.4 Keeping logs
Jamal learns that logs must be kept long enough to investigate problems, protected from alteration and stored where attackers cannot delete them. Riverbend keeps server logs for only seven days on each server. He recommends forwarding them to a central store kept for ninety days, with access limited to IT staff. He notes that logs containing personal data must also follow retention rules. The IT manager sets up the central store using existing server space. Jamal checks after a week that logs from every server are arriving, and finds one server that was not forwarding because of a typing error in its configuration.
Worked example
Jamal reads several log formats, extracts fields with tested patterns, finds an eleven-minute camera clock error and recommends central log storage for ninety days.
Synthetic case. Logs are synthetic.
Errors and limits
Unsynchronised clocks make events from different systems impossible to order.
Exercises
- EX016 — Pattern module. Which module extracts fields?
Show marking criteria
Rubric: 3 = re;; 2 = partly correct;; 1 = other;; 0 = none. - EX017 — Clock error. How slow is the camera recorder clock?
Show marking criteria
Rubric: 3 = eleven minutes;; 2 = partly correct;; 1 = wrong;; 0 = none. - EX018 — Log retention. How long should central logs be kept?
Show marking criteria
Rubric: 3 = ninety days;; 2 = partly correct;; 1 = wrong;; 0 = none.