Backup provenance, freshness and recovery objectives

Lesson 13 — Backup provenance, freshness and recovery objectives

Week 3, day 3. Work through the teaching and the worked checkpoint before attempting the assigned exercise(s). Compare your artifact with the final worked result only afterward.

3.8 Guided backup-candidate and isolated-restore lab

Section 3.5 introduced recovery objectives; section 3.6 compared booking states; section 3.7 checked whether the comparison itself silently discarded rows. A recovery lead must join those three kinds of evidence before proposing a return to service. The most recent copy can contain the very error being investigated. An older copy can appear correct for the four displayed bookings while omitting legitimate work. A valid checksum can show that a particular copy matches a recorded digest without showing that its source was trustworthy or that its contents meet the service owner's target. NIST SP 800-184 provides recovery-planning and testing guidance; the three-candidate gate below is an original, deliberately small classroom method.

This is an optional exercise-only replay, introduced after P1/C1 and V1. It does not add an event to R1–R24, does not describe a real Harbor Library backup, and cannot be used to revise the 09:37 containment choice. In this replay, imagine a service interruption at 09:45 UTC and a candidate review at 09:46 UTC. The service owner's illustrative targets from section 3.5 are a thirty-minute RPO and a two-hour RTO. Neither target was established by R7 or R23. The candidate descriptions below are supplied solely so a learner can practice deciding what still needs verification. The exercise assumes an approved isolated environment and permission to inspect only synthetic booking IDs; it authorizes no production restore.

Card K1 — three proposed comparison copies. A snapshot timestamp is the claimed data point. An isolated import duration is a measured classroom trial, not the duration of restoring the production service. The checksum comparison is against a digest recorded with that same candidate; the exercise does not establish independent authenticity of the digest or the snapshot source.

The data-point ages relative to the hypothetical 09:45 interruption are 45, 25 and 7 minutes respectively. The thirty-minute RPO would screen out K-A as a stand-alone copy if there were no verified way to recover legitimate changes after 09:00. K-B is within the time target, but its digest mismatch is a stop-and-investigate result: a responder cannot replace the recorded digest with one from the observed bytes and call the copy verified. K-C is fresh enough by the timestamp alone, yet its B19/B20 state conflicts with the later owner-approved P1 target. The matching digest does not cure that logical error. These observations do not prove that K-A is clean throughout, that K-B is malicious, or that K-C is unusable for every limited purpose.

Four separate gates before a restore proposal

1. Provenance and permission. Identify the snapshot identifier, originating system, claimed creation time and clock basis, capture method, recorded digest, custody of the copy, allowed test environment and owner who approved the test. K1 omits most of these details. A custodian should supply them in an actual case; a learner should leave the fields open rather than infer them from the candidate label. Preserve the original candidate and test a controlled working copy.
2. Copy integrity and usability. Compare bytes with the previously recorded digest, then attempt a bounded isolated import with error logging and a repeatable test. A match supports only a limited byte-integrity claim; a successful import shows that the tested platform read the copy under the trial conditions. K-B fails the first check and has no second check. Neither K-A's nor K-C's successful import proves that the complete database, dependencies or access controls work.
3. Data correctness and loss. Compare the isolated booking state with an owner-approved target and account for legitimate transactions between the snapshot and interruption. P1 approves only B17–B20. K-A matches those four rows but may lack later legitimate bookings; K-C contains the known B19/B20 mismatches. A real reconciliation needs a source for intervening work, its provenance, a conflict rule, and an owner who accepts any unrecoverable gap. If that evidence does not exist, the team reports an unknown loss bound, not “zero loss.”
4. Service acceptance and control. In a safe test scope, verify authorized and unauthorized actions, actor-labelled audit records, manual-queue reconciliation, dependencies, user-facing behavior and a monitoring/rollback condition. V1 supplies two limited access observations and one failed audit-attribution observation. It does not become a pass because a different copy imported quickly. The service owner and incident lead must decide whether restricted operation with a documented compensating check is acceptable; a full-return claim requires the failed control and other acceptance criteria to be resolved.

These gates can fail independently. A candidate can pass an integrity check and fail the owner's data comparison; another can match four rows and fail freshness; a third can be fresh but have an unverified copy. Writing one label such as “good backup” hides the decision. Use PASS, FAIL or UNKNOWN per gate, with the supplied observation beside it. UNKNOWN is not an implied pass. Where a source cannot be trusted, the correct next step is to seek a more reliable copy or independent corroboration, not to force the matrix to a single winner.

An RPO calculation is only a screening calculation

For this replay, subtract the claimed snapshot time from 09:45: K-A is 45 minutes old; K-B is 25 minutes; K-C is 7 minutes. That arithmetic tests the age of the stated recovery point. It does not measure the number of valid bookings lost, the effect of a write queue, whether a snapshot is internally consistent, or how far a restored application would lag once other dependencies are rebuilt. If the clock or source metadata is wrong, even the age calculation is only conditional. An RPO pass needs an acceptable recovered data point under the service owner's rules, not merely a recent filename.

Likewise, 11 or 14 minutes of isolated import is not an RTO result. The two-hour target concerns usable service after the interruption. A meaningful elapsed time would need a defined start and stop, a tested platform, data and access checks, downstream dependencies, and an acceptance decision. In this fixture, no production restart or owner acceptance occurred. Record the isolated times as component measurements and leave the service-level recovery time unmeasured. This avoids an attractive but invalid equation: import duration = outage duration.

Practice: write a candidate decision card

Create a four-row gate matrix for each candidate, using the gate names above. For each row, state one K1/P1/V1 observation and the reason the result is PASS, FAIL or UNKNOWN. Then write a six-line recommendation: (1) the 09:46 review time and status as proposal; (2) whether any candidate independently supports both data-point and correctness claims; (3) which copy, if any, merits a further isolated trial and under whose approval; (4) what records are needed to reconstruct legitimate changes after its snapshot; (5) how the failed V1 actor field and unresolved cause affect acceptance; and (6) the next review trigger and named owner. Do not combine K1 with the canonical case history as though these candidate tests had actually happened.

For a variation, imagine that the only independent transaction journal also contains a gap from 09:10 to 09:18 UTC. Explain why a “replay all missing transactions” plan is no longer verified. Identify the narrowest additional source or service-owner decision that could bound the missing interval. This inject belongs to the exercise only. Removing it returns you to the K1 facts; do not add it to R1–R24 or assert a real journal gap.

Expected submission: three candidate matrices, the six-line decision card and a two-sentence variation response. Assessment rubric (six points): one point each for correct age arithmetic; treating K-B's mismatch as a failed integrity gate; distinguishing K-A's four-row match from complete or timely recovery; identifying K-C's B19/B20 conflict; keeping isolated import time separate from RTO and V1's audit failure open; and proposing an authorized, reviewable next test without claiming a completed production restore. A 6/6 answer is coherent for this fixture only. The model result follows the Module 3 source-reading answer in the final appendix.

Return to service is a separate decision

A team can reduce further edits while leaving the service unusable or its booking data wrong. Equally, a restored interface may allow unauthorized changes or fail to produce an attributable audit. The return-to-service owner therefore needs a paired gate: acceptable protective behavior and acceptable legitimate business behavior, plus correct state and sufficient observability. A failed mandatory gate means the full acceptance claim is blocked.

Rollback is not simply "reverse the command." It needs a named owner, scope, the state or configuration to restore, the treatment of queued work, and tests for both renewed user service and restored security. If a broad pause must remain temporarily, the service owner should record how manual requests are received, protected from duplicates, later reconciled and reviewed. A risk exception is explicit, time-bounded and approved; it must not be hidden under the label "recovered."

The last exercise of the chapter should therefore leave four distinct artifacts: a response decision with approval state, a causal hypothesis and falsifier, an isolated recovery test request with gate results, and an owner-reviewed service acceptance decision. No one artifact alone proves the others.

3.8a Measure recovery time and the recovered data point

Learning objectives and instructional fit

After this supplement, a new incident-response learner should be able to explain which two events bound an outage-to-service-acceptance measurement, calculate elapsed time from those events when trustworthy timestamps exist, and identify why a short installation or import time is not the same measurement. The learner should also screen snapshot age against a hypothetical recovery-point target without presenting age as a count or duration of actually lost business transactions.

The earlier Chapter 03 teaches that R23's backup-job success does not establish recovery readiness. Section 3.5 introduces recovery time and point objectives; Section 3.8 uses K1 to compare candidate timestamps, copy integrity and booking correctness. Those are necessary foundations. The distinct next step is to design a measurement record from interruption through recovery testing and service-owner acceptance, so that a responder can say precisely whether an RTO result was measured, merely targeted, or impossible to determine.

The method below uses invented small records and standard-library Python. It deliberately avoids repeating the K1 candidates, using P1's four booking values as synthetic output, simulating a live write pause, or replacing the chapter's separate access-and-audit tests. The aim is to improve reasoning from the evidence the learner actually possesses.

RTE3.1 Define the unit of service that starts and stops the clock

A statement such as "the system recovered in twelve minutes" is uninterpretable until it identifies which system function, the starting event, the stopping event and the source of both times. A server process might restart in twelve minutes while a customer-facing booking workflow remains unavailable. A database might become queryable while staff writes are still blocked. Neither event necessarily proves that the library resumed its accepted reservation service.

For this classroom, define the measured unit as the owner's approved room-booking service outcome for the scoped exercise. The measurement starts when that outcome is first documented as interrupted under the agreed exercise rule. It stops only when the scoped functional, booking-data, access and audit acceptance conditions have passed and the designated service owner records an acceptance decision. The owner's decision may include a restricted-mode exception, but the claimed scope must then say restricted, not unrestricted recovery.

This is an operational measurement contract chosen for RTE3, not a universal formula required by NIST. A different organization may define outage onset, business impact and RTO scope differently in its business impact analysis. NIST SP 800-34 Rev. 1 provides official contingency-planning context, and the NIST RTO glossary ties recovery time objectives to the effect of recovery duration on mission or business processes.

The interval only answers an honest question if both endpoints apply to the same declared service scope. A start time for staff booking writes and an end time for the home page are mismatched endpoints. A measured end for one isolated test copy does not prove a live service accepted production requests. Every measurement card should therefore name the function and whether the environment is an isolated training fixture, an authorized rehearsal or a real system under appropriate controls.

RTE3.2 An RTO target is not an observed recovery duration

An objective expresses how quickly a function should recover under an owner-approved plan. A measurement reports how long an identified event actually took under a specified method. A pass/fail comparison requires both the approved objective and enough evidence for the corresponding measurement. It is improper to report "RTO achieved" merely because someone proposed a target or because a restore command ended within that target.

The current book uses a hypothetical two-hour RTO for a planning example. RTE3 retains the same illustrative magnitude: 120 minutes. The classroom accepts it only as a fixture input. It is not established that Harbor Library's real or canonical fictional service owner adopted that target. R7 and R23 supply no actual service-interruption timestamp, final accepted-service time or measured restore duration.

To make measurement defensible, use at least three distinct time concepts. Interruption time denotes when the named service outcome became unavailable. Technical-ready time records when a restricted copy or component can perform a bounded technical test. Service-accepted time records the owner's later decision after the required business and security checks. Detection and recovery-start times are additional useful milestones but are not substitutes for the two contract endpoints.

If the technical environment starts at 10:20 but the service owner accepts the complete scoped service only at 11:10, the recovery interval does not end at 10:20 under RTE3's contract. Those fifty minutes may contain evidence reviews, identity tests, corrections, queue reconciliation and owner decision-making. Ignoring them makes a technical component look faster than the service actually became acceptable to use.

RTE3.3 Construct a milestone ledger with traceable meanings

A recovery measurement ledger should capture what each timestamp means and what produced it, not simply a sequence of clock readings. The following columns form a useful minimum:

A timestamp should be stored together with its timezone, source and uncertainty. RTE3 uses one artificial calendar date and explicit UTC offset in Python solely to avoid ambiguity. In a real collection, time from different systems may need independent clock and ingestion validation. Sorting timestamp strings is not enough to establish that each source observed the same event boundary.

The ledger also needs a status, such as planned, performed, verified, failed or unavailable, for every milestone. It is inaccurate to fill a planned 11:10 acceptance slot with a value and later report it as measured if the owner never accepted the service. Preserve the original planned time and mark the actual receipt missing.

RTE3.4 What counts as an acceptable stopping event?

The end of the recovery interval must be tied to the declared outcome. For a fictional booking service, plausible acceptance gates include:

1. Data: the restored or reconciled state matches the relevant owner-approved expected objects, with explicitly bounded coverage.
2. Authorized operation: a permitted synthetic staff booking can succeed in the authorized test scope.
3. Denied operation: a prohibited synthetic edit is rejected without an unintended state change.
4. Audit: the permitted action produces the expected actor and event time within the tested route.
5. Owner decision: the accountable service owner records acceptance for the tested operating mode and notes residual restrictions.

These gates are not a substitute for real organization-specific criteria. They are intentionally separate because passing one cannot repair another. The existing V1 fixture reports two limited access outcomes and a blank actor field; that failed audit attribution criterion would prevent an unrestricted acceptance claim under the relevant later exercise. RTE3 does not change V1's outcome or pretend that any new booking correction was executed.

Owner acceptance itself should state its boundary. A service may be conditionally accepted for restricted use while a different function remains blocked, but the resulting duration can only be attributed to that restricted service definition. Labeling such a measured time as "full booking service restored" would erase the material exception.

An observed technical-ready time is valuable as a phase measurement. It can help a team understand how long platform work took. It is not, by itself, the stopping event of the declared end-to-end service-acceptance measurement.

RTE3.5 Contrastive worked example: measured service acceptance

Entirely synthetic RTE3 example, not a Harbor Library event. Imagine an isolated training service on a fictional date with the following documented teaching milestones:

The elapsed interruption-to-acceptance interval is 85 minutes: from 09:45 to 11:10. The technical-ready interval is 35 minutes, from 09:45 to 10:20. Those values answer different questions. The response detected the disruption after ten minutes, but beginning the interval at 09:55 would conceal those ten minutes of unavailable service under this contract.

Against the illustrative 120-minute target, the 85-minute simulated measurement is within target for this defined classroom service and gate set. It does not establish that Harbor Library met an approved organizational RTO, nor does it prove that every unmodeled booking function was healthy.

Suppose the recovery lead's candidate copy is labeled 09:35 and the interruption occurs at 09:45. The apparent age of the copy at interruption is ten minutes. That is a separate data-point screening calculation. Whether accepted writes after 09:35 were recovered must be evaluated from accepted-operation records and restored outcomes, not inferred from the 85-minute recovery duration.

The useful conclusion is precise: "The isolated RTE3 scenario reached owner acceptance after 85 minutes under the listed fixture gates; the copy was labeled ten minutes old at the declared interruption; and data coverage is assessed separately." It is not a claim of full operational recovery.

RTE3.6 Incorrect example: the eleven-minute import

Consider the following incorrect fictional handoff:

"The copy was imported in eleven minutes. The planned RTO is two hours, so RTO passed. The newest copy was only seven minutes old, so we lost seven minutes of data. There was no reason to wait for the service owner."

The handoff combines three unsupported conversions. Import duration is not necessarily elapsed time from interruption to accepted service. Snapshot age is a gap between two timestamps, not the amount of accepted business work permanently missing. And a successful technical activity does not replace required owner acceptance or security/audit tests.

A correct alternative would read:

"The isolated import step took eleven minutes in the separately supplied exercise. The outage start, tested dependencies, completed acceptance gates and service-owner acceptance time are not all supplied for the full service, so an interruption-to-acceptance result is not measured. The seven-minute snapshot age is a screening value under stated clock assumptions, not proof of seven minutes of lost accepted writes. Further source and owner evidence is needed before a service-level RTO or RPO conclusion."

This alternative is less dramatic but more useful. It tells the next responder exactly which evidence must be requested and which numbers remain defensible.

The distinction applies directly to K1's isolated import durations of 11 and 14 minutes. Those existing numbers describe only the classroom import-and-read trial. They do not measure a completed production outage, and they must not be reused as proof that the owner accepted correct Harbor Library booking service.

RTE3.7 A copy's age screens risk; it does not measure lost transactions

A recovery point objective (RPO) describes the point in time to which data must be recovered after an outage under the approved organizational requirement. The NIST RPO glossary links the definition to NIST SP 800-34 Rev. 1. The classroom's hypothetical RPO magnitude is 30 minutes; that is a target, not an operational result.

Snapshot age at interruption can be calculated if the relevant source/copy timestamp and interruption timestamp are known and comparable. For a copy labeled 09:35 and interruption at 09:45, the apparent age is ten minutes. A copy labeled 08:50 is apparently 55 minutes old at 09:45. This arithmetic is often a helpful initial screen for choosing further tests.

An age that exceeds 30 minutes weakens the case for relying on the older snapshot alone. It does not establish that 55 minutes of accepted business changes were actually lost. Perhaps no approved booking was accepted during part of that interval. Perhaps an independently protected transaction journal can replay later changes. Conversely, a copy only five minutes old can still contain incorrect or inconsistent state.

The correct sequence is: verify copy identity and time basis; identify the last recoverable and acceptable business state; investigate any approved journal/replay sources; compare the recovered state with an authorized ledger of accepted operations; and determine which known accepted operations remain unaccounted for.

K1 itself illustrates why copy age is not sufficient: K-B's digest mismatch blocks its integrity gate, K-C's recent state conflicts with the known approved four-booking target, and K-A's selected matches do not establish completeness of other legitimate work. RTE3 preserves those separate results rather than overwriting them.

RTE3.8 Distinguish a data-point gap from a proved loss count

Suppose a candidate snapshot is labeled 09:35 and two fictional accepted writes are recorded after that time, at 09:38 and 09:42. These teaching operation IDs are W-A and W-B. If the isolated restored state demonstrably includes both accepted effects, the controlled comparison has zero known unrecovered writes within those two listed operations.

That limited count is not a claim that the whole business lost zero data. It requires the supplied accepted-write ledger to be complete for the declared test frame. If the ledger itself is incomplete, the analyst cannot set the total loss bound to zero, even when every listed operation is present. Unknown accepted transactions might still exist outside the frame.

If W-B is absent from the verified restored state, the exercise has one known unrecovered accepted operation in its specified set. That is stronger evidence than mere snapshot age for the missing item. It still does not provide a population-wide count unless the source coverage and business effect are appropriately verified.

The learner should maintain three separate fields: claimed snapshot time, complete or incomplete accepted-operation ledger, and restored/verified operation coverage. Combining them into a single number called "minutes lost" conceals what is known and unknown.

A transaction's timestamp is not automatically the moment it became durable and accepted under every system design. In real recovery, the source owner must define its commit/acknowledgement semantics and correlation keys. Our toy IDs deliberately avoid making that assumption about Harbor Library's live reservation database.

RTE3.9 When accepted operations are missing, stop the shortcut to RPO success

The most tempting mistake in a recovery report is to declare an RPO pass because the snapshot age is below the target, even though a known accepted operation was not recovered. A data-point screen does not repair a business-state discrepancy.

For an isolated exercise, compare the declared set of accepted operation IDs with the set verified in the restored state. The set difference identifies known, specifically referenced operations requiring attention. It cannot identify operations absent from the accepted-write source, and it does not prove that an unrecovered entry was permanently destroyed rather than awaiting a later authorized replay or reconciliation.

If one known accepted write is missing, the recovery lead should record its identifier, source/time semantics, expected state, attempted replay, observed result and owner. The owner then decides whether and how the operation can be reconstructed and how the remaining risk affects service acceptance. An incident responder should not silently discard it merely to make the count zero.

This is particularly important when multiple people are working during a disruption. Manual queue receipts, application acceptance records and transaction journal entries can represent different stages of a request. A manual receipt is not necessarily an accepted booking; a successful HTTP response is not necessarily a proven durable commit. The definition of "accepted operation" needs an owner and source semantics.

RTE3 uses one simple boolean ledger_complete for the synthetic scope. In practice, proving coverage requires retention, time-window, source-health, inclusion and reconciliation checks. The boolean is a teaching premise, not an independently authenticated production attestation.

RTE3.10 What to do when the interruption timestamp is missing

Sometimes a team knows that restoration work started but cannot establish when the essential business service became unavailable. It might have an alert receipt time, a patron complaint or an operator's recollection. Those observations can help reconstruct an interval, but they are not necessarily the actual outage onset.

If the interruption time is absent, the correct result of the elapsed-time calculation is unknown. The analyst can record known milestones, for example, "first confirmed bad service observation" and "last documented good service observation," and then seek a bounded source or owner-accepted estimate. A range may be possible in a more advanced analysis; an exact duration must not be fabricated from a convenient timestamp.

Do not replace missing interruption time with detection time without saying the measurement has changed. "Minutes since alert" and "minutes the service was unavailable" are distinct metrics. A service may have been unavailable before the detector noticed it. Conversely, a detector may fire before a business disruption is established.

In the offline lab, MISSING_INTERRUPTION sets the start to None while retaining a technical-ready time and owner-acceptance receipt from the baseline. The program refuses to compute the outage-to-acceptance interval. Even the age of the snapshot at interruption becomes unknown without the reference time.

That result is a successful demonstration of evidence discipline. It is not a failed calculation in the sense of arithmetic error; it correctly declines to report an unsupported measurement.

RTE3.11 What to do when the acceptance receipt is missing or negative

A recovery team may finish all of its planned technical checks yet lack an owner decision for returning the service to use. The correct status is not "accepted by default." An absent owner receipt is a missing business decision, and the elapsed interval to owner acceptance remains unmeasured under RTE3's endpoint contract.

There is a second, stronger case: a required acceptance gate fails. Suppose a data comparison explicitly reveals an unacceptable mismatch, or the audit test lacks the required actor field. That is not merely an absent receipt. It is a supplied negative result that blocks the declared full-service acceptance. The team should record the failed criterion, its source, the responsible correction owner and the next test, without manufacturing a later acceptance timestamp.

The lab therefore distinguishes NO_OWNER_RECEIPT from ACCEPTANCE_FAILED. In the first case, the selected gates pass in the invented fixture but the final owner receipt is absent. In the second, a required gate has failed, so the service cannot be accepted under that contract despite the technical copy's availability.

An accountable owner could separately approve a restricted operating mode with explicit residual risk. That would create a different measured service scope, with its own start/stop contract and decision evidence. It does not turn a full-service failure into an unrestricted recovery pass.

No real approval, production restoration or scoped risk exception is supplied by RTE3. All owner flags are artificial program inputs.

RTE3.12 When clocks disagree, the measurement needs verification

Calculating a duration assumes that the two endpoint times can be compared. A timestamp may have the wrong timezone, a clock offset, an imprecise event definition or an ingestion delay. Merely formatting it as ISO 8601 does not establish accuracy.

A robust technical record distinguishes producer event time, collector time, review time and decision time, then documents conversions. Chapter 02 teaches those distinctions for investigation logs. In recovery measurement, the same discipline applies to outage start, copy creation and final acceptance.

RTE3 uses one fictional date and UTC-aware Python timestamps, avoiding ambiguous local clock conversion. It also includes an artificial clock_verified flag. When that flag is false, the program refuses the exact age and RTO calculations rather than invent a result. In reality, verification would require evidence about the sources and clocks; a Boolean field is only a classroom stand-in.

If the reported technical-ready time precedes the interruption under the same clock basis, or if the accepted time precedes technical readiness contrary to the defined sequence, the measurement should be marked internally inconsistent. The analyst must determine whether the records refer to different scopes or contain timing errors. The script checks some ordering conditions but is not a full forensic clock-reconstruction system.

An owner who receives an inconsistent timeline should not simply choose the timestamp that yields a passing target. Corrections need source pointers, reasoning, version history and reviewer acceptance.

RTE3.13 Use an evidence matrix rather than one recovery percentage

A single score such as "recovery 95%" can hide the facts a service owner actually needs. At the rapid level, a more useful assessment lists separate questions:

The matrix avoids turning one satisfied row into a universal pass. A service can be technically ready yet unaccepted because data is wrong. A copy may have a recent timestamp yet be unusable. An accepted restricted service may provide a useful function without meeting every normal operating criterion.

A useful handoff preserves all these statuses and names who owns the unresolved decision. The incident lead should not force a single green/red result when different functions and evidence sources remain in different states.

RTE3.14 Reproducible measurement contracts support later improvement

A recovery exercise can improve the organization only if the next rehearsal can use the same definitions. Record the version of the measurement contract, objective owner, starting event, stopping event, required gates, source timestamps, missing-data rules, and tests for an unacceptable condition. A second analyst should be able to repeat the calculation and explain why an unknown result stayed unknown.

Useful retrospective measurements include time from detection to recovery authorization, time from restoration work start to technical readiness, and time from technical readiness to owner acceptance. Those are phase measures. Their sum does not automatically equal the entire outage unless their endpoints and intervals are continuous and consistent. The end-to-end measurement still uses its separately declared start and stop.

These measures can reveal process gaps. If technical readiness occurs promptly but business acceptance takes much longer, a team might need clearer data targets, approver availability or faster audit verification. It would be premature to blame one role without examining the milestone evidence. The exercise should produce a focused improvement ticket with an owner, a test and a review date.

NIST SP 800-184 describes the role of planned recovery testing and improvement in cyber resilience. RTE3's precise timeline and fixture arithmetic are original examples built to make that principle learnable at the rapid level.

RTE3.15 Release integrity and conclusion boundaries

The canonical case's release boundary is not relaxed by this supplement. At 09:37, the learner has R1–R24 but not the later Branch S/P outcome or the separately released P1/C1 and V1 cards. R23 says the last backup job succeeded and the most recent documented restore test was thirty days earlier. It does not provide an interruption timestamp, accepted-service time or data-loss ledger. No canonical RTO or actual RPO result follows.

The K1 candidate timestamps and import durations belong only to their later classroom replay. P1/C1's four booking-state targets belong to a separate controlled comparison. V1's missing audit actor is a separate supplied test failure. MQ3's requests are invented queue rows. The RTE3 W-A/W-B events and artificial acceptance milestones may not be substituted for any of those records.

The practical success of this section is a student able to make three different statements without confusion: "The copy is this old under verified timestamp assumptions"; "These specified accepted operations were or were not found in the tested recovered state"; and "The declared service was accepted after this measured interval, if all necessary receipts exist."

If any input is missing, the conclusion must narrow rather than expand. Unknown timing is not proof of a target miss or target pass; unknown ledger completeness is not zero loss; and unverified service acceptance is not recovery completion.

RTE3 offline laboratory — measure only what the receipts support

Classification: Entirely fictional, independent RTE3 exercise. The date, interruption time, snapshot label, W-A/W-B operations, readiness milestone, gate flags and owner receipt have no relationship to R1–R24, K1, P1/C1, V1, W1, G3-DG, MQ3 or the Branch S/P cards.

Goal: Under a declared 120-minute hypothetical RTO and 30-minute hypothetical RPO, calculate snapshot age, known uncovered accepted writes in a small certified fixture ledger and elapsed interruption-to-owner-acceptance time only when justified. Negative controls must handle a missing interruption timestamp, an old copy, a failed acceptance gate, no owner receipt, a known missing accepted operation, an incomplete ledger and an unverified clock.

Environment: Python 3 standard library, no files, network connections, accounts, personal data or service operations. All rows are reset independently.

Inputs and step-by-step reproduction

The baseline's interruption is the invented 09:45 service failure on 2026-01-15 UTC; the copy is labeled 09:35, technical readiness is 10:20 and owner acceptance is recorded at 11:10. Required toy gates cover data comparison, permitted access, denied unauthorized operation and audit attribution. The declared accepted-operation set after the copy consists only of W-A and W-B; both are initially represented in the recovered set. A ledger_complete=True flag is an explicit toy premise for that selected ledger, not proof about all bookings.

Before running, predict the 85-minute accepted-service measurement; then predict which negative variants must refuse an RTO calculation or return unbounded data coverage. Copy the following into an isolated Python 3 session.

from datetime import datetime
from copy import deepcopy

RTO_TARGET_MIN = 120
RPO_TARGET_MIN = 30
DAY = "2026-01-15"

def at(clock):
    return None if clock is None else datetime.fromisoformat(
        f"{DAY}T{clock}:00+00:00"
    )

BASE = {
    "interrupted": at("09:45"),
    "copy_at": at("09:35"),
    "technical_ready": at("10:20"),
    "accepted_at": at("11:10"),
    "clock_verified": True,
    "gates": {
        "data": True,
        "authorized": True,
        "denied": True,
        "audit": True,
    },
    "owner_receipt": True,
    "accepted_writes": {"W-A", "W-B"},
    "restored_writes": {"W-A", "W-B"},
    "ledger_complete": True,
}

VARIANTS = [
    ("BASELINE_ACCEPTED", {}),
    ("MISSING_INTERRUPTION", {"interrupted": None}),
    (
        "OLD_COPY",
        {
            "copy_at": at("08:50"),
            "accepted_at": None,
            "owner_receipt": False,
            "ledger_complete": False,
            "gates": {
                "data": None, "authorized": True,
                "denied": True, "audit": True,
            },
        },
    ),
    (
        "ACCEPTANCE_FAILED",
        {
            "accepted_at": None,
            "owner_receipt": False,
            "gates": {
                "data": False, "authorized": True,
                "denied": True, "audit": True,
            },
        },
    ),
    ("NO_OWNER_RECEIPT", {"accepted_at": None, "owner_receipt": False}),
    (
        "KNOWN_MISSING_WRITE",
        {
            "accepted_at": None,
            "owner_receipt": False,
            "restored_writes": {"W-A"},
            "gates": {
                "data": False, "authorized": True,
                "denied": True, "audit": True,
            },
        },
    ),
    (
        "INCOMPLETE_LEDGER",
        {
            "accepted_at": None,
            "owner_receipt": False,
            "ledger_complete": False,
        },
    ),
    ("UNVERIFIED_CLOCK", {"clock_verified": False}),
]

def measure(row):
    i, s, t, a = (
        row[key] for key in
        ("interrupted", "copy_at", "technical_ready", "accepted_at")
    )

    if s is None or i is None or s > i or not row["clock_verified"]:
        age = "UNKNOWN"
    else:
        minutes = int((i - s).total_seconds() // 60)
        age = (
            f"{minutes}m_"
            f"{'WITHIN' if minutes <= RPO_TARGET_MIN else 'OVER'}_"
            f"{RPO_TARGET_MIN}m"
        )

    missing = row["accepted_writes"] - row["restored_writes"]
    if not row["ledger_complete"]:
        data = "UNBOUNDED"
    else:
        data = f"{len(missing)}_KNOWN_UNRECOVERED"

    if i is None:
        rto = "UNKNOWN_INTERRUPTION"
    elif not row["clock_verified"]:
        rto = "UNKNOWN_CLOCK"
    elif t is not None and t < i:
        rto = "INVALID_ORDER"
    elif any(value is not True for value in row["gates"].values()):
        rto = "NO_ACCEPTANCE_GATES"
    elif not row["owner_receipt"] or a is None:
        rto = "NO_OWNER_ACCEPTANCE"
    elif a < i or (t is not None and a < t):
        rto = "INVALID_ORDER"
    else:
        minutes = int((a - i).total_seconds() // 60)
        rto = (
            f"{minutes}m_"
            f"{'WITHIN' if minutes <= RTO_TARGET_MIN else 'OVER'}_"
            f"{RTO_TARGET_MIN}m"
        )

    return f"age={age} unrecovered={data} RTO={rto}"

for label, changes in VARIANTS:
    row = deepcopy(BASE)
    row.update(changes)
    print(label, measure(row))

assert measure(BASE) == (
    "age=10m_WITHIN_30m "
    "unrecovered=0_KNOWN_UNRECOVERED "
    "RTO=85m_WITHIN_120m"
)

Supplemental RTE3 full-service acceptance filter

The original eight-case program reproduces its stated fixture, but its RTO branch trusts gate flags without checking that the four required gates exist, the declared ledger is complete, all accepted writes were restored, or a technical-ready milestone exists. With otherwise green flags, removing W-B only from restored_writes still prints RTO=85m_WITHIN_120m; setting gates={} does the same. This is an unsafe full-service acceptance inference. An elapsed interval between supplied timestamps can be arithmetically plausible even when acceptance is unsupported. The supplemental validator below retains the original fixture and demonstrates why those cases must not receive a favorable full-service result.

from copy import deepcopy
from datetime import datetime

DAY = '2026-01-15'
RTO_TARGET_MIN, RPO_TARGET_MIN = 120, 30
REQUIRED_GATES = {'data', 'authorized', 'denied', 'audit'}

def at(clock):
    return None if clock is None else datetime.fromisoformat(f'{DAY}T{clock}:00+00:00')

BASE = {
    'interrupted': at('09:45'), 'copy_at': at('09:35'),
    'technical_ready': at('10:20'), 'accepted_at': at('11:10'),
    'clock_verified': True,
    'gates': {'data': True, 'authorized': True, 'denied': True, 'audit': True},
    'owner_receipt': True,
    'accepted_writes': {'W-A', 'W-B'}, 'restored_writes': {'W-A', 'W-B'},
    'ledger_complete': True,
}

def stricter_measure(row):
    i, s, t, a = (row[k] for k in ('interrupted', 'copy_at', 'technical_ready', 'accepted_at'))
    clocks_ok = row['clock_verified'] is True
    age = 'UNKNOWN'
    if clocks_ok and i is not None and s is not None and s <= i:
        delta = int((i-s).total_seconds() // 60)
        age = f'{delta}m_{"WITHIN" if delta <= RPO_TARGET_MIN else "OVER"}_{RPO_TARGET_MIN}m'
    missing = row['accepted_writes'] - row['restored_writes']
    coverage = 'UNBOUNDED' if row['ledger_complete'] is not True else f'{len(missing)}_KNOWN_UNRECOVERED'
    if i is None:
        outcome = 'UNKNOWN_INTERRUPTION'
    elif not clocks_ok:
        outcome = 'UNKNOWN_CLOCK'
    elif t is None:
        outcome = 'UNKNOWN_TECHNICAL_READY'
    elif t < i or (a is not None and (a < t or a < i)):
        outcome = 'INVALID_ORDER'
    elif set(row['gates']) != REQUIRED_GATES:
        outcome = 'REQUIRED_GATES_MISSING'
    elif any(row['gates'][k] is not True for k in REQUIRED_GATES):
        outcome = 'NO_ACCEPTANCE_GATES'
    elif row['ledger_complete'] is not True:
        outcome = 'DATA_COVERAGE_UNKNOWN'
    elif missing:
        outcome = 'KNOWN_UNRECOVERED_WRITES'
    elif row['owner_receipt'] is not True or a is None:
        outcome = 'NO_OWNER_ACCEPTANCE'
    else:
        minutes = int((a-i).total_seconds() // 60)
        outcome = f'{minutes}m_{"WITHIN" if minutes <= RTO_TARGET_MIN else "OVER"}_{RTO_TARGET_MIN}m'
    return f'age={age} unrecovered={coverage} RTO={outcome}'

VARIANTS = [
    ('BASELINE', {}),
    ('MISSING_WRITE_DESPITE_GREEN_GATES', {'restored_writes': {'W-A'}}),
    ('INCOMPLETE_LEDGER_DESPITE_GREEN_GATES', {'ledger_complete': False}),
    ('NO_REQUIRED_GATES', {'gates': {}}),
    ('NO_TECHNICAL_READY', {'technical_ready': None}),
    ('AUDIT_FAILED', {'gates': {**BASE['gates'], 'audit': False}}),
    ('NO_OWNER_RECEIPT', {'owner_receipt': False, 'accepted_at': None}),
    ('MISSING_INTERRUPTION', {'interrupted': None}),
    ('BAD_TIME_ORDER', {'accepted_at': at('10:19')}),
]
for label, changes in VARIANTS:
    case = deepcopy(BASE)
    case.update(changes)
    print(f'{label}: {stricter_measure(case)}')
assert stricter_measure(BASE).endswith('RTO=85m_WITHIN_120m')

Predict the nine outputs, then compare the supplemental worked result. ledger_complete and gate flags are classroom premises, not authenticated records. The favorable baseline means only that this declared fictional scope passes the toy contract; it cannot establish complete institutional data recovery, zero actual loss, or an accepted Harbor Library service. A restricted service needs a separately approved scope and receipt.

Check your eight-line transcript and reasoning in the final worked result. The code demonstrates a declared toy contract; its flags and timestamps are not authenticated recovery receipts.

The script uses artificial UTC-aware timestamps on a single fictional day and Boolean gate values. It does not collect a source, verify that an owner signed anything, reconcile a real database, ensure durable journal replay, test actual access controls, recover a live service or authenticate clock sources. It also assumes no unmodeled interval ambiguity and counts accepted IDs rather than evaluating every business effect.

Additional optional negative controls can be performed on a copied fixture: change accepted_at to precede technical_ready, or set technical_ready before interrupted. The expected status should be INVALID_ORDER when the contradictory condition is reached under the existing gates. Restore the baseline before recording the principal eight-line transcript. These are mathematical-input checks, not extra canonical incident injects.

The student should submit a matrix with scenario, start evidence, snapshot-age screening status, accepted-operation coverage, acceptance-gate status, computed elapsed duration or reason for refusal, and the next specific owner/source request. The successful measurement must state its scope, not merely the numbers.

Lab pass criteria: Reproduce all eight lines; explain why 85 minutes includes the complete fictional interruption-to-acceptance interval; identify the old-copy screening failure without inventing lost-data duration; explain why UNKNOWN_INTERRUPTION, NO_ACCEPTANCE_GATES and NO_OWNER_ACCEPTANCE are different; and keep RTE3 outside R1–R24, K1, P1/C1 and V1.

Practical activity — Build a Recovery Evidence Receipt

Learning goal: Turn a loose recovery statement into an auditable, owner-reviewable recovery measurement request.

Inputs: Use the RTE3 fixture and the pre-existing Chapter 03 objectives for context, but do not import the simulated W-A/W-B records into the canonical case.

Steps:

1. Write a one-sentence service acceptance scope, including the functions a user must perform and the operating restrictions, if any.
2. State who owns the outage-start definition and what evidence would establish the starting timestamp.
3. Name the planned technical-ready measurement and explain why it does not end the service outage under the selected contract.
4. Define four required acceptance gates: booking correctness, permitted action, prohibited action and actor/time audit result. Give each a result state and owner.
5. Name the owner-acceptance receipt and identify what happens if the owner rejects or does not respond.
6. For the data-point question, list claimed snapshot time, interruption time, post-snapshot accepted-operation ledger, restored operations, and the source that would establish completeness.
7. Calculate snapshot age separately from the accepted-service interval when both are supported. If either cannot be calculated, record the missing source and do not substitute a guess.
8. Provide a short handoff with the planned next observation, decision owner and explicit no-production-change boundary.

Expected artifact: A Markdown table headed RTE3 recovery evidence receipt, followed by one paragraph interpreting an age-only result and one paragraph interpreting a measured acceptance time. State the limitations of every figure. Do not call a copied example a completed test unless the learner executed it.

Evaluation: Full credit requires distinguishable start and stop conditions, independently supported acceptance gates, age-versus-loss separation, explicit missing-data states, source attribution and permission boundaries. A paper calculation alone does not prove a restore occurred.

Section synthesis

A recoverable copy, a short import, an owner-approved service and a valid recovery objective are distinct concepts. To measure interruption-to-service-acceptance time, define the user-facing service, verify the start event, keep intermediate readiness tests separate, require all mandatory acceptance evidence and record the owner's decision as the scoped stop event. To discuss the recovery point, screen timestamp age but separately establish what accepted business changes are recoverable, missing, unverified or outside source coverage.

RTE3 is deliberately fictional and offline. Its results do not change the canonical Harbor Library chronology, the separate K1/P1/C1/V1 evidence or the public book's completion status.

Worked checkpoint

A completed backup job gives a candidate to evaluate, not a usable service. An isolated restore must compare provenance, recovered data point, elapsed recovery time and owner-approved booking correctness before any return decision.

Practice — Lesson 13

EX033 — Recovery test

Objective: distinguish restoration planning from a measured result. Inputs: R23 and, after the later lab card is released, P1; V1 remains a separate access-and-audit trial. Output: a one-paragraph isolated-copy test plan plus a four-row B17–B20 expected-state table. Name the copy identifier and timestamp as fields to collect, actual restore start/end as fields to measure, service-owner-approved target, pass/fail/not-tested rule, and privacy boundary. A score of 2 keeps all four rows not tested until the restore is performed and never changes a live booking.

RTE3 extension: Extend the original one-paragraph isolated-copy plan with a separate, compact measurement card naming (a) the owner-defined service interruption start and source, (b) isolated restoration start and technical-ready times, (c) the booking, access and audit gate receipt fields, (d) the owner acceptance timestamp and scope, and (e) the rule for declaring elapsed time unknown. The existing B17–B20 target table remains not tested until the relevant later card and actual isolated result are supplied.

RTE3 scoring note: A plan with no actual outage or acceptance timestamps does not report achieved RTO. The learner distinguishes proposed and executed steps, remains within an authorized isolated environment and does not infer an accepted service from a copied test plan.

Worked answer: EX033 in the final appendix.

EX034 — Backup limit

Objective: test what a successful backup job can and cannot demonstrate. Inputs: R23 and the hypothetical two-hour RTO, thirty-minute RPO, midnight-only copy and 09:45 interruption in section 3.5. Output: a calculation note with 09:45 − 00:00, comparison to the thirty-minute target, the missing timing measurement for RTO, and two additional authorized checks before calling the service recoverable. Do not assert that the fictional library approved those targets or that the actual backup was made at midnight. A score of 2 shows the arithmetic, separates copy age from actual data-loss measurement and identifies booking-state and elapsed-restore tests.

RTE3 extension: Keep the existing midnight-to-09:45 arithmetic and hypothetical 30-minute RPO/120-minute RTO. Add three explicit columns: copy age screen, accepted-transaction evidence after the copy, and actual restored/verified transaction result. Ask whether any record proves a specific legitimate write was irrecoverably lost. If the accepted ledger, replay evidence or clock semantics are missing, record unknown, not "9 hours 45 minutes of data lost."

RTE3 scoring note: Correct age arithmetic must be paired with an appropriately limited inference. R23 supplies only a backup-job status and old restore-test record, not interruption, service acceptance, actual recovered point or measured data loss. An isolated import duration alone is not a service-level RTO.

Worked answer: EX034 in the final appendix.

Continue studying — lesson 13 of at least 20

← Lesson 12 · Course lesson index · Lesson 14 →

Continue to the next authored daily lesson.