A backup you've never restored is a hypothesis

A backup job that exits zero has proved one thing: a backup job exited zero. It has not proved that the archive is readable, that the write-ahead log chain following it is complete, that PostgreSQL will start on the result, or that the result is consistent.

Those are different claims, and the only way to establish them is to restore the backup and look.

What a verification run actually does #

On a cadence set by database size, we take one of your real backups and:

1. Restore it into an isolated temporary environment.
2. Replay the write-ahead log.
3. Start PostgreSQL and confirm it accepts connections.
4. Run consistency checks against a sampled set of relations.
5. Record how long each of those took.
6. Destroy the environment.

Two things come out of it. You can see when your backup was last proven restorable, rather than when it was last taken. And because every run is timed, our commitments about recovery stay current instead of ageing quietly after one pre-launch benchmark.

Why the cadence follows size #

Restoring a 20 GB database is cheap. Restoring a 2 TB database is not: it occupies storage bandwidth, object storage reads and a verification host for a long time. If we verified every database at the same interval, one of two things would be true — either small databases would be verified far less often than they could be, or large ones would be verified so often that verification became a significant fraction of the platform's running cost, which would end up in the price.

So the interval scales with the thing that makes it expensive:

On top of the cadence there are mandatory triggers, because the moments when a backup is most likely to be wrong are the moments something changed:

• Within 48 hours of a service's first successful backup. A backup configuration that has never been exercised is exactly the one you do not want to discover during an incident.
• After any plan change, storage expansion or major version change.
• After any change to backup configuration or to the repository.
• After any archive pipeline failure is resolved.

Verification runs in the same region and provider as the primary repository, so it does not generate object storage egress charges that would end up in the price for no benefit to you.

Continuous checks, in between #

Full restores are the expensive proof. Cheaper checks run all the time:

• Archive pipeline monitoring, continuously. This is the one that matters most.
• Repository and archive consistency, daily.
• Repository checksum verification, weekly.

The archive monitor is what makes the recovery point objective meaningful. Our RPO is the gap between now and the latest point we can actually restore you to, and reaching a point in time requires a verified base backup and a gap-free chain of write-ahead log segments after it. A recent segment upload proves nothing on its own: if a segment in the middle is missing, everything after it is unreachable. We evaluate the chain rather than the last upload, and a gap wakes somebody up.

When the evaluation itself is stale, the RPO shows as unknown rather than as a stale number. An unknown that says so is more useful than a figure that is quietly wrong.

A verification run is not a recovery time #

This is the part we got wrong in an earlier revision of our own specification, and it is worth being explicit about.

A verification run measures one component of recovery: restore and replay, on a dedicated verification host, with no contention, for one database, with the target already available.

Real recovery includes detection, the decision to act, host selection, provisioning, fencing the failed host, the restore, the replay, verification, the endpoint change and its propagation. It usually happens while several other services are being recovered at the same time, competing for the same storage bandwidth.

So there are two measurements and they are not interchangeable:

• restore_test_duration — the uncontended, single-service restore and replay. It tracks the restore component over time and catches regressions. It is a lower bound on recovery, not a recovery time.
• measured_recovery_time — end to end, from a scheduled drill, including everything above. This is the only input to a published recovery time objective.

Verification runs continuously and tells us whether the drill figure is drifting. Drills are scheduled exercises and produce the number.

The numbers we have not published #

We have not published a recovery time objective for a host loss, because we have not run the drills. When we do, the drills will include the cases that make the number honest:

1. A restore from the second repository, which is in a different country and therefore has a different throughput profile. It is the repository we would actually reach for in the scenario that matters most.
2. Recovery of every service from one host, concurrently, after a single host loss. Contention between simultaneous restores is precisely what makes real recovery slower than a lab measurement. Measuring one service in isolation and multiplying by the number of services is not a substitute.
3. A drill in which the failed host comes back unexpectedly partway through, to exercise the fencing and generation checks.

Until those have run, our reliability page publishes no recovery time for a host loss. That is not a hedge. It is the difference between a number you can plan against and a number somebody estimated in a meeting.

What you see #

Per service: when the last successful restore test ran, how long it took, when the next one is scheduled, and your current recovery point exposure. If a service is overdue for verification, that is visible to you and alarming to us.