How we recover a service when a host fails

Balta services run on a single node. One PostgreSQL instance, on one host, with no standby and no automatic failover.

That sentence is on our pricing page, our reliability page and in our service level agreement. Putting it in the place where somebody is deciding whether to pay us is deliberate, and this post is the argument for doing it that way.

What we are not claiming #

There is no replication, no warm standby and no automatic failover in the first version. Nothing on this site should be read as promising continuous availability, and if you find a sentence that implies otherwise, it is a defect and we would like to know about it.

Words like "highly available", "always on" and "zero downtime" are banned from our copy. Not discouraged — banned, and checked by a script that fails the build. Those phrases are the ones a reader most needs to be able to trust, which is exactly why they are the ones most often used loosely.

Why not build it #

The honest reason is that doing it properly is more work than we can do well before launch.

A standby is the easy part. What makes high availability real is everything around it: a connection routing layer that moves clients to the new primary, a failover decision that does not trigger on a network blip, fencing that guarantees the old primary cannot accept a write after the new one has, and a replication mode whose data-loss behaviour you can state in one sentence. Get the fencing wrong and you have built a split-brain generator with a nice dashboard.

It also roughly doubles the resource cost of every service, and that cost lands in the price whether or not the customer needs it.

So the first version does one thing that works exactly as described, rather than two things that mostly do. Replication with a warm standby is the first thing we build after V1. It is not on the site as a promise and it is not in the price you pay today.

The failure modes, stated in advance #

There are no recovery times in that table on purpose. Recovery time from a host loss comes from scheduled drills that include restoring from the second repository and recovering every service from a failed host concurrently, and the time an automatic restart takes comes from measuring it on production hardware. Each is published once it has been measured, and not estimated before.

Why the commitment is 99.5% #

We commit to 99.5% monthly availability per service, measured externally, excluding announced maintenance. That allows roughly 3 hours and 39 minutes of unplanned unavailability per month.

We publish the arithmetic because it is what makes the figure honest for this architecture. A single host loss requiring a rebuild from backup consumes a meaningful part of that budget for a mid-sized database. A 99.9% commitment allows under 44 minutes a month, and one such event exceeds it. Committing to it would be committing to something the architecture cannot deliver, and the first time it mattered we would be explaining why the credit does not apply.

Availability is to be measured from at least two external probe locations on different networks, at 30-second intervals, and a service counts as unavailable only when a majority of probes fail. A service is available when it accepts a connection, completes a TLS handshake, authenticates and answers a trivial query. Probe data is retained and becomes the evidence for any credit claim, including one we did not notice first.

The probe loop, the majority rule and the uncovered state are written, and a probe set on fewer than two networks refuses to start rather than publish a measurement it cannot defend. For any period no probe record covers, the SLA says how a credit claim is assessed.

The part that is genuinely good #

Three things follow from single-node that we would not want to give up.

The resource guarantee is simple. Your service is one set of cgroup limits on one host. There is no standby quietly consuming a second set of resources, no replication lag to explain, and no question about which node a number refers to.

Recovery is one path, and we exercise it. There is one way your service comes back, and it is the one we test on a cadence with real backups. A failover path that is used once a year is a path nobody has debugged.

The data survives things the service does not. Backups go to two repositories and the second is in a different country. A whole location can be lost and your data is still there. That is a different property from availability, and it is the one that actually keeps a company alive.

Say it before they buy #

The reason this is on the pricing page rather than in a support article is straightforward: it is the fact most likely to make this product the wrong choice for someone.

If your workload genuinely cannot tolerate a rebuild from backup, we would rather tell you here than have you find out during an incident, be angry, and be right. Publishing the limitation costs us some customers we would have signed. It also means that the ones who do sign know what they bought, which is the only kind of trust that survives the first bad night.