Overview

The constraint above is the entire brief: one self-hosted VM, one real Postgres database, and a release cadence of dozens of deploys a day with no acceptable downtime and no acceptable data loss. There's no orchestrator underneath this — no Kubernetes, no managed load balancer, no second region to fail over to. Everything below is built out of a reverse proxy, two running copies of the same image, and a deploy script that refuses to take the easy way out. The app is stateful and has real users on it at any given hour, so every shortcut that trades a little safety for a little convenience has a real failure mode behind it.

Architecture: build once, promote unchanged

Most of the risk in a deploy comes from rebuilding. If staging and production each run their own docker build, they are not actually running the same software — a base image update, a cache miss, a dependency resolved slightly differently, and the artifact that passed every test is not the one serving traffic. The fix is to remove rebuilding from the production path entirely: CI builds exactly one image per release, tags it with the commit SHA rather than latest, and pushes it to a registry. That tag is the only thing that ever changes between staging and production.

Staging runs that exact image against a staging database, migrations included — not a mocked database, not a subset of the schema, the same migration the production database is about to receive. Promotion is then just re-pointing production's deploy script at a tag that already passed; nothing is rebuilt, nothing is re-resolved, and the gap between "tested" and "running" closes to zero.

Why not just restart the container in place

The naive alternative — docker compose up against a new image, in place — is a single container that has to be stopped before its replacement can bind the same port. For the seconds in between, the app is down. Worse, if the new image's migration runs as part of container startup and is wrong, it has already run by the time anyone notices; there's no "before" state left to fall back to without restoring from a backup. Two colors exist specifically to put a health check between "the new version exists" and "the new version takes traffic."

• The image running in production is byte-identical to the one that passed staging — not "the same code, rebuilt".
• A failed staging run can never reach production, because production has nothing to build from a bad commit — only a tag to promote.
• Rollback at the image level is just re-promoting the previous tag, which already has a known-good track record.

Diagram: the build-once, promote-only pipeline — CI builds and tags one image, staging exercises it against live migrations, and production promotes the same tag unchanged.

Health gate design

Production runs two colors of the same image behind a reverse proxy — call them blue and green. At any moment one is live and one is idle. A release boots the idle color, leaves the live color untouched, and only after the idle color proves itself does the proxy move traffic onto it. The proving mechanism is Docker's native container healthcheck: a command the engine runs on an interval, with its own timeout, retry count, and a start-period grace window so a slow-booting app isn't marked unhealthy before it's had a chance to come up.

services:
  app-green:
    image: registry.internal/app:1.42.0
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/healthz"]
      interval: 5s
      timeout: 3s
      retries: 6
      start_period: 20s

Compose healthcheck for the idle color

Each field is tuned against a real failure mode. interval is short enough that a regression introduced right after boot gets caught quickly, but not so short that the check itself adds meaningful load. start_period exists because a cold-starting process — warming caches, opening pool connections — will fail a strict check for the first few seconds without actually being broken; without that grace window the gate would reject every release on principle. retries absorbs a single blip — a slow GC pause, a momentary network hiccup — without escalating it to a rollback, while still catching a check that's consistently failing.

What counts as healthy

A probe that only checks "is the process accepting connections" passes too easily — a process can be up and still be serving garbage. The check this setup runs confirms three separate things, and any one of them failing keeps the color marked unhealthy and out of the routing pool:

• the HTTP server is accepting connections
• the database connection opens successfully
• the database's applied-migration version matches what this image expects

docker compose up -d app-green

until [ "$(docker inspect -f '{{.State.Health.Status}}' app-green)" = "healthy" ]; do
  sleep 2
done

update_proxy_target green
docker compose stop app-blue

Cutover script — wait for healthy, then flip

The deploy script's job is mechanical on purpose: boot the idle color, poll its health status, and only flip the proxy target once Docker itself reports healthy. It never inspects application logs or makes a judgment call — the healthcheck already made the call, and the script just acts on it.

Rollback mechanics

The detail that makes this reliable is what rollback is not: it is not a second code path. The live color is never stopped, never drained, and never touched until the idle color has already proven itself healthy. If the healthcheck never reports healthy, the deploy script simply never reaches the line that flips the proxy — production keeps serving the exact version it was serving before the deploy started, because nothing about it changed.

That removes an entire category of bugs: the rollback path that only gets exercised during an actual incident, written once, never tested, and subtly wrong when it's finally needed. There is no separate revert script to rot in a drawer. Forward progress — taking traffic on the new color — is the thing that has to be earned; staying on the old color is simply what happens by default when nothing earns it.

Diagram: failed-probe timeline — the idle color boots, the healthcheck never reports healthy, and the proxy target never moves; the live color keeps serving without interruption.

This doesn't make rollback foolproof. A healthcheck script with its own bug can report healthy when it shouldn't, or unhealthy when the app is actually fine — the gate is only as honest as the check it runs. And a regression that only shows up under real production load, not the synthetic check, can still cut over cleanly and only surface afterward. That's a separate, harder problem this setup doesn't solve on its own; it's why the check keeps growing more specific over time rather than staying a generic ping.

Migration strategy: expand and contract

Schema changes are where the two-color setup gets demanding, because for the duration of a cutover — and for as long afterward as a rollback might still happen — both the old and new application code have to work against the same database. A migration that only the new code can tolerate is a migration that breaks the old color the moment it's applied, which defeats the entire point of keeping the old color alive as a fallback.

Expand, migrate, contract

The discipline used here is the parallel-change pattern: every breaking schema change is split into three separate steps instead of one. Expand adds the new shape without touching the old — a new nullable column, a new table — so the previous code, which doesn't know it exists, keeps working untouched. Migrate moves data and application logic over to the new shape while both shapes coexist, backfilling existing rows in batches rather than one long-running statement, since a migration that takes milliseconds against a thousand test rows can take hours against a production table with millions. Contract, run only once nothing depends on the old shape anymore, removes it.

-- Expand: add the new column, nullable, no default backfill yet
ALTER TABLE accounts ADD COLUMN display_name text;

-- Migrate: backfill in batches, then have new code read/write
-- display_name while old code keeps reading/writing legacy_name
UPDATE accounts SET display_name = legacy_name
WHERE display_name IS NULL AND id BETWEEN 1 AND 10000;

-- Contract: a later, separate release, once no running
-- instance still reads legacy_name
ALTER TABLE accounts DROP COLUMN legacy_name;

Renaming a column without breaking the running version

Notice the contract step isn't part of the same release as the expand step — it ships later, after the expand has been live long enough that a rollback to the previous version is no longer realistic. Doing it in one step would mean the rollback target itself was deleted by the migration it's supposed to protect against.

Pre-migration snapshots

Expand/contract reduces how often a migration is destructive, but it doesn't reduce that risk to zero — a contract step is destructive by definition, and a mistake in any migration's SQL is still possible no matter how carefully it's staged. Before any migration runs against production, the database is snapshotted, giving every migration — however careful — a known-good point to restore to that has nothing to do with whether the migration itself was reversible.

pg_dump -Fc --no-owner appdb > backups/pre-migrate-$(date +%Y%m%d-%H%M%S).dump

Snapshot taken immediately before a migration runs

This isn't a replacement for continuous backups or point-in-time recovery — it's a cheap, fast-to-reach-for safety net scoped specifically to the next thirty seconds of risk, taken at the one moment risk is about to spike.

Tradeoffs and limitations

None of this is free, and being honest about what it doesn't solve is part of the design:

• Two live colors roughly double the steady-state resource footprint during a cutover window — proportionate on one VM for a side project, a real line item at higher scale.
• It's still one machine. This protects against shipping a bad release, not against the VM itself failing — there's no cross-host failover here.
• The health gate only proves what it checks. An app that returns a healthy status while quietly writing corrupted data still passes.
• Expand/contract has to be applied by discipline and review, not enforced by tooling — nothing stops a rushed migration from being written as one breaking step.
• The contract step is the one most likely to be skipped under deadline pressure, leaving a slow accumulation of columns nothing reads anymore.
• Side effects outside the database — an email sent, a webhook fired, a charge captured — aren't covered by any of this. Only the deploy rolls back; the things it already did in the outside world don't.

References and further reading

The patterns here aren't original — they're well-established, and worth reading in their original form:

• Martin Fowler — Blue-Green Deployment
• Martin Fowler — Parallel Change
• Compose, Break, Repeat — Blue-green deployments with Traefik
• Docker HEALTHCHECK reference
• Bytebase — Postgres rollback and point-in-time recovery