Backups and restore
This page is for whoever looks after a Colophon: what to keep copies of, how to take them, and how to come back from them without serving deleted content or firing old webhooks again. Read the section on what has been rehearsed first: some of these steps have been rehearsed end to end, and others are written down but not yet tried.
What to back up
Section titled “What to back up”| What | Where it lives | Back it up? |
|---|---|---|
| The database | PostgreSQL | Yes. It is the only source of truth: content and its history, releases, accounts, tokens (as hashes), jobs. |
| Uploads | The S3 bucket (Garage in the quick start) | Yes, with the store’s own tools. Each verified upload is written once, under a name made from its content, and never changed afterwards, so a plain copy or the provider’s replication is enough. |
| Secrets | The quick start’s four secrets-* volumes; /etc/colophon/credentials/ under systemd |
Yes, separately and safely. A restored cluster keeps its logins’ old passwords, so the server needs the credentials that match them. |
| The backup key | A passphrase or age key file |
Yes, away from the backups. Losing it makes every backup unreadable. |
| Deletion-ledger exports | Files next to each backup | Yes, off the database host. A restore needs them; see The deletion ledger. |
| The cache | Valkey | No. It holds nothing Colophon can’t rebuild. |
What has been rehearsed
Section titled “What has been rehearsed”| Path | State |
|---|---|
| Quick start: database dump and uploads copy, and restoring them | Drilled end to end by this documentation site’s CI, against Colophon’s main branch, on every change to these pages and once a night: back up, remove every container and volume, install afresh, restore, and check that the published entries and the old API token are back. The CI quick start holds no uploads, so for them it proves only that the volume is copied and unpacked; not drilled on a released version |
Encrypted base backups, verify and the daily schedule |
Rehearsed on the repository’s development stack, on a laptop |
| Retention (which backups to keep) | Proven on synthetic backups |
| The staged restore onto clean infrastructure | Rehearsed on the development stack: deletions made after the backup were replayed, an edit made after it was absent, an in-flight job was held, and the restored server served |
| WAL archiving | Runs on the development stack, into a local volume; shipping it to object storage is your own archive_command |
| Restoring to a point in time after the base backup, by replaying archived WAL | Not built. A restore brings the database back as it was when the base backup was taken; edits since are lost, deletions since are reapplied from the ledger |
| A production host (systemd, or a managed database) | Not yet followed end to end |
So, today: take base backups as often as you can afford to lose work, export the deletion ledger with each one, and practise a restore before you need it. A backup existing is not proof that it restores.
The quick start
Section titled “The quick start”The quick start’s backup is a logical dump of the database plus a copy of the uploads volume. Run these in the
quick start’s folder, where compose.yaml is.
Back up
Section titled “Back up”# The database, as the owner, with the password read inside the container (never on your shell):docker compose exec -T postgres sh -c \ 'PGPASSWORD="$(cat /run/secrets/postgres/owner_password)" pg_dump -Fc -h 127.0.0.1 -U colophon colophon' \ >colophon.dump# The uploads, with Garage stopped so the copy is consistent:docker compose stop garagedocker run --rm -v colophon-quickstart_garage:/data:ro -v "$PWD":/backup alpine:3.23 \ tar -czf /backup/uploads.tar.gz -C /data .docker compose start garageUploads can’t be made while Garage is stopped, so do this at a quiet time. Both files are plain, unencrypted copies
of everything Colophon holds: encrypt them (for example gpg --symmetric, or age) and keep the encrypted copies
off this machine, with the key stored apart from them. Never remove the four secrets-*
volumes while you keep the postgres volume: they hold the generated passwords that database expects.
docker volume ls lists every volume; their names start with the Compose project’s name, colophon-quickstart.
Restore
Section titled “Restore”-
Install a fresh quick start and start it once:
docker compose up -d --wait. -
Stop Colophon and Garage:
Terminal window docker compose stop colophon garage -
Load the dump, with the same wrapper that keeps the password inside the container:
Terminal window docker compose exec -T postgres sh -c \'PGPASSWORD="$(cat /run/secrets/postgres/owner_password)" pg_restore --clean --if-exists -h 127.0.0.1 -U colophon -d colophon' \<colophon.dump -
Unpack the uploads into the
garagevolume:Terminal window docker run --rm -v colophon-quickstart_garage:/data -v "$PWD":/backup alpine:3.23 \tar -xzf /backup/uploads.tar.gz -C /data -
Empty the cache, then start everything again:
Terminal window docker compose restart valkeydocker compose up -d --waitValkey keeps nothing on disk, so a restart empties it. Without that, answers cached from releases made after the backup could be served again under the same release numbers until they expire.
-
Check that the content came back: the published entries, through the same token as before the backup:
Terminal window curl -s "http://localhost:8080/content/v1/$PROJECT/production/recipes" | jq -r '.data[].title'curl -s -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $COLOPHON_TOKEN" \"http://localhost:8080/api/v1/projects/$PROJECT/environments/production/collections/recipes/entries"GingerbreadBannockParkinLemon tart200
Production: encrypted base backups
Section titled “Production: encrypted base backups”The production path is continuous WAL archiving plus encrypted, verified base backups of the whole PostgreSQL
cluster, taken with pg_basebackup (a logical dump can’t be the base for replaying WAL).
1. Archive WAL continuously
Section titled “1. Archive WAL continuously”PostgreSQL archives every finished WAL segment through its archive_command. Set, in postgresql.conf, and restart
PostgreSQL:
wal_level = replicaarchive_mode = onarchive_timeout = 60sarchive_command = '/usr/local/bin/archive-wal.sh %p %f'archive_timeout = 60s makes even an idle database switch segments within a minute. The command must exit 0 only
once the segment is stored durably. If it fails, PostgreSQL keeps the segment and retries forever: a growing pg_wal
directory is your alarm. Never make the command “succeed” by dropping a segment, and never delete archived WAL to
free space.
The repository’s archive-wal.sh is that contract for a local directory (WAL_ARCHIVE_DIR, default
/wal-archive): it copies under a temporary name and renames, treats an identical copy as success, refuses to
overwrite a different file, and flushes to disk before succeeding. For object storage, use a tool such as wal-g
with the same behaviour.
Archived WAL holds every change made to the database, so it is as sensitive as a backup, and archive-wal.sh copies
it unencrypted. Keep the archive off the database host, readable only by the backup operator, and encrypted where it
is stored (a tool such as wal-g can encrypt as it uploads).
2. A login for backups
Section titled “2. A login for backups”Base backups run as colophon_backup, a login with replication rights and no access to any table, so the backup
credential is separate from the server’s. colophon admin init-roles makes it when you give it a password file;
existing roles are left as they are, so running it again is safe:
head -c 32 /dev/urandom | base64 | tr '+/' '-_' | tr -d '=\n' | sudo tee /etc/colophon/credentials/backup_password >/dev/nullsudo chmod 0400 /etc/colophon/credentials/backup_passwordsudo colophon admin init-roles --database-url-file /etc/colophon/credentials/migration_database_url \ --app-password-file /etc/colophon/credentials/app_password \ --backup-password-file /etc/colophon/credentials/backup_passwordAllow it a replication connection in PostgreSQL’s pg_hba.conf.
3. Take, verify and keep backups
Section titled “3. Take, verify and keep backups”scripts/backup.sh create [root] # a base backup into <root>/<UTC stamp>/ (default root: backups)scripts/backup.sh verify <dir> # decrypt and check one backupscripts/backup.sh daily [root] # create + verify + ledger export + retention, for a schedulerscripts/backup.sh retention [root] [--apply] # show which backups the keep policy keeps; --apply deletes the restcreate runs pg_basebackup -X stream as colophon_backup, checks the copy with pg_verifybackup against
PostgreSQL’s own manifest, and streams it, compressed, straight into the encryption tool. The unencrypted copy
exists only in a scratch folder inside the PostgreSQL container while it is checked, and is removed once it has been
streamed; it is never written to the backup directory. Each backup directory holds base.tar.gz.gpg (or .age) and manifest.json,
which records the PostgreSQL version, the applied migrations, the payload’s SHA-256, which key file encrypted it
(never the key), and the WAL archive’s state. A failed run leaves only a dot-file, never something that looks like a
backup.
Encryption. With age and age-keygen installed the script uses an age key; otherwise
gpg --symmetric with AES-256. The secret always comes from a file, never a flag:
COLOPHON_BACKUP_PASSPHRASE_FILE (gpg) or COLOPHON_BACKUP_AGE_KEY_FILE (age). Without either, the first create
makes a key file in the clone, mode 600 and ignored by git. In production keep it in your secret store, and keep a
copy somewhere other than the backups. Either way the same file both encrypts and decrypts (with age, the script
encrypts to the public half of the identity file it is given), so the machine that takes backups holds a key that
can read them all: keep that machine, and the file, as private as the database.
verify is what makes a backup count. It checks the payload against the manifest’s SHA-256, decrypts it into a
scratch folder under $TMPDIR (removed when it ends; keep that on a disk only you can read, since it briefly holds
the whole database unencrypted), checks that the result is a PostgreSQL data directory with its WAL, and, with the development stack
running, runs pg_verifybackup on those same decrypted bytes. It ends verify ok: N checks passed, or exits 1.
daily is the job to schedule: it creates a backup, verifies it, writes the deletion-ledger export into the
backup’s own directory as ledger-export.jsonl, applies retention, and appends one line to <root>/daily.jsonl.
The export needs the colophon CLI, built in the clone (target/release/colophon) or named by COLOPHON_BIN, and
runs as the migration login from COLOPHON_MIGRATION_DATABASE_URL, or the development stack’s own when that is unset.
There is no daemon; use cron, a systemd timer or launchd:
17 3 * * * cd /path/to/checkout && scripts/backup.sh daily >> backups/daily.log 2>&1Retention keeps, among verified backups only, the newest of each of the 7 most recent days that have a
backup and the newest of each of the 4 most recent ISO weeks, and always the newest verified backup. A stopped
schedule therefore keeps more old backups, never fewer. Without --apply it only prints the plan (keep, prune).
A directory that doesn’t verify is never pruned: it is listed as hold for a person to look at. Retention never
touches the WAL archive.
The deletion ledger
Section titled “The deletion ledger”Every deletion Colophon makes is recorded in its deletion ledger. A restore replays the ledger so that content
deleted after the backup, such as a page taken down for legal reasons, stays deleted. Export it off the database host
with every backup (daily does), and more often if you can: a restore can only replay deletions an export holds.
colophon admin ledger-export --database-url-file /etc/colophon/credentials/migration_database_url \ --out /backups/ledger-export.jsonlexported the deletion ledger to file /backups/ledger-export.jsonl (0 rows, through id 0)--s3-key deletion-ledger/<high-water>.jsonl puts the export in the COLOPHON_S3_* bucket instead (the CLI reads
COLOPHON_S3_SECRET_KEY from its environment, with no _FILE form). Unlike the server, the CLI doesn’t refuse a
plain http endpoint, so check that COLOPHON_S3_ENDPOINT is https unless the store is on this machine. Each export is complete up to its high-water mark; --after <mark>
writes only the rows past an earlier export, but a restore still needs every file back to before its backup.
Managed databases
Section titled “Managed databases”Base backups need a replication connection, which some managed databases, Neon among them, don’t offer. There, use the provider’s own backups and point-in-time restore, and keep exporting the deletion ledger. That path hasn’t been drilled.
Restore
Section titled “Restore”A restore happens on clean infrastructure: a new, empty PostgreSQL, never the damaged one. It runs in six stages, in order. Each stage must finish before the next may start, and nothing serves, and no webhook or purge is sent, until the database has a new recovery epoch, the deletion ledger is replayed and you have reconciled the jobs that were in flight.
| Stage | What it does | The command it runs |
|---|---|---|
1. prepare |
Checks the payload’s SHA-256 against manifest.json, decrypts it with the key the manifest names, and unpacks it into a fresh data directory |
gpg or age, and tar |
2. verify |
Runs pg_verifybackup on the unpacked files before PostgreSQL first starts, then compares the restored schema with the manifest and with this binary |
colophon admin restore verify --manifest manifest.json (exits 4 if the backup holds migrations this binary doesn’t know: upgrade the binary first) |
3. epoch |
Starts a new recovery epoch in every environment, exactly once, so cached answers and worker claims from before the incident can never be mistaken for new ones | colophon admin restore epoch |
4. ledger |
Replays every deletion made after the backup | colophon admin ledger-replay --input <file>, for each export in order |
5. reconcile |
Freezes every unfinished job as held and lists what you must check; resumes them only when you say so |
colophon admin restore hold, colophon admin restore jobs, then colophon admin restore resume |
6. serve |
Starts the server on the restored database, with an empty cache, and waits for /readyz to say it is ready |
colophon-server, then colophon doctor (the script runs the api role on a scratch Valkey) |
Every colophon admin command here runs as the migration login, against the restored database, with the URL in
COLOPHON_MIGRATION_DATABASE_URL or a file in COLOPHON_MIGRATION_DATABASE_URL_FILE, never on the command line.
With the script
Section titled “With the script”scripts/restore.sh runs the six stages against a backup directory. It restores onto a scratch PostgreSQL and
Valkey that it starts on 127.0.0.1, so it proves a backup comes back without touching anything live; putting the
restored database into service is then your step. It signs in to the restored cluster with the development stack’s
logins, so today it restores backups of that stack only. prepare decrypts the payload into the workspace before
unpacking it, and deletes it after: keep the workspace on a disk only you can read.
scripts/restore.sh backups/<stamp> --ledger backups/<stamp>/ledger-export.jsonl # every stage still pendingscripts/restore.sh status restores/<stamp> # where each stage standsscripts/restore.sh backups/<stamp> --step ledger --ledger <file> # one stage again, after fixing what blocked itscripts/restore.sh teardown restores/<stamp> # remove the scratch containers and volumeEach stage writes a marker in the workspace (restores/<stamp>/ by default, or --into DIR): done with its
result, or blocked with the reason it refused. A stage won’t run while an earlier one is missing or blocked, and a
done stage never runs twice unless you pass --force: a second epoch or a second hold is exactly the mistake the
markers prevent. --from STAGE runs that stage and the ones after it. --ledger may be given more than once; the
files must reach back to the backup’s own point in the ledger, or the ledger stage blocks.
Reconcile before anything is sent
Section titled “Reconcile before anything is sent”held is a real job state: no worker claims it, no retry or expiry touches it, and only resume brings a job back.
Before resuming, go through colophon admin restore jobs:
- Jobs taken from
was=runningwere mid-attempt when the backup was made: the webhook or purge may already have happened. A resumed job is sent with the same idempotency key, so a receiver that honours it applies it once, but check the website connection is still the one you want. - Due schedules fire as soon as work resumes, including a publish scheduled while the system was down. Check they are still wanted.
- Upload, bulk and content-migration work: resume what you keep, cancel what the incident made wrong.
- What happened outside: CDN purges and site builds that ran after the backup aren’t in the database. Note them yourself.
Then colophon admin restore resume returns held jobs to pending, and jobs whose deadline passed get a fresh one
(an hour by default, --lifetime-secs). The script’s --resume-jobs does this at the end of the reconcile stage,
straight after holding, so pass it only when you have already reconciled; otherwise run
colophon admin restore resume against the restored database once you have.
When something goes wrong
Section titled “When something goes wrong”| What you see | What it means | What to do |
|---|---|---|
pg_wal keeps growing; the PostgreSQL log shows archive errors |
The archive destination is failing; PostgreSQL is keeping the segments, not losing them | Fix the destination (mount, credentials, bucket); the backlog archives itself. Never clear pg_wal by hand. |
create stopped part way |
The capture failed; a dot-file may be left in the backup directory | Run it again. The partial file doesn’t count as a backup. |
verify fails the SHA-256 |
The stored bytes aren’t what was captured | Don’t trust that backup; capture again and check the storage. |
pg_verifybackup fails after decrypting |
The files disagree with PostgreSQL’s own manifest | The backup is corrupt; restore from an earlier one. |
verify says skipped (stack not running) |
The deep check didn’t run | Run it again with PostgreSQL available. |
retention prints hold |
A backup doesn’t verify, so it is kept | Look at it by hand. |
The ledger stage is blocked: no exports, or a gap |
Deletions after the backup can’t all be reapplied | Find the missing exports. Nothing serves until the gap is covered. |
The ledger stage is blocked: replay conflicts |
A ledger row can’t be applied to this state | Resolve the object it names, then run --step ledger again. |
restore verify exits 4 |
The backup is newer than this binary | Upgrade the binary, then restore again. |
Related
Section titled “Related”- Installation: the quick start and the systemd install, and upgrading (take a backup first).
- Database TLS: the migration login every
colophon admincommand uses. - Monitoring:
colophon doctorand/readyzafter a restore. - Security notes: where the secrets and the backup key belong.
Was this page helpful?
Either answer opens a short issue on GitHub, so you can say what worked or what was missing.
Colophon home · the product, its blog and how to try it.