Monitoring
This page is for whoever keeps a Colophon running: how to ask whether it is up and ready, what each answer means, what it writes to its logs, and how it stops. Everything here works the same in the quick start and under systemd.
Health endpoints
Section titled “Health endpoints”Three endpoints answer without a credential and are never rate-limited. Every answer carries
cache-control: no-store, so no proxy keeps them.
| Endpoint | Answers | Use it for |
|---|---|---|
GET /livez |
Whether the process responds. It checks nothing else. | Restarting a process that has hung |
GET /readyz |
Whether this process can safely serve now, and which dependency isn’t up | Load balancers, uptime checks, colophon doctor |
GET /version |
Which build is running | Upgrades, bug reports |
/livez
Section titled “/livez”curl -i https://cms.example.com/livezHTTP/1.1 200 OKcontent-type: application/jsoncache-control: no-storex-request-id: 01a1ee83-ee17-7418-aede-2ca52dcafe80content-length: 17date: Wed, 18 Nov 2026 11:56:26 GMT
{"status":"live"}/livez deliberately ignores the database: restarting a process can’t fix a database outage, so a liveness check
that failed then would only make things worse.
/readyz
Section titled “/readyz”curl -i https://cms.example.com/readyzHTTP/1.1 200 OKcontent-type: application/jsoncache-control: no-storex-request-id: 01a1ee83-ee26-73e8-a236-b18cfe47d115content-length: 71date: Wed, 18 Nov 2026 11:56:26 GMT
{"status":"ready","role":"all","checks":{"database":"up","cache":"up"}}The answer never runs a query. Background checks ping PostgreSQL and the cache every
COLOPHON_PROBE_INTERVAL_MS (2 seconds), each with a deadline of COLOPHON_PROBE_TIMEOUT_MS (1 second), and
/readyz only reads their last result, so calling it often costs nothing and a slow database can’t make it slow.
The database check uses a connection of its own, never the pool requests use, so a busy server isn’t mistaken for a
down database. If a check ever stops running, its dependency is marked down: readiness can’t stay stuck at up.
status is one of five values:
status |
HTTP | When | Send traffic? |
|---|---|---|---|
starting |
503 |
The database hasn’t been checked yet | No |
ready |
200 |
The database is up, and the cache is up or not configured | Yes |
degraded |
200 |
The database is up, but the cache is down or not checked yet | Yes |
not_ready |
503 |
The database is down | No |
draining |
503 |
The server is shutting down | No |
Each entry in checks is up, down, unknown (not checked yet) or not_configured (no cache is set). role is
the process’s role. The body names states only: never hosts, ports or error
messages.
Degraded means the cache is unreachable. Colophon keeps serving everything, reads and writes, from PostgreSQL;
delivery reads are slower because every one reaches the database, and each server counts rate limits on its own
instead of sharing the count. That is why degraded still answers 200: a load balancer should keep sending
traffic. To see it on the quick start, stop its cache, then ask again:
docker compose stop valkeycurl -i https://cms.example.com/readyzHTTP/1.1 200 OKcontent-type: application/jsoncache-control: no-storex-request-id: 01a1ef2b-6265-7274-ba59-a921b9819a29content-length: 76date: Wed, 18 Nov 2026 14:59:20 GMT
{ "status": "degraded", "role": "all", "checks": { "database": "up", "cache": "down" }}And the same server while its database didn’t answer:
HTTP/1.1 503 Service Unavailablecontent-type: application/jsoncache-control: no-storex-request-id: 01a1ee89-b998-74cc-bcb1-b370cc8c2d80content-length: 79connection: closedate: Wed, 18 Nov 2026 12:02:45 GMT
{"status":"not_ready","role":"all","checks":{"database":"down","cache":"down"}}A server without a cache at all is ready, with "cache":"not_configured".
/version
Section titled “/version”curl -s https://cms.example.com/version{"version":"0.1.0-alpha.1","commit":"…","build_date":"2026-11-18","api":["v1"]}commit is the full commit the binary was built from (shortened here). colophon-server --version and
colophon --version print the same build. Quote it when you report a problem.
colophon doctor
Section titled “colophon doctor”colophon doctor reads a server’s /readyz and says, in words and with its exit status, whether the server is
ready and which dependency isn’t. It reports exactly what a load balancer sees.
colophon doctor --server http://127.0.0.1:58000 # systemd installdocker compose exec colophon colophon doctor --server http://127.0.0.1:8080 # quick startserver http://127.0.0.1:8080role allstatus readycache updatabase upA server whose cache is down (the quick start’s, still stopped from above):
docker compose exec -T colophon colophon doctor --server http://127.0.0.1:8080server http://127.0.0.1:8080role allstatus degradedcache downdatabase upWith the cache started again, --json prints one result instead, the same envelope every colophon
command uses:
docker compose start valkeydocker compose exec -T colophon colophon doctor --server http://127.0.0.1:8080 --json{"command":"doctor","data":{"checks":{"cache":"up","database":"up"},"role":"all","server":"http://127.0.0.1:8080","status":"ready"},"version":2}When nothing answers, it says so on standard error, and with --json also as an error result:
docker compose exec -T colophon colophon doctor --server http://127.0.0.1:8099error: http://127.0.0.1:8099: connection faileddocker compose exec -T colophon colophon doctor --server http://127.0.0.1:8099 --json{"command":"doctor","error":{"code":"unreachable","message":"http://127.0.0.1:8099: connection failed"},"version":2}| Exit status | Meaning |
|---|---|
0 |
ready |
5 |
Anything else: degraded, not_ready, starting, draining, or no answer within the timeout |
2 |
--server isn’t an http:// or https:// URL |
6 |
Something answered, but not with a readiness report |
Note that degraded exits 5: doctor asks “is everything up?”, where /readyz asks “may I send traffic?”.
| Option | Default | |
|---|---|---|
--server |
http://localhost:58000, or COLOPHON_SERVER |
The server’s base URL |
--timeout-ms |
5000 |
Give up after this many milliseconds (1 to 120,000) |
--json |
One JSON result instead of text |
The container’s health check
Section titled “The container’s health check”The image’s HEALTHCHECK runs colophon doctor --server http://127.0.0.1:8080 --timeout-ms 3000 every 10 seconds,
with a 5-second timeout, a 60-second start period and 3 retries. It is what docker compose up --wait waits for and
what docker compose ps shows.
Because doctor exits 5 when degraded, a cache outage marks the container unhealthy even though /readyz
still answers 200 and Colophon still serves. Docker doesn’t restart a container for being unhealthy; look at
docker compose ps and docker compose logs valkey.
Load balancers and uptime checks
Section titled “Load balancers and uptime checks”- Point a load balancer at
/readyz, and take a server out of rotation on anything but200. - Keep
COLOPHON_DRAIN_DELAY_MS(5 seconds) longer than the balancer takes to notice: with checks every 500 milliseconds and two failures to drop a server, that is about a second. - Check
/readyzfrom another machine as well as locally, so an outage of the host or its network is noticed too. - Under systemd there is no built-in watchdog: run
colophon doctorfrom your monitoring and alert on a non-zero exit.
Colophon writes structured logs to its standard output. Under systemd they are in the journal
(journalctl -u colophon); in the quick start, docker compose logs colophon. The first administrator’s setup code
is printed to standard error, in a plain box, beside them.
| Setting | Default | |
|---|---|---|
COLOPHON_LOG_FORMAT |
json |
json: one object per line, for a log pipeline. pretty: multi-line and coloured, for a terminal. The quick start sets pretty. |
COLOPHON_LOG |
info |
Which events to log, as tracing filter directives, such as warn or info,colophon_api=debug. An invalid filter stops the server at start. |
At info, a JSON log shows the server starting, its dependencies changing state (once per change, not once per
check), and work events such as releases:
{"timestamp":"2026-11-18T12:02:00.427492Z","level":"INFO","message":"listening","local_addr":"127.0.0.1:8080","role":"all","version":"0.1.0-alpha.1","commit":"…","target":"colophon_server"}{"timestamp":"2026-11-18T12:02:00.432178Z","level":"WARN","message":"dependency is down","dependency":"cache","cause":"Connection refused (os error 111): Connection refused (os error 111)","target":"colophon_server::probes"}{"timestamp":"2026-11-18T12:02:00.446435Z","level":"INFO","message":"dependency is up","dependency":"database","target":"colophon_server::probes"}{"timestamp":"2026-11-18T12:02:00.511837Z","level":"INFO","message":"worker started","owner":"colophon-1-1795003320485927513","target":"colophon_worker"}What to alert on:
ERRORlines:request failedis a server-side failure, with the errorcodeand its cause.WARNlines:dependency is down, and saturation, which you can act on:request refused: no database connection free,request refused: project work bound full,request refused: installation payload bound full.
A client’s mistake (a 404, a validation error) is logged only at debug, as request rejected, so it never
pages anyone.
A line per request
Section titled “A line per request”Request lines are at debug, because at serving rates a line per request would cost more than the requests. Turn
them on with COLOPHON_LOG=info,tower_http=debug. Each request then gets lines like this:
{"timestamp":"2026-11-18T12:08:40.857154Z","level":"DEBUG","message":"finished processing request","latency":"2247 μs","status":404,"target":"tower_http::trace::on_response","span":{"commit":"…","method":"GET","path":"/content/v1/00000000-0000-7000-8000-000000000000/production/recipes","request_id":"01a1ee8f-2496-71be-a025-001fed29dc48","route":"/content/v1/{project}/{environment}/{collection}","version":"0.1.0-alpha.1","name":"request"}}request_idis the same id as the response’sx-request-idheader and an error’srequest_id, so a user’s report leads to exactly one line. A caller’s ownx-request-idis kept if it is safe to log, and replaced if not.routeis the route’s template, good for grouping;pathis the path without its query string. That request asked for?limit=5; the log doesn’t say so.versionandcommitsay which build answered.
Counting answers by route and status from a JSON log:
jq -r 'select(.target=="tower_http::trace::on_response") | "\(.span.route) \(.status)"' colophon.log \ | sort | uniq -c | sort -rnWhat is never logged
Section titled “What is never logged”Colophon keeps these out of its logs by design: secrets from its configuration (a refused setting names the
variable, never the value, and a refused database URL never includes the URL), passwords, tokens and session
cookies, query strings, the token in a sign-in link and the code in a device sign-in (a path that holds one is
written with {token} or {code} in its place), the mail provider’s key and the links it sends, and the text of
drafts.
Colophon doesn’t rotate or cap its own logs. Set rotation in journald or in Docker’s logging driver.
Metrics
Section titled “Metrics”Colophon has no metrics endpoint: there is no /metrics, no Prometheus format and no OpenTelemetry export. The
signals are in the places above:
| Signal | Where |
|---|---|
| Ready, degraded, down | /readyz and colophon doctor |
| Errors and latency | Request lines (status, latency) at tower_http=debug |
| Cache hits | The delivery API’s colophon-cache header: hit, miss, coalesced or bypass |
| Deliveries waiting or failed | The jobs table, read as the migration login |
| Database connections, locks and WAL | PostgreSQL’s own views, such as pg_stat_activity |
| Cache memory and evictions | Valkey’s INFO |
For example, the delivery backlog by state, with its oldest due time. The migration login owns every table, so
run this from a trusted machine; a URL given to psql as an argument holds its password and shows in the process
list, so on a shared host put the password in a ~/.pgpass file (mode 0600) instead:
psql "$COLOPHON_MIGRATION_DATABASE_URL" -XAtq -c \ "select status, count(*), min(next_attempt_at) from jobs group by status"A job’s state is pending, running, retrying, held, succeeded, failed or superseded.
Graceful shutdown
Section titled “Graceful shutdown”On SIGTERM (what systemctl stop and docker stop send) or SIGINT, the server stops in order:
/readyzanswersdraining(503), so load balancers stop sending new requests;- after
COLOPHON_DRAIN_DELAY_MS(5 seconds) it stops accepting connections; - requests already running finish;
- if they haven’t finished
COLOPHON_SHUTDOWN_GRACE_MS(25 seconds) after the signal, it exits anyway and says so; - the worker stops claiming work. A delivery it abandons is claimed again by another worker once its lease
(
COLOPHON_WORKER_LEASE_MS, 30 seconds) runs out.
While it drains:
HTTP/1.1 503 Service Unavailablecontent-type: application/jsoncache-control: no-storex-request-id: 01a1ee89-f34f-7526-9ca7-a36118046ca6content-length: 76connection: closedate: Wed, 18 Nov 2026 12:03:00 GMT
{"status":"draining","role":"all","checks":{"database":"up","cache":"down"}}And its log:
{"timestamp":"2026-11-18T12:02:59.379580Z","level":"INFO","message":"shutdown signal received","signal":"SIGTERM","target":"colophon_server"}{"timestamp":"2026-11-18T12:02:59.379628Z","level":"INFO","message":"shutting down: readiness is now draining","drain_delay_ms":"5000","target":"colophon_server::shutdown"}{"timestamp":"2026-11-18T12:03:04.379413Z","level":"INFO","message":"drain delay over: no longer accepting connections","target":"colophon_server::shutdown"}{"timestamp":"2026-11-18T12:03:04.382378Z","level":"INFO","message":"shutdown complete","target":"colophon_server"}If requests outlive the grace, the last lines are shutdown deadline passed with requests still in flight and
shutdown complete, abandoning requests still in flight, at WARN.
Give the process manager at least the grace before it kills: the systemd unit waits 35 seconds
(TimeoutStopSec=35). Docker waits only 10 seconds by default, so in the quick start add stop_grace_period: 35s
to the colophon service in your compose.override.yaml.
Related
Section titled “Related”- Configuration: every probe, shutdown and log setting.
- Installation: where the quick start and the systemd unit run
colophon doctor. - Backups and restore: what to watch in the backup schedule.
- Rate limits: what clients see when a limit refuses them.
Was this page helpful?
Either answer opens a short issue on GitHub, so you can say what worked or what was missing.
Colophon home · the product, its blog and how to try it.