Operations
The one fact to hold onto: a control-plane outage doesn't turn flags off. SDKs keep serving the last good rules from memory. Most incidents here are "changes stopped arriving", not "users got wrong values".
Health
| Path | For |
|---|---|
/live | Liveness probe. Never checks a dependency. |
/ready | Readiness probe. Fails on Postgres or partition runway. |
/health | People. The same body as /ready, always 200, with snapshot age per environment. |
/metrics | Prometheus. |
curl -s http://127.0.0.1:3000/health | jq .snapshotAgeSeconds
# { "production": 4, "staging": 11 }
Alerts
Each one fires before a user sees a wrong value, or says plainly that it doesn't.
Page someone
| Alert | Means |
|---|---|
max by (environment) (featurekit_snapshot_age_seconds) > 300 | Changes aren't reaching this node. What it serves is still valid, just frozen. |
increase(featurekit_snapshot_compile_failures_total[10m]) > 0 | A configuration won't compile. The last snapshot keeps serving; changes silently aren't landing. |
sum(rate(featurekit_sdk_default_served_total[5m])) > 0 | SDKs are serving your code defaults: the closest thing to "users are affected". The backstop. |
min(featurekit_partition_runway_days) < 2 | Analytics will start failing when the last partition ends. Fires days early. |
| Readiness failing on more than one node | Usually Postgres. |
Ticket
| Alert | Means |
|---|---|
featurekit_snapshot_degraded_serves_total rising | Serving from memory because the snapshot store can't be read. Often the first sign of database trouble. |
featurekit_stream_connections above 80% of the cap | Add nodes before SDKs start polling. |
featurekit_rollup_lag_seconds > 900 | Metrics are behind. |
featurekit_events_dropped_total rising for 15+ minutes | The analytics buffer is too small, not just absorbing a spike. |
featurekit_rate_limited_total above 1/s on a route | An abusive caller, or a limit too low for a real one. |
5xx above 1% of featurekit_http_requests_total | Look at the logs by x-request-id. |
There are no per-flag or per-user metrics, on purpose: a series per flag per variation would take down your monitoring before it told you anything. Per-flag numbers are in the flag metrics API. No metric carries a context key.
Runbooks
Changes aren't reaching SDKs
/health: is it one environment or all of them?- Look for
snapshot pipeline errorin the logs. The usual cause is configuration changed outside the API that won't compile, like a prerequisite cycle. - Fix it, then make any write to the flag to recompile.
Streaming SDKs miss changes that polling ones see
The node's LISTEN connection is down. Behind pgbouncer, set DATABASE_LISTEN_URL to a
direct Postgres address. It reconnects on its own after a blip, and its snapshots re-check every 10 seconds
meanwhile. Restarting a stuck node also fixes it.
Postgres is down
SDKs keep serving. /ready fails, so nodes leave the load balancer. All writes stop, including the
kill switch over the API. If a flag must go off for one service meanwhile, change that service's own default or
bootstrap. Otherwise, restore Postgres.
Partition runway below two days
pg_partman maintenance stopped, usually because pg_cron isn't running. Safe to run now,
and idempotent: SELECT featurekit_ensure_partitions(7);
An SDK key leaked
- Revoke it: DELETE …/keys/:keyId. It stops working everywhere within 30 seconds.
- Make a new one and roll it out.
- A leaked client key isn't an incident: it's public by design. A leaked server key is: it reads every rule.
Someone broke a flag's configuration
Find the change with GET …/audit?flag=<key>, then
revert
to its auditId. The revert is validated and audited like any change.
A bad control-plane deploy
SDKs hold their snapshot in memory, so the worst case is that changes stop arriving. Roll back: migrations only
add, so rolling code back is safe. Then check /ready and that snapshot age is falling, and make a
no-op change to a test flag to confirm propagation.
A reconnect storm after an outage
Let it drain. Reconnects are jittered and the stream rate limit bounds the rest. Remove a node that's unhealthy under the load; its SDKs spread across the others.
The full alert definitions and runbooks are in alerts.md and runbooks.md.