Star
Run it

Operations

The one fact to hold onto: a control-plane outage doesn't turn flags off. SDKs keep serving the last good rules from memory. Most incidents here are "changes stopped arriving", not "users got wrong values".

Health

PathFor
/liveLiveness probe. Never checks a dependency.
/readyReadiness probe. Fails on Postgres or partition runway.
/healthPeople. The same body as /ready, always 200, with snapshot age per environment.
/metricsPrometheus.
curl -s http://127.0.0.1:3000/health | jq .snapshotAgeSeconds
      # { "production": 4, "staging": 11 }

Alerts

Each one fires before a user sees a wrong value, or says plainly that it doesn't.

Page someone

AlertMeans
max by (environment) (featurekit_snapshot_age_seconds) > 300Changes aren't reaching this node. What it serves is still valid, just frozen.
increase(featurekit_snapshot_compile_failures_total[10m]) > 0A configuration won't compile. The last snapshot keeps serving; changes silently aren't landing.
sum(rate(featurekit_sdk_default_served_total[5m])) > 0SDKs are serving your code defaults: the closest thing to "users are affected". The backstop.
min(featurekit_partition_runway_days) < 2Analytics will start failing when the last partition ends. Fires days early.
Readiness failing on more than one nodeUsually Postgres.

Ticket

AlertMeans
featurekit_snapshot_degraded_serves_total risingServing from memory because the snapshot store can't be read. Often the first sign of database trouble.
featurekit_stream_connections above 80% of the capAdd nodes before SDKs start polling.
featurekit_rollup_lag_seconds > 900Metrics are behind.
featurekit_events_dropped_total rising for 15+ minutesThe analytics buffer is too small, not just absorbing a spike.
featurekit_rate_limited_total above 1/s on a routeAn abusive caller, or a limit too low for a real one.
5xx above 1% of featurekit_http_requests_totalLook at the logs by x-request-id.

There are no per-flag or per-user metrics, on purpose: a series per flag per variation would take down your monitoring before it told you anything. Per-flag numbers are in the flag metrics API. No metric carries a context key.

Runbooks

Changes aren't reaching SDKs
  1. /health: is it one environment or all of them?
  2. Look for snapshot pipeline error in the logs. The usual cause is configuration changed outside the API that won't compile, like a prerequisite cycle.
  3. Fix it, then make any write to the flag to recompile.
Streaming SDKs miss changes that polling ones see

The node's LISTEN connection is down. Behind pgbouncer, set DATABASE_LISTEN_URL to a direct Postgres address. It reconnects on its own after a blip, and its snapshots re-check every 10 seconds meanwhile. Restarting a stuck node also fixes it.

Postgres is down

SDKs keep serving. /ready fails, so nodes leave the load balancer. All writes stop, including the kill switch over the API. If a flag must go off for one service meanwhile, change that service's own default or bootstrap. Otherwise, restore Postgres.

Partition runway below two days

pg_partman maintenance stopped, usually because pg_cron isn't running. Safe to run now, and idempotent: SELECT featurekit_ensure_partitions(7);

An SDK key leaked
  1. Revoke it: DELETE …/keys/:keyId. It stops working everywhere within 30 seconds.
  2. Make a new one and roll it out.
  3. A leaked client key isn't an incident: it's public by design. A leaked server key is: it reads every rule.
Someone broke a flag's configuration

Find the change with GET …/audit?flag=<key>, then revert to its auditId. The revert is validated and audited like any change.

A bad control-plane deploy

SDKs hold their snapshot in memory, so the worst case is that changes stop arriving. Roll back: migrations only add, so rolling code back is safe. Then check /ready and that snapshot age is falling, and make a no-op change to a test flag to confirm propagation.

A reconnect storm after an outage

Let it drain. Reconnects are jittered and the stream rate limit bounds the rest. Remove a node that's unhealthy under the load; its SDKs spread across the others.

The full alert definitions and runbooks are in alerts.md and runbooks.md.