Scaling for large events

Flagpost scales one box, up then out: a single backend process is the default and handles typical CTFs; since v1.4.0 a busy event on a multi-core host can opt into multiple worker processes on the same machine. Multi-machine (horizontal) replicas remain unsupported.

When to go multi-worker

The signature that says the event loop is the bottleneck: one core pinned while the others idle, connection resets or 502s despite DB-pool and CPU headroom. From the project’s published load tests (docs/load-testing — re-run on your own hardware before a big event):

• A single worker on 4 vCPU / 4 GB held 200 concurrent users comfortably, but shed ~45% of steady-state requests as 502s at 500 users — with DB and CPU headroom left. That’s the event-loop ceiling.
• At 1,500 users, the 4-worker A/B took steady-state 502s from 30–45% to 0% and flag-submit p50 from ~1.9 s to 14 ms.
• Multi-worker does not help a fully CPU-saturated box — which is why the default stays 1 worker.

Enabling it

Under Docker (compose): one line —

.env

WEB_CONCURRENCY=4

The entrypoint migrates once before any worker serves, starts the workers, and runs the scheduler sidecar as a single extra process so time-triggered automations, the daily update check, and retention fire once — not N times. Redis is already in the stack, and the compose file already raises Postgres to max_connections=200 for pool headroom.

Without Docker: three things are mandatory —

1. export WEB_CONCURRENCY=N and pass the same N to uvicorn --workers N. The env var is what the app reads: with --workers 4 but WEB_CONCURRENCY unset, no cross-worker relay attaches — broadcasts reach only each socket’s own worker, presence fragments, and the scheduler fires in every worker.
2. REDIS_URL must be set — multi-worker refuses to boot without it, loudly, at startup.
3. Run the singleton scheduler as its own process with the same environment: .venv/bin/python -m scheduler (from backend/). Under multi-worker the web workers deliberately skip the in-process scheduler.

Size Postgres to match: each worker’s pool is DB_CONNECTION_BUDGET // N (+ half that as overflow), so keep the budget under your max_connections.

What changes under multi-worker

• Redis becomes mandatory — WebSocket broadcast frames, and since v1.7.1 the socket evictions that follow a revoked access (a ban, a role change), are relayed across workers over a single Redis pub/sub channel behind the connection manager (ADR-0025), and presence becomes a Redis liveness view with per-worker heartbeats — a crashed worker’s members age out within the TTL (ADR-0026).
• The event bus stays per-worker — each event’s handlers run once, on the emitting worker, so webhooks and emails never double-fire. Only the WebSocket fan-out crosses workers.
• The DB pool becomes a budget — a fixed connection budget is split across workers so N workers can’t oversubscribe Postgres.
• The scheduler runs once, as a sidecar — and since v1.8.0 the code judge’s worker runs with it, so exactly one process judges.

Single-worker attaches none of this and behaves exactly as before.

Tuning knobs

Read by the backend process directly (under compose, add them to the backend service’s environment — only WEB_CONCURRENCY is plumbed through .env for you):

Variable	Default	Meaning
WEB_CONCURRENCY	1	Worker processes. 1 = single process, no Redis needed for realtime; >1 = relay + presence via Redis, scheduler sidecar
DB_POOL_SIZE / DB_MAX_OVERFLOW	30 / 30	Single-worker Postgres pool (60 total checkouts)
DB_POOL_TIMEOUT	30	Seconds to wait for a free connection before failing fast
DB_CONNECTION_BUDGET	100	Multi-worker only: total steady connections, split budget // workers per worker (overflow half that). Keep under Postgres max_connections
REDIS_MAX_CONNECTIONS / REDIS_ACQUIRE_TIMEOUT_SECONDS	50 / 10	Per-component cap on the bounded Redis client pools — the rate limiter, relay, and presence store each hold their own pool at this size (so budget up to ~3× per worker for Redis maxclients); bursts queue instead of erroring
WS_PRESENCE_TTL_SECONDS / WS_PRESENCE_HEARTBEAT_SECONDS	30 / 10	Presence liveness window; TTL must exceed heartbeat + grace
WS_HANDSHAKE_RATE_LIMIT / WS_HANDSHAKE_RATE_WINDOW_SECONDS	60 / 30	Per-user WebSocket handshake rate limit — a reconnect herd can’t recompute a snapshot per socket
WS_MESSAGE_RATE_LIMIT / WS_MESSAGE_RATE_WINDOW_SECONDS	300 / 10	Per-user limit on frames sent to a room that handles them (the collaborative-notes relay; keepalive pings count) — far above fast typing. A socket over it is closed and its client reconnects with its full state. New in v1.8.0
WS_SEND_TIMEOUT_SECONDS	5	Per-socket send timeout; a stalled client is reaped instead of stalling its room
SCOREBOARD_CACHE_TTL_SECONDS	5	TTL backstop on the cached scoreboard read model (cleared on every board-moving event)
ACTIVITY_COALESCE_WINDOW_SECONDS	0.5	Server-side burst coalescing for activity pings

Doors-open logins

v1.4.0 also fixed the login-storm failure mode: argon2 hashing now uses the OWASP server configuration (p=1, memory and time unchanged) and runs on bounded pools, so a burst of first-time logins during onboarding can’t saturate CPU. In the published run this lifted doors-open onboarding from 1,017 to 1,416 of 1,500 users, and login p50 from 3.9 s to 424 ms — on a single worker. No operator action; existing hashes keep verifying.