The Capacity Ladder - Symbology Engineering

Infrastructure configurations

Where should the database live, how many app replicas, is a cache tier worth its footprint — each answered by a run, not a debate.

Configuration	Knee	Time to knee	Aborted on	API p95	Pages p95	Health p95
A — dedicated database own Postgres cluster · 1 app replica	13.4it/s	4m16s	api p95 > 800ms	856ms	1.30s	690ms
Memory squeeze: the dedicated Postgres left no page cache on the node — 380 MiB/s disk reads, the app paging to compressed swap, the database-free control endpoint collapsing along with everything else.
B — shared database one Postgres instance for the whole node · 1 app replica	14.4it/s +7% vs A	4m38s	pages p95 > 1.5s	613ms	1.50s	256ms
Consolidation freed the memory; the failure mode flipped to server-side page rendering with the API holding comfortably. Worst-case request tail shrank 3×.
B × 2 replicas shared database · 2 app replicas	9.6it/s −33% vs B	2m58s	pages p95 > 1.5s	531ms	1.61s	595ms
Regression: two 10-connection pools doubled admitted database concurrency — 512 MiB/s disk reads, node at 90% CPU. The single replica’s pool had been functioning as protective backpressure.
C — B + cache tier shared database · 1 app replica · cache-aside reads (valkey)	16.9it/s +17% vs B	5m28s	pages p95 > 1.5s	324ms	1.51s	439ms
Timeline, search, and list queries served from a 96 MB in-memory cache — API p95 halved, cache footprint negligible. What remained at the knee: the uncached per-page queries, the next lever.

Application rungs

Config C held fixed; each release measured as its own rung — ten-minute cooldowns between runs, a known ±2 it/s noise band, one variable at a time.

Rung	Knee	Wire / req	API p95	Pages p95	Health p95
Config C baseline identity encoding, pre-diet payloads	16.9it/s	83 KB	324ms	1.51s	439ms
The winning config before the page-data diet. Filing pages shipped 289 KB raw (46% serialized data); synthesis pages up to 143 KB (79–80% data).
+ gateway gzip compression middleware at the ingress	18.3it/s series best (to here)	21 KB	480ms	1.71s	252ms
4× wire reduction, and compression CPU proved free-to-positive at the gateway — smaller writes release sockets and connections faster. Two invalid runs along the way exposed a load-tool trap (k6 sends no Accept-Encoding by default) and the cooldown rule between runs.
r1 — fetch what you render release 0.1.4 · reference projections instead of full documents	17.6it/s flat, within noise	17.4 KB	469ms	1.54s	293ms
The win landed in payload, not the knee: synthesis pages −72% raw (143 → 40.5 KB; 9.7 KB on the wire) once the sources sidebar stopped fetching full documents it only ever named.
r2 — load-function flatten + streaming release 0.1.5 · 6 serial stages → 3, below-fold sections streamed	12.4 · 15.6it/s regression, 2 runs	17.8 KB	729 / 702ms	1.56s	726 / 558ms
The ladder’s first real catch: the flatten fires ~7 concurrent queries per view where the old waterfall drip-fed them — the serial shape was accidental backpressure on the connection pool. Kept anyway: real users get ~50ms first byte at organic load, and r3 defuses the burst.
r3 — per-page reads join the cache tier release 0.1.6 · page content, hash lookups, diff halves	>30it/s survived the full ramp	17.5 KB	32ms	104ms	30ms
The suite ran out of ramp: first full survival in the series — 21,974 requests, 0 errors, pages p95 104ms against a 1,500ms limit at the 30 it/s hold. r2 vindicated: the burst is toothless against cache hits, and repeat views of the heaviest page class went from seconds to 37ms.
Ceiling hunt — 60 it/s ramp the true post-diet limit	~60it/s 4.5× the Config A knee	17.7 KB	152ms	1.51s	195ms
Survived the entire 1→60 ramp and aborted 78 seconds into the 60 it/s hold, pages p95 just over the line with the database-free control still under its own 200ms target. 38,575 requests, 0 errors. Same host that kneeled at 13.4 when the ladder began.
The knee in people — closed-loop readers release 0.1.7 · one action + ~15s think per reader · persistent cookies	~960readers	17.7 KB	186ms	1.58s	168ms
The same wall, measured in people: ramping simulated readers aborted at 961 concurrent users, whose ~15.3s action cycle arrives at ≈63 it/s — the open-loop and closed-loop suites agree on where the knee is, within 7% of the Little’s-law prediction. 33,429 requests, 0 errors.

METHOD

k6 ramping-arrival-rate over a weighted mix of real routes against the full dataset. Abort thresholds: pages p95 < 1.5s, API p95 < 800ms, errors < 5%. Requests negotiate gzip like a browser.

THE HEALTH CONTROL

One endpoint never touches the database. When it degrades alongside everything else, the node itself (memory, disk, CPU) is the bottleneck — not query queueing. It separated three different collapses in this series.

WIRE BYTES

Mean bytes received per request during the run, measured on the wire — compression and payload diets both show up here.

REFERENCE POINTS

Production on Config B measured 16.3 it/s warm and 12.0 cache-cold after a reboot — cold caches cost ~35%. Run-to-run variance on identical config: ±2 it/s; single-run deltas inside that band are noise.