- Shipped
- September 5, 2026 at 3:05 PM UTC
- Author
- Kamo
- Commit
- 86421e8
Next's not-found is a PAGE. It renders through the root layout, which resolves the tenant's organization over the network and serializes the whole app shell and message catalogue — 1,450,309 bytes and ~0.5-0.75s of server render, for /backup_new.zip. So a commodity scanner walking a backup-file wordlist did not get cheap 404s, it got full application renders, and Node runs one request loop per process. On 2026-09-05 that was 1,033 such requests in 21 minutes. The liveness probe on /api/health returns a fixed string and still could not be answered inside its 5s timeout, so kubelet restarted both pods for being busy rather than wedged. Every one of the scanner's requests was refused — 572 x 404, 461 x 499 — so nothing was reached; it was expensive, not dangerous. The proxy now answers those paths itself with a fixed string, using a DENY list of extensions and prefixes rather than an allow list of real routes: an allow list has to be right about every page this app has and every one it grows, and being wrong serves a 404 for a working screen. DELIBERATELY not applied under /api. Those routes already answer an unknown path in a few dozen bytes with no render, and they carry tenant domains inside the path. `.zip` and `.sh` are real top-level domains, so a suffix rule there would refuse a legitimate tenant lookup — for whichever customer owns such a name. The test pins that reason so the exemption is not tidied away later. Two further things this incident exposed: The layout fetched its own organization over the PUBLIC internet. It addressed `https://${host}/api/org/...` where host is the tenant's public hostname, so every server render left the cluster, crossed to our own edge and came back in through Traefik to reach a handler in the process that made the call — a TLS handshake and two Traefik traversals per page, counted against the tenant's own rate limit. Now over loopback, carrying x-forwarded-host explicitly because the handler derives the upstream API base from it and would otherwise resolve every tenant to the platform default. A host that resolves to no organization is also remembered briefly: one such tenant produced 538 upstream lookups in 21 minutes, all re-asking a question with the same answer. And the container declared no resources at all, which is not "unlimited" but BestEffort — the first QoS class evicted and the last given CPU, on a node also running translation inference at eight cores. A CPU limit is still deliberately omitted: CFS throttling stalls the event loop in 100ms quanta, and a throttled SSR render looks exactly like a wedged one, which would cause the very restarts this prevents.