- Shipped
- September 3, 2026 at 5:06 AM UTC
- Author
- Kamo
- Commit
- 199e367
Builds had gone from 8 minutes to 30+. I blamed the heap and was wrong. Reading the runner instead of the numbers found it: THREE `next build` processes at 213% CPU and ~4.5 GB each, two of them belonging to runs Forgejo had already marked cancelled, the oldest thirty minutes in. concurrency.cancel-in-progress cancels the RUN, and the runner kills the act container the step was executing in. It does not stop the build. `docker buildx build` talks to the HOST daemon through the mounted socket, so the compile is in buildkit OUTSIDE that container and runs to completion unattended. Every rapid push therefore left a build behind, and they accumulated — which is also why pushing a fix made things worse rather than better. Killing the one orphan by hand took the runner's load from 20 to 10 immediately. Identifying them needs care, and my first attempt would not have worked: the act container's own command is `tail -f /dev/null` and its name carries only a task id, so grepping the container Command for the image name matches nothing. What identifies it is the buildx process INSIDE, whose --tag names this image. The step reaps a container only when it is building THIS image at a DIFFERENT commit and is old enough not to be a sibling that just started. Dry-run against the live runner before shipping: it skipped the dictionary sync job and picked out the stale kamo-internal build, which is exactly the discrimination it needs. The Dockerfile comment is corrected in the same commit rather than left to mislead. The heap was raised to 8 GB on the theory that 4.39 GB RSS against a 4096 MB cap was GC thrashing. It was not: at the 8192 cap the same build uses 4.3 GB — LESS. A process starved of heap does not shrink when given more. The raise is kept as headroom because it costs nothing on a 123 GB host, but it fixed nothing and the comment now says so.