ci(deploy): pin base images, cached builds, retried up -d, cache pruning #77

Merged

gertjan merged 2 commits from ci/deploy-hardening into main

2026-07-07 11:11:45 +00:00

gertjan commented

2026-07-07 11:01:01 +00:00

Owner

Hardening from the 2026-07-07 deploy incident:

Dockerfile base images pinned by digest. Builds run on the prod host, so
an unpinned python:3.14-slim silently adopts whatever the registry serves
on deploy day — interpreter/libc drift no CI run tested. Same for the uv
copy stage. Digest bumps become deliberate, reviewable PRs.
Drop --no-cache from the server build. The lockfile COPY already
invalidates the dependency layer correctly; --no-cache only added a
~10-minute five-image build/unpack I/O storm per deploy — heavy enough
to fail bodega-mongodb's healthcheck mid-deploy, which aborted
docker compose up -d after the old containers were already removed and
left the stack down.
Retry up -d twice (30/60 s backoff) so a transiently unhealthy
dependency doesn't strand freshly recreated containers in Created.
Prune build cache (cap 20 GB; had grown to 107 GB) and dangling images
after each deploy.

Co-Authored-By: Claude Fable 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01JRnthdVhGGt5FeSQ8m3aUA

Hardening from the 2026-07-07 deploy incident: - Dockerfile base images pinned by digest. Builds run on the prod host, so an unpinned python:3.14-slim silently adopts whatever the registry serves on deploy day — interpreter/libc drift no CI run tested. Same for the uv copy stage. Digest bumps become deliberate, reviewable PRs. - Drop --no-cache from the server build. The lockfile COPY already invalidates the dependency layer correctly; --no-cache only added a ~10-minute five-image build/unpack I/O storm per deploy — heavy enough to fail bodega-mongodb's healthcheck mid-deploy, which aborted `docker compose up -d` after the old containers were already removed and left the stack down. - Retry `up -d` twice (30/60 s backoff) so a transiently unhealthy dependency doesn't strand freshly recreated containers in Created. - Prune build cache (cap 20 GB; had grown to 107 GB) and dangling images after each deploy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JRnthdVhGGt5FeSQ8m3aUA

gertjan added 1 commit

2026-07-07 11:01:02 +00:00

ci(deploy): pin base images, cached builds, retried up -d, cache pruning

Deploy / check (pull_request) Successful in 4m46s

Details

Deploy / deploy (pull_request) Has been skipped

Details

3e9f6d0870

Hardening from the 2026-07-07 deploy incident:

- Dockerfile base images pinned by digest. Builds run on the prod host, so
  an unpinned python:3.14-slim silently adopts whatever the registry serves
  on deploy day — interpreter/libc drift no CI run tested. Same for the uv
  copy stage. Digest bumps become deliberate, reviewable PRs.
- Drop --no-cache from the server build. The lockfile COPY already
  invalidates the dependency layer correctly; --no-cache only added a
  ~10-minute five-image build/unpack I/O storm per deploy — heavy enough
  to fail bodega-mongodb's healthcheck mid-deploy, which aborted
  `docker compose up -d` after the old containers were already removed and
  left the stack down.
- Retry `up -d` twice (30/60 s backoff) so a transiently unhealthy
  dependency doesn't strand freshly recreated containers in Created.
- Prune build cache (cap 20 GB; had grown to 107 GB) and dangling images
  after each deploy.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JRnthdVhGGt5FeSQ8m3aUA

gertjan added 1 commit

2026-07-07 11:06:56 +00:00

fix(api): raise uvicorn worker healthcheck timeout 5s → 30s

Deploy / check (pull_request) Successful in 4m13s

Details

Deploy / deploy (pull_request) Has been skipped

Details

fd257fddab

The app import (numpy/scipy/pandas/web3/...) takes ~5 s on a warm host —
exactly uvicorn's default timeout_worker_healthcheck. A spawned worker
that loses that coin flip is SIGKILLed mid-import by its own supervisor,
silently, and the respawn loop repeats: the 2026-07-07 incident where
bodega-app served on one lucky worker while the other three died every
5 seconds. 30 s gives cold starts real headroom; a genuinely hung worker
still gets reaped.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JRnthdVhGGt5FeSQ8m3aUA