fix(scheduler): stagger analytics jobs + loosen timeouts to stop nightly OOM

gertjan commented

2026-07-06 18:39:50 +00:00

Owner

The analytics container was suffering recurring host-level OOM kills
(kernel OOM of the ~39 GB python process; docker OOMKilled=false because
the cgroup 48 GB limit was never reached — the host ran out of RAM).
dmesg confirms kills on Jun 22/24/25/28 and Jul 4/5.

Root cause: the nightly analytics window packed scores (23:30),
correlation (23:45), momentum (00:00) and erp (00:15) into an
overlapping 45-min band, so at ~00:15-00:35 up to four memory-heavy
jobs ran concurrently in one process, driving RSS toward 39 GB.

Effects observed in task_runs:

scores_refresh: 1 success in 6 runs (timeout 3735-3919s, or interrupted)
correlation_refresh: last success 2026-06-25; interrupted almost nightly
ranging_refresh: timing out at ~1805s vs an 1800s limit

Changes (config-only):

Stagger heavy jobs so none overlap: correlation 23:45->01:00,
momentum 00:00->03:00, erp 00:15->04:00.
scores_refresh timeout 3600->5400 (needs up to ~3900s).
ranging_refresh timeout 1800->2400 (was set ~5s below its runtime).

Follow-up (not in this PR): the process holds ~39 GB while idle at 02:00,
implying memory is retained across jobs — evaluate a post-batch restart
of bodega-scheduler-analytics once RSS is measured.

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com

The analytics container was suffering recurring host-level OOM kills (kernel OOM of the ~39 GB python process; docker OOMKilled=false because the cgroup 48 GB limit was never reached — the host ran out of RAM). dmesg confirms kills on Jun 22/24/25/28 and Jul 4/5. Root cause: the nightly analytics window packed scores (23:30), correlation (23:45), momentum (00:00) and erp (00:15) into an overlapping 45-min band, so at ~00:15-00:35 up to four memory-heavy jobs ran concurrently in one process, driving RSS toward 39 GB. Effects observed in task_runs: - scores_refresh: 1 success in 6 runs (timeout 3735-3919s, or interrupted) - correlation_refresh: last success 2026-06-25; interrupted almost nightly - ranging_refresh: timing out at ~1805s vs an 1800s limit Changes (config-only): - Stagger heavy jobs so none overlap: correlation 23:45->01:00, momentum 00:00->03:00, erp 00:15->04:00. - scores_refresh timeout 3600->5400 (needs up to ~3900s). - ranging_refresh timeout 1800->2400 (was set ~5s below its runtime). Follow-up (not in this PR): the process holds ~39 GB while idle at 02:00, implying memory is retained across jobs — evaluate a post-batch restart of bodega-scheduler-analytics once RSS is measured. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

gertjan added 1 commit

2026-07-06 18:39:50 +00:00

fix(scheduler): stagger analytics jobs + loosen timeouts to stop nightly OOM

Deploy / check (pull_request) Successful in 5m24s

Details

Deploy / deploy (pull_request) Has been skipped

Details

2c6d524dc5

The analytics container was suffering recurring host-level OOM kills
(kernel OOM of the ~39 GB python process; docker OOMKilled=false because
the cgroup 48 GB limit was never reached — the host ran out of RAM).
dmesg confirms kills on Jun 22/24/25/28 and Jul 4/5.

Root cause: the nightly analytics window packed scores (23:30),
correlation (23:45), momentum (00:00) and erp (00:15) into an
overlapping 45-min band, so at ~00:15-00:35 up to four memory-heavy
jobs ran concurrently in one process, driving RSS toward 39 GB.

Effects observed in task_runs:
- scores_refresh: 1 success in 6 runs (timeout 3735-3919s, or interrupted)
- correlation_refresh: last success 2026-06-25; interrupted almost nightly
- ranging_refresh: timing out at ~1805s vs an 1800s limit

Changes (config-only):
- Stagger heavy jobs so none overlap: correlation 23:45->01:00,
  momentum 00:00->03:00, erp 00:15->04:00.
- scores_refresh timeout 3600->5400 (needs up to ~3900s).
- ranging_refresh timeout 1800->2400 (was set ~5s below its runtime).

Follow-up (not in this PR): the process holds ~39 GB while idle at 02:00,
implying memory is retained across jobs — evaluate a post-batch restart
of bodega-scheduler-analytics once RSS is measured.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>