fix(analytics): offload correlation math off the event loop + bump timeouts #74

Merged

gertjan merged 1 commit from fix/correlation-offload-timeouts into main

2026-07-07 07:56:50 +00:00

gertjan commented

2026-07-07 07:49:26 +00:00

Owner

Follow-up to #73. First night on the staggered schedule removed the OOM
crashes (zero interruptions), but surfaced two issues:

correlation_refresh timed out at 8865s (~2h28m). With swap now backing
the box the run no longer OOM-crashes, but its working set spills to
swap and thrashes — the same computation took 4288s in RAM on Jun 25.
momentum_refresh was skipped entirely: correlation's synchronous
numpy/scipy blocked the scheduler event loop from 01:00 to ~03:28, so
momentum's 03:00 trigger misfired and was dropped.
erp_refresh timed out at 1801s, 1s over its 1800s limit.

Changes:

Extract the CPU-bound correlation math (matrix build, N×N correlation,
neighbour extraction, clustering, lead-lag) into a synchronous
_compute_artifacts() helper and dispatch it via asyncio.to_thread, so
the heavy BLAS/scipy passes no longer block the event loop. Behaviour
is unchanged — same computations, same results, same process memory
(thread, not process pool — no memory doubling). Sibling jobs and the
hourly watchdog can now fire while correlation runs.
correlation_refresh timeout 7200 -> 14400 (4h) to accommodate the
swap-bound runtime until the memory footprint is reduced.
erp_refresh timeout 1800 -> 2400 (matches the ranging_refresh bump).

Note: correlation's peak memory now exceeds physical RAM as the universe
has grown; the timeout bump is a stopgap. Reducing its footprint (chunk
the lead-lag pass / cap the universe) remains a follow-up.

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com

Follow-up to #73. First night on the staggered schedule removed the OOM crashes (zero interruptions), but surfaced two issues: - correlation_refresh timed out at 8865s (~2h28m). With swap now backing the box the run no longer OOM-crashes, but its working set spills to swap and thrashes — the same computation took 4288s in RAM on Jun 25. - momentum_refresh was skipped entirely: correlation's synchronous numpy/scipy blocked the scheduler event loop from 01:00 to ~03:28, so momentum's 03:00 trigger misfired and was dropped. - erp_refresh timed out at 1801s, 1s over its 1800s limit. Changes: - Extract the CPU-bound correlation math (matrix build, N×N correlation, neighbour extraction, clustering, lead-lag) into a synchronous _compute_artifacts() helper and dispatch it via asyncio.to_thread, so the heavy BLAS/scipy passes no longer block the event loop. Behaviour is unchanged — same computations, same results, same process memory (thread, not process pool — no memory doubling). Sibling jobs and the hourly watchdog can now fire while correlation runs. - correlation_refresh timeout 7200 -> 14400 (4h) to accommodate the swap-bound runtime until the memory footprint is reduced. - erp_refresh timeout 1800 -> 2400 (matches the ranging_refresh bump). Note: correlation's peak memory now exceeds physical RAM as the universe has grown; the timeout bump is a stopgap. Reducing its footprint (chunk the lead-lag pass / cap the universe) remains a follow-up. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

gertjan added 1 commit

2026-07-07 07:49:27 +00:00

fix(analytics): offload correlation math off the event loop + bump timeouts

Deploy / check (pull_request) Successful in 4m59s

Details

Deploy / deploy (pull_request) Has been skipped

Details

2bac54a174

Follow-up to #73. First night on the staggered schedule removed the OOM
crashes (zero interruptions), but surfaced two issues:

- correlation_refresh timed out at 8865s (~2h28m). With swap now backing
  the box the run no longer OOM-crashes, but its working set spills to
  swap and thrashes — the same computation took 4288s in RAM on Jun 25.
- momentum_refresh was skipped entirely: correlation's synchronous
  numpy/scipy blocked the scheduler event loop from 01:00 to ~03:28, so
  momentum's 03:00 trigger misfired and was dropped.
- erp_refresh timed out at 1801s, 1s over its 1800s limit.

Changes:
- Extract the CPU-bound correlation math (matrix build, N×N correlation,
  neighbour extraction, clustering, lead-lag) into a synchronous
  _compute_artifacts() helper and dispatch it via asyncio.to_thread, so
  the heavy BLAS/scipy passes no longer block the event loop. Behaviour
  is unchanged — same computations, same results, same process memory
  (thread, not process pool — no memory doubling). Sibling jobs and the
  hourly watchdog can now fire while correlation runs.
- correlation_refresh timeout 7200 -> 14400 (4h) to accommodate the
  swap-bound runtime until the memory footprint is reduced.
- erp_refresh timeout 1800 -> 2400 (matches the ranging_refresh bump).

Note: correlation's peak memory now exceeds physical RAM as the universe
has grown; the timeout bump is a stopgap. Reducing its footprint (chunk
the lead-lag pass / cap the universe) remains a follow-up.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>