fix(scheduler): honest task statuses, non-destructive snapshot saves, dependency-gate fixes #78

Merged

gertjan merged 1 commit from fix/task-status-and-snapshot-persistence into main

2026-07-08 14:45:08 +00:00

gertjan commented

2026-07-08 14:34:24 +00:00

Owner

Post-mortem fixes for last night's correlation run (computed in 14 min ✅, persisted nothing ❌) and the failures it uncovered. Root causes and evidence in the investigation thread on #75.

The four bugs

task_runs lied. instrumented recorded success for any coroutine that returned — including tasks that caught their own exception and returned {"success": False}. The correlation persist has been failing since June 25 with green rows every night. Now: status='failure' + error message; last-success gauge untouched; dependency gate and watchdog see the truth.
save_snapshot was destructive on failure. delete-then-insert wiped peers+clusters when the June 25 save died between the steps — the correlation API served nothing for 13 days and nothing alarmed. Now: upsert new docs stamped with snapshot_ts first, prune older stamps after, meta written before the prune. A failed save leaves yesterday's snapshot serving. (Also applied to lead-lag.)
Model construction starved the event loop. ~50k pydantic docs with millions of nested models built on the loop for ~50s after the compute thread returned → pymongo heartbeats starved → the save's connection handshake was cancelled (_OperationCancelled, last night 01:14:05). Now built via asyncio.to_thread, and saves retry once after 10s on transient PyMongoError.
The dependency gate deadlocked the 03:00/04:00 tasks. All analytics gate on a daily_refresh success within 4 hours, but #73 staggered momentum to 03:00 and erp to 04:00 — 5–6h after daily_refresh (22:00). The gate could never pass, polled its full 3600s max_wait, and because it ran inside asyncio.wait_for, the task timeout fired first (momentum: 3600.2s vs 3600s, lost by 0.2s — every night since the stagger). Now: window 4h → 8h, and the gate wraps outside the instrumented timeout so waiting never consumes the task's budget.

Tests

instrumented failure-dict → failure row + error message; success-dict unaffected
save ordering: upserts strictly before prune; bulk-write failure leaves delete_many uncalled; snapshot_ts stamped; empty lead-lag no-op
_wrap: gate runs before write_start; a slow gate + tiny task timeout still lets the task run
ruff / mypy / 1044 unit tests green

After merge

Manual bodega admin analytics refresh-correlation in the analytics container to restore the missing peers/clusters (13 days stale) without waiting for tonight's cron.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JRnthdVhGGt5FeSQ8m3aUA

Post-mortem fixes for last night's correlation run (computed in 14 min ✅, persisted nothing ❌) and the failures it uncovered. Root causes and evidence in the investigation thread on #75. ## The four bugs 1. **`task_runs` lied.** `instrumented` recorded `success` for any coroutine that returned — including tasks that caught their own exception and returned `{"success": False}`. The correlation persist has been failing since **June 25** with green rows every night. Now: `status='failure'` + error message; last-success gauge untouched; dependency gate and watchdog see the truth. 2. **`save_snapshot` was destructive on failure.** delete-then-insert wiped peers+clusters when the June 25 save died between the steps — the correlation API served nothing for 13 days and nothing alarmed. Now: upsert new docs stamped with `snapshot_ts` first, prune older stamps after, meta written before the prune. A failed save leaves yesterday's snapshot serving. (Also applied to lead-lag.) 3. **Model construction starved the event loop.** ~50k pydantic docs with millions of nested models built on the loop for ~50s after the compute thread returned → pymongo heartbeats starved → the save's connection handshake was cancelled (`_OperationCancelled`, last night 01:14:05). Now built via `asyncio.to_thread`, and saves retry once after 10s on transient `PyMongoError`. 4. **The dependency gate deadlocked the 03:00/04:00 tasks.** All analytics gate on a `daily_refresh` success within **4 hours**, but #73 staggered momentum to 03:00 and erp to 04:00 — 5–6h after daily_refresh (22:00). The gate could never pass, polled its full 3600s max_wait, and because it ran *inside* `asyncio.wait_for`, the task timeout fired first (momentum: 3600.2s vs 3600s, lost by 0.2s — every night since the stagger). Now: window 4h → 8h, and the gate wraps *outside* the instrumented timeout so waiting never consumes the task's budget. ## Tests - `instrumented` failure-dict → `failure` row + error message; success-dict unaffected - save ordering: upserts strictly before prune; bulk-write failure leaves `delete_many` uncalled; snapshot_ts stamped; empty lead-lag no-op - `_wrap`: gate runs before `write_start`; a slow gate + tiny task timeout still lets the task run - ruff / mypy / 1044 unit tests green ## After merge Manual `bodega admin analytics refresh-correlation` in the analytics container to restore the missing peers/clusters (13 days stale) without waiting for tonight's cron. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01JRnthdVhGGt5FeSQ8m3aUA

gertjan added 1 commit

2026-07-08 14:34:24 +00:00

fix(scheduler): honest task statuses, non-destructive snapshot saves, gate fixes

Deploy / check (pull_request) Successful in 5m5s

Details

Deploy / deploy (pull_request) Has been skipped

Details

1f1482543e

Four coupled fixes from the 2026-07-08 post-mortem of the correlation
snapshot that computed in 14 min but persisted nothing — and had been
silently serving no peers/clusters since 2026-06-25:

- metrics.instrumented: a task returning {"success": False, ...} is now
  recorded as status='failure' with its error message, instead of a green
  row. Fake successes hid the broken persist for 13 nights and misled the
  dependency gate and watchdog.
- CorrelationRepository.save_snapshot / save_lead_lag_snapshot: upsert the
  new snapshot's docs (stamped with snapshot_ts) first, prune older stamps
  after. The old delete-then-insert order left the API with zero peers and
  clusters when the 2026-06-25 save failed between the two steps.
- correlation_refresh: build the ~50k persistence models on a worker
  thread — constructing millions of nested pydantic objects on the event
  loop starved pymongo's heartbeats and got the save's connection
  handshake cancelled (_OperationCancelled, 2026-07-08 01:14). Saves are
  retried once after 10s for transient Mongo failures.
- Dependency gate: max_age_hours 4 → 8 so the daily_refresh (22:00)
  freshness window covers the #73-staggered 03:00/04:00 consumers —
  momentum/erp had waited the gate's full 3600s max_wait every night. The
  gate also now runs outside the instrumented timeout, so waiting for a
  predecessor cannot consume the task's own budget (momentum lost that
  race by 0.2s nightly and never ran).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JRnthdVhGGt5FeSQ8m3aUA