fix(analytics): momentum loop starvation, statements period mismatch, empty-snapshot guards #79

Merged

gertjan merged 1 commit from fix/nightly-analytics-silent-failures into main

2026-07-09 08:14:13 +00:00

gertjan commented

2026-07-09 08:05:23 +00:00

Owner

Follow-up to #78: its honest-status fix surfaced a momentum failure on night one; pulling that thread found three more silent production failures. Full evidence trail in the session; summary:

Momentum (`_OperationCancelled` at save, nightly)

_percentile_rank linear-scans the full ~24k-value universe per symbol per window — ~1.9 billion Python comparisons (~9 min, 03:08→03:18 in last night's log) on the event loop. pymongo's heartbeats starve, and the save's delete_many gets its connection cancelled — the same mechanism that killed the correlation persist. Worse: the delete had already executed server-side, so momentum_snapshot is currently empty in production (stale meta only).

Fixed: sort-once + bisect ranking (O(N log N)) on a worker thread; bars processed per fetch batch instead of accumulating ~6.5M pydantic objects; save retried once via a shared retry_once (correlation_refresh now uses it too).

Statements period vocabulary (four months of empty fundamentals)

FinancialStatementsRepository queries {"period": "annual"}, but the collection has stored FMP's vocabulary (FY, Q1..Q4) since the 2026-03-09 refetch. Every reader got nothing: ERP computed 0 symbols every run since March, scores silently lost their fundamentals inputs, and the /batch + /assets statement endpoints returned empty. Queries now translate at the repo boundary (annual→FY, quarter→Q1..Q4, raw values pass through).

Destructive saves + empty-snapshot guards

Momentum and ERP both used delete-then-insert. ERP's 0-symbol run "successfully" replaced its snapshot with an empty one (currently live: total_symbols: 0). Both repos now upsert snapshot_ts-stamped docs first and prune stale after (the #78 correlation pattern), and both tasks and repos refuse to persist an empty snapshot — upstream data regressions must fail loudly instead of quietly overwriting good data.

Tests

bisect ranking ≡ naive scan (randomized); _build_rank_docs ranks/dispersion/sector edge cases
momentum orchestration: empty universe → failure without save; happy path persists
period filter mapping (annual/quarter/passthrough)
momentum + ERP save ordering: upsert before prune, failure leaves old docs, empty snapshot refused
ruff / mypy / 1057 unit tests green

After merge (restores)

Momentum and ERP snapshots are both empty in prod right now. Tonight's crons (03:00 / 04:00) should restore them with this fix deployed — ERP for the first time since March.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JRnthdVhGGt5FeSQ8m3aUA

Follow-up to #78: its honest-status fix surfaced a momentum failure on night one; pulling that thread found three more silent production failures. Full evidence trail in the session; summary: ## Momentum (`_OperationCancelled` at save, nightly) `_percentile_rank` linear-scans the full ~24k-value universe per symbol per window — ~1.9 **billion** Python comparisons (~9 min, 03:08→03:18 in last night's log) on the event loop. pymongo's heartbeats starve, and the save's `delete_many` gets its connection cancelled — the same mechanism that killed the correlation persist. Worse: the delete had already executed server-side, so **momentum_snapshot is currently empty in production** (stale meta only). Fixed: sort-once + `bisect` ranking (O(N log N)) on a worker thread; bars processed per fetch batch instead of accumulating ~6.5M pydantic objects; save retried once via a shared `retry_once` (correlation_refresh now uses it too). ## Statements period vocabulary (four months of empty fundamentals) `FinancialStatementsRepository` queries `{"period": "annual"}`, but the collection has stored FMP's vocabulary (`FY`, `Q1`..`Q4`) since the 2026-03-09 refetch. Every reader got nothing: **ERP computed 0 symbols every run since March**, scores silently lost their fundamentals inputs, and the `/batch` + `/assets` statement endpoints returned empty. Queries now translate at the repo boundary (`annual`→`FY`, `quarter`→`Q1..Q4`, raw values pass through). ## Destructive saves + empty-snapshot guards Momentum and ERP both used delete-then-insert. ERP's 0-symbol run "successfully" **replaced its snapshot with an empty one** (currently live: `total_symbols: 0`). Both repos now upsert `snapshot_ts`-stamped docs first and prune stale after (the #78 correlation pattern), and both tasks and repos refuse to persist an empty snapshot — upstream data regressions must fail loudly instead of quietly overwriting good data. ## Tests - bisect ranking ≡ naive scan (randomized); `_build_rank_docs` ranks/dispersion/sector edge cases - momentum orchestration: empty universe → failure without save; happy path persists - period filter mapping (annual/quarter/passthrough) - momentum + ERP save ordering: upsert before prune, failure leaves old docs, empty snapshot refused - ruff / mypy / 1057 unit tests green ## After merge (restores) Momentum and ERP snapshots are both empty in prod right now. Tonight's crons (03:00 / 04:00) should restore them with this fix deployed — ERP for the first time since March. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01JRnthdVhGGt5FeSQ8m3aUA

gertjan added 1 commit

2026-07-09 08:05:23 +00:00

fix(analytics): momentum loop starvation, statements period mismatch, empty-snapshot guards

Deploy / check (pull_request) Successful in 4m42s

Details

Deploy / deploy (pull_request) Has been skipped

Details

e342119378

Three silent nightly failures found chasing the momentum failure that
PR #78's honest statuses surfaced on their first night:

- momentum_refresh: _percentile_rank linear-scanned the full universe per
  symbol per window (~1.9e9 comparisons, ~9 min of event-loop-blocking
  CPU) — starving pymongo's heartbeats until the save's connection was
  cancelled, the same mechanism as the correlation persist failure.
  Ranking now sorts once and bisects, runs on a worker thread, and bars
  are processed per fetch batch instead of holding ~6.5M pydantic bar
  objects at once. Save retried once (shared retry_once helper, also
  adopted by correlation_refresh).
- FinancialStatementsRepository queried {"period": "annual"} but the
  collection has stored FMP vocabulary ("FY", "Q1".."Q4") since the
  2026-03-09 refetch — every ERP run since then computed 0 symbols and
  scores silently lost their fundamentals inputs; API routes passing
  period="annual" returned empty too. Queries now translate at the repo
  boundary ("annual" → "FY", "quarter" → Q1..Q4; raw values pass through).
- Momentum/ERP saves were destructive delete-then-insert: momentum's
  2026-07-09 cancelled save wiped the collection (delete executed
  server-side, inserts never ran) and ERP's "successful" 0-symbol run
  replaced its snapshot with an empty one. Both repos now upsert stamped
  docs first and prune stale after (correlation's #78 pattern), and both
  tasks + repos refuse to persist an empty snapshot — an upstream data
  regression must fail loudly, not overwrite good data quietly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JRnthdVhGGt5FeSQ8m3aUA