fix(pipelines): persist anomalies without embeddings, chunk earnings-calendar fetch, unbreak AQR cutoff #87

Merged

gertjan merged 1 commit from fix/silent-pipeline-failures into main

2026-07-14 18:29:54 +00:00

gertjan commented

2026-07-14 18:00:43 +00:00

Owner

Fix A — anomaly persistence discarded on embedding failure

Root cause: src/scheduler/tasks/anomaly_detection.py:embed_and_deduplicate set embeddings = [] when perplexity.embed(texts) raised. The subsequent for anomaly, embedding in zip(anomalies, embeddings) then iterated zero times, so upsert_anomaly was never called for any detected anomaly.

Evidence: a live run detected 169 anomalies above the severity threshold; the anomalies MongoDB collection had 0 documents.

Fix: when embeddings are unavailable, or the returned count doesn't match the anomaly count (a partial/malformed response can't be trusted positionally either), every anomaly is now persisted with embedding=None and check_dedup=False (dedup needs an embedding to compare against). When embeddings are available and line up 1:1, behavior is unchanged (check_dedup=True). A warning logs how many anomalies were persisted without embeddings. Downstream stages (assign_clusters, search_context, interpret_anomalies) already tolerate embedding=None/missing-embedding docs — assign_clusters queries {"embedding": {"$exists": True}} and Anomaly.model_dump(exclude_none=True) omits the field entirely, so no further guard changes were needed.

Fix B — earnings calendar silently near-empty on the default 365-day window

Root cause: src/sources/fmp.py:get_earnings_calendar passes from/to straight through to FMP's /api/v3/earning_calendar. For wide ranges, FMP silently truncates the result from the front of the range instead of erroring — near-term events (the ones the job actually needs) get dropped as the window widens.

Empirical findings (live GETs against FMP, 2026-07-14, all anchored at "today" unless noted):

requested window	events	actual coverage
14d (old `refresh_earnings` default)	4,208	full, correct
90d	29,965	full, correct (07-14..10-12)
95d	29,307	starts 07-20 — 6 days already dropped
120d	13,149	starts 08-14 — 31 days dropped
365d (`populate_earnings_calendar` default)	15	garbage — span 2027-04-19..07-01

Non-zero-start 60-day chunks (e.g. offset 60→120, 120→180 days out) returned full, correct coverage for their span, confirming the corruption is driven by total requested width, not by anchor position.

Fix: added FMPSource.get_earnings_calendar_chunked(from_date, to_date, chunk_days=45) — 45 days is half of the largest window (90d) that still returned full, correct coverage, leaving a comfortable safety margin. Chunks are fetched sequentially and aggregated with dedup on (symbol, date); a failing chunk is logged and skipped rather than aborting the whole run. populate_earnings_calendar now calls this instead of the raw single-shot fetch, and its completion log always reports total events/symbols written (and logs at warning level, not info, when nothing was fetched) so a fast no-op run can't pass for a healthy one in the logs again.

Fix C — AQR adapter: pinned featured article aborted every page

Root cause: src/adapters/aqr_adapter.py:get_articles iterates each page's parsed articles in order and breaks the whole pagination loop the first time it sees an article older than since. _parse_listing_page always puts the site's pinned "featured" article first, regardless of its actual publish date — so whenever that featured article is older than since (which is most of the time, since it's rarely refreshed), the cutoff fires on the very first iteration and every regular article on the page — however recent — is discarded.

Live verification (read-only GETs against aqr.com, 2026-07-14): the real listing page's featured article is dated 2026-03-18, while the newest regular article is 2026-06-15 — 3 months newer than the featured pin. since=now-14d (2026-06-30) correctly returned 0 articles (nothing is that recent right now); since=now-60d (2026-05-15) returned 2 articles (2026-06-15 and 2026-05-19), confirming the fixed cutoff logic no longer aborts on the stale featured article and correctly walks past it to the real, recent articles.

Fix: _parse_listing_page now tags each article with is_featured. In get_articles, a featured article is only included if it independently passes the date filter — it's never allowed to set reached_cutoff, so the scan continues into the regular articles regardless of the featured article's age.

Tests

tests/unit/test_scheduler_anomaly_detection.py (new): embed failure → all anomalies persisted with embedding=None/check_dedup=False; embed count mismatch → same fallback; embed success → embeddings attached and check_dedup=True.
tests/unit/test_sources_fmp.py: get_earnings_calendar_chunked splits a 365-day range into 9 contiguous 45-day windows; aggregates + dedupes across chunks; one chunk raising doesn't stop the others; events missing symbol/date are skipped; a narrow range collapses to a single chunk.
tests/unit/test_adapters_aqr.py: fixture HTML (trimmed from the real site) with an old featured article + newer regular items — cutoff doesn't fire on the featured article, since before the featured date includes it, and since=None returns everything.

ruff check src/, mypy src/, and pytest -m "not integration" are all green (1075 passed, 27 deselected).

Follow-ups (not in scope for this PR)

Verify the Perplexity embeddings endpoint/model (pplx-embed-v1-0.6b) still exists and is reachable — the pipeline now degrades gracefully and persists anomalies without embeddings when it fails, but dedup/clustering silently stop working until that's fixed.
Consider alerting when a task "succeeds" with zero output (all three bugs here passed task_runs cleanly while doing nothing useful — a cheap output-count check on completion would have caught this much earlier).

Co-Authored-By: Claude Fable 5 noreply@anthropic.com

## Fix A — anomaly persistence discarded on embedding failure **Root cause:** `src/scheduler/tasks/anomaly_detection.py:embed_and_deduplicate` set `embeddings = []` when `perplexity.embed(texts)` raised. The subsequent `for anomaly, embedding in zip(anomalies, embeddings)` then iterated zero times, so `upsert_anomaly` was never called for any detected anomaly. **Evidence:** a live run detected 169 anomalies above the severity threshold; the `anomalies` MongoDB collection had 0 documents. **Fix:** when embeddings are unavailable, or the returned count doesn't match the anomaly count (a partial/malformed response can't be trusted positionally either), every anomaly is now persisted with `embedding=None` and `check_dedup=False` (dedup needs an embedding to compare against). When embeddings are available and line up 1:1, behavior is unchanged (`check_dedup=True`). A warning logs how many anomalies were persisted without embeddings. Downstream stages (`assign_clusters`, `search_context`, `interpret_anomalies`) already tolerate `embedding=None`/missing-embedding docs — `assign_clusters` queries `{"embedding": {"$exists": True}}` and `Anomaly.model_dump(exclude_none=True)` omits the field entirely, so no further guard changes were needed. ## Fix B — earnings calendar silently near-empty on the default 365-day window **Root cause:** `src/sources/fmp.py:get_earnings_calendar` passes `from`/`to` straight through to FMP's `/api/v3/earning_calendar`. For wide ranges, FMP silently truncates the result from the *front* of the range instead of erroring — near-term events (the ones the job actually needs) get dropped as the window widens. **Empirical findings** (live GETs against FMP, 2026-07-14, all anchored at "today" unless noted): | requested window | events | actual coverage | |---|---|---| | 14d (old `refresh_earnings` default) | 4,208 | full, correct | | 90d | 29,965 | full, correct (07-14..10-12) | | 95d | 29,307 | starts 07-20 — 6 days already dropped | | 120d | 13,149 | starts 08-14 — 31 days dropped | | 365d (`populate_earnings_calendar` default) | **15** | garbage — span 2027-04-19..07-01 | Non-zero-start 60-day chunks (e.g. offset 60→120, 120→180 days out) returned full, correct coverage for their span, confirming the corruption is driven by total requested width, not by anchor position. **Fix:** added `FMPSource.get_earnings_calendar_chunked(from_date, to_date, chunk_days=45)` — 45 days is half of the largest window (90d) that still returned full, correct coverage, leaving a comfortable safety margin. Chunks are fetched sequentially and aggregated with dedup on `(symbol, date)`; a failing chunk is logged and skipped rather than aborting the whole run. `populate_earnings_calendar` now calls this instead of the raw single-shot fetch, and its completion log always reports total events/symbols written (and logs at `warning` level, not `info`, when nothing was fetched) so a fast no-op run can't pass for a healthy one in the logs again. ## Fix C — AQR adapter: pinned featured article aborted every page **Root cause:** `src/adapters/aqr_adapter.py:get_articles` iterates each page's parsed articles in order and breaks the whole pagination loop the first time it sees an article older than `since`. `_parse_listing_page` always puts the site's pinned "featured" article first, regardless of its actual publish date — so whenever that featured article is older than `since` (which is most of the time, since it's rarely refreshed), the cutoff fires on the very first iteration and every regular article on the page — however recent — is discarded. **Live verification** (read-only GETs against aqr.com, 2026-07-14): the real listing page's featured article is dated 2026-03-18, while the newest regular article is 2026-06-15 — 3 months newer than the featured pin. `since=now-14d` (2026-06-30) correctly returned 0 articles (nothing is that recent right now); `since=now-60d` (2026-05-15) returned 2 articles (2026-06-15 and 2026-05-19), confirming the fixed cutoff logic no longer aborts on the stale featured article and correctly walks past it to the real, recent articles. **Fix:** `_parse_listing_page` now tags each article with `is_featured`. In `get_articles`, a featured article is only included if it independently passes the date filter — it's never allowed to set `reached_cutoff`, so the scan continues into the regular articles regardless of the featured article's age. ## Tests - `tests/unit/test_scheduler_anomaly_detection.py` (new): embed failure → all anomalies persisted with `embedding=None`/`check_dedup=False`; embed count mismatch → same fallback; embed success → embeddings attached and `check_dedup=True`. - `tests/unit/test_sources_fmp.py`: `get_earnings_calendar_chunked` splits a 365-day range into 9 contiguous 45-day windows; aggregates + dedupes across chunks; one chunk raising doesn't stop the others; events missing symbol/date are skipped; a narrow range collapses to a single chunk. - `tests/unit/test_adapters_aqr.py`: fixture HTML (trimmed from the real site) with an old featured article + newer regular items — cutoff doesn't fire on the featured article, `since` before the featured date includes it, and `since=None` returns everything. `ruff check src/`, `mypy src/`, and `pytest -m "not integration"` are all green (1075 passed, 27 deselected). ## Follow-ups (not in scope for this PR) - Verify the Perplexity embeddings endpoint/model (`pplx-embed-v1-0.6b`) still exists and is reachable — the pipeline now degrades gracefully and persists anomalies without embeddings when it fails, but dedup/clustering silently stop working until that's fixed. - Consider alerting when a task "succeeds" with zero output (all three bugs here passed `task_runs` cleanly while doing nothing useful — a cheap output-count check on completion would have caught this much earlier). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

gertjan added 1 commit

2026-07-14 18:00:43 +00:00

fix(pipelines): persist anomalies without embeddings, chunk earnings-calendar fetch, unbreak AQR cutoff

Deploy / check (pull_request) Successful in 3m18s

Details

Deploy / deploy (pull_request) Has been skipped

Details

b4820b3c58

Three scheduler/adapter pipelines were reporting successful task_runs while
silently producing zero or stale output:

- Anomaly detection dropped 100% of detected anomalies whenever the
  Perplexity embeddings call failed (embeddings=[] made the zip() over
  anomalies iterate zero times). Now every anomaly is persisted regardless,
  with embedding=None and dedup skipped when embeddings are unavailable.
- Earnings calendar population silently returned near-empty data for its
  365-day default window because FMP's /earning_calendar endpoint truncates
  wide ranges from the front. Requests are now chunked into 45-day windows,
  aggregated, and deduped; per-chunk failures no longer abort the whole run.
- The AQR adapter's pagination cutoff aborted immediately on every run
  because the site pins a stale "featured" article first on every listing
  page, which the cutoff loop treated as the newest article. Featured
  articles no longer trigger the cutoff.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

gertjan force-pushed fix/silent-pipeline-failures from b4820b3c58

Deploy / check (pull_request) Successful in 3m18s

Details

Deploy / deploy (pull_request) Has been skipped

Details

to 1a61f0ad05

Deploy / check (pull_request) Successful in 3m43s

Details

Deploy / deploy (pull_request) Has been skipped

Details

2026-07-14 18:26:01 +00:00

Compare