gertjan/bodega
1
0
Fork 0
Code Issues 13 Pull requests Projects Releases Packages Wiki Activity Actions

feat(watchdog): duration-drift detection to front-run task OOM/timeout #68

Merged
gertjan merged 1 commit from feat/watchdog-duration-drift into main 2026-06-21 13:57:55 +00:00
Conversation 0 Commits 1 Files changed 4 +249
gertjan commented 2026-06-21 13:53:26 +00:00
Owner
Copy link

The watchdog only ever inspected each task's single most recent run
(staleness, failed status, low coverage) — all post-mortem signals that
fire after a task has already missed or crashed. But task_runs already
stores duration_seconds for every run, and a task creeping toward OOM or
timeout gets slower for several runs first.

Add TaskRunRepository.get_duration_drift(): a window query comparing the
latest successful run to the rolling median of the prior N successes
(latest excluded from its own baseline; returns None when history is too
thin to judge). The watchdog opts a task in via a new optional
max_duration_ratio key and raises a "Duration drift" issue through the
existing consolidated-alert path when latest/baseline exceeds it.

Wired up for scores_refresh and correlation_refresh (the heavy analytics
tasks; correlation_refresh has a prior interrupted-run incident). Tasks
without the key are unaffected — fully opt-in, no schema change.

Co-Authored-By: Claude Opus 4.8 (1M context) noreply@anthropic.com

The watchdog only ever inspected each task's single most recent run (staleness, failed status, low coverage) — all post-mortem signals that fire after a task has already missed or crashed. But task_runs already stores duration_seconds for every run, and a task creeping toward OOM or timeout gets slower for several runs first. Add TaskRunRepository.get_duration_drift(): a window query comparing the latest successful run to the rolling median of the prior N successes (latest excluded from its own baseline; returns None when history is too thin to judge). The watchdog opts a task in via a new optional max_duration_ratio key and raises a "Duration drift" issue through the existing consolidated-alert path when latest/baseline exceeds it. Wired up for scores_refresh and correlation_refresh (the heavy analytics tasks; correlation_refresh has a prior interrupted-run incident). Tasks without the key are unaffected — fully opt-in, no schema change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
feat(watchdog): duration-drift detection to front-run task OOM/timeout
All checks were successful
Deploy / check (pull_request) Successful in 4m10s
Details
Deploy / deploy (pull_request) Has been skipped
Details
aab950ad1e 
The watchdog only ever inspected each task's single most recent run
(staleness, failed status, low coverage) — all post-mortem signals that
fire after a task has already missed or crashed. But task_runs already
stores duration_seconds for every run, and a task creeping toward OOM or
timeout gets slower for several runs first.

Add TaskRunRepository.get_duration_drift(): a window query comparing the
latest successful run to the rolling median of the prior N successes
(latest excluded from its own baseline; returns None when history is too
thin to judge). The watchdog opts a task in via a new optional
max_duration_ratio key and raises a "Duration drift" issue through the
existing consolidated-alert path when latest/baseline exceeds it.

Wired up for scores_refresh and correlation_refresh (the heavy analytics
tasks; correlation_refresh has a prior interrupted-run incident). Tasks
without the key are unaffected — fully opt-in, no schema change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
gertjan deleted branch feat/watchdog-duration-drift 2026-06-21 13:57:55 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
bug

   
documentation

   
duplicate

   
enhancement

   
wontfix
No labels
bug
documentation
duplicate
enhancement
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
gertjan/bodega!68
No description provided.