Tune watchdog max_duration_ratio after observing real drift data #69

New issue

Open

opened 2026-06-21 13:58:59 +00:00 by gertjan · 0 comments

gertjan commented

2026-06-21 13:58:59 +00:00

Owner

Follow-up to #68 (duration-drift detection).

The max_duration_ratio threshold is currently a guess of 2.5 for scores_refresh and correlation_refresh in config/settings.yaml. It is untuned.

Action: after the drift check has run for a few weeks, inspect task_runs.duration_seconds history for these tasks and adjust:

If alerts are noisy (normal variance trips 2.5x), raise toward 3.0.
If real slowdowns sit under 2.5x of the rolling median, lower toward 2.0.
Consider adding the key to other heavy tasks (daily_refresh, anomaly_detection) once the threshold is trusted.

Reference: get_duration_drift() in src/database/repositories/task_runs.py, check in src/scheduler/tasks/watchdog.py.

Follow-up to #68 (duration-drift detection). The `max_duration_ratio` threshold is currently a guess of `2.5` for `scores_refresh` and `correlation_refresh` in `config/settings.yaml`. It is untuned. **Action:** after the drift check has run for a few weeks, inspect `task_runs.duration_seconds` history for these tasks and adjust: - If alerts are noisy (normal variance trips 2.5x), raise toward 3.0. - If real slowdowns sit under 2.5x of the rolling median, lower toward 2.0. - Consider adding the key to other heavy tasks (`daily_refresh`, `anomaly_detection`) once the threshold is trusted. Reference: `get_duration_drift()` in `src/database/repositories/task_runs.py`, check in `src/scheduler/tasks/watchdog.py`.