fix(timeseries): make rekey_symbol linear on compressed chunks #90

Merged

gertjan merged 1 commit from fix/rekey-compressed-chunk-perf into main

2026-07-17 09:39:05 +00:00

gertjan commented

2026-07-17 09:34:55 +00:00

Owner

Rewrite as copy-then-delete in one transaction:

INSERT..SELECT .. ON CONFLICT DO NOTHING copies the old symbol's rows
to the new symbol (new-symbol rows still win on conflict, unchanged
semantics), then DELETE removes the old rows. moved/dropped counts are
derived from the two command tags.
Both statements are bounded by the old symbol's own min/max timestamp
(fetched first) purely for chunk pruning: compressed chunks keep their
full-size empty raw heaps and the planner seq-scans them on DML —
measured 44.9s per statement across the whole hypertable vs 2.4s
pruned on production.
SET LOCAL enable_seqscan = off inside the transaction: even pruned,
the planner prefers seq scans over the bloated raw heaps; index scans
are ~5x faster (8.1s vs 44.9s unbounded).
Bonus: symbols with no bars (most merge actions) return instantly
without opening a write transaction.

Co-Authored-By: Claude Fable 5 noreply@anthropic.com

The re-key UPDATE's correlated NOT EXISTS cannot be pushed into compressed chunks, so it decompressed per candidate row and went quadratic: the 260-day backfill hung 18+ minutes on a single symbol with 1.1k bars (planner cost ~2.9M). Rewrite as copy-then-delete in one transaction: - INSERT..SELECT .. ON CONFLICT DO NOTHING copies the old symbol's rows to the new symbol (new-symbol rows still win on conflict, unchanged semantics), then DELETE removes the old rows. moved/dropped counts are derived from the two command tags. - Both statements are bounded by the old symbol's own min/max timestamp (fetched first) purely for chunk pruning: compressed chunks keep their full-size empty raw heaps and the planner seq-scans them on DML — measured 44.9s per statement across the whole hypertable vs 2.4s pruned on production. - SET LOCAL enable_seqscan = off inside the transaction: even pruned, the planner prefers seq scans over the bloated raw heaps; index scans are ~5x faster (8.1s vs 44.9s unbounded). - Bonus: symbols with no bars (most merge actions) return instantly without opening a write transaction. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

gertjan added 1 commit

2026-07-17 09:34:55 +00:00

fix(timeseries): make rekey_symbol linear on compressed chunks

Deploy / check (pull_request) Successful in 3m42s

Details

Deploy / deploy (pull_request) Has been skipped

Details

3edb7ddfaf

The re-key UPDATE's correlated NOT EXISTS cannot be pushed into
compressed chunks, so it decompressed per candidate row and went
quadratic: the 260-day backfill hung 18+ minutes on a single symbol
with 1.1k bars (planner cost ~2.9M).

Rewrite as copy-then-delete in one transaction:
- INSERT..SELECT .. ON CONFLICT DO NOTHING copies the old symbol's rows
  to the new symbol (new-symbol rows still win on conflict, unchanged
  semantics), then DELETE removes the old rows. moved/dropped counts are
  derived from the two command tags.
- Both statements are bounded by the old symbol's own min/max timestamp
  (fetched first) purely for chunk pruning: compressed chunks keep their
  full-size empty raw heaps and the planner seq-scans them on DML —
  measured 44.9s per statement across the whole hypertable vs 2.4s
  pruned on production.
- SET LOCAL enable_seqscan = off inside the transaction: even pruned,
  the planner prefers seq scans over the bloated raw heaps; index scans
  are ~5x faster (8.1s vs 44.9s unbounded).
- Bonus: symbols with no bars (most merge actions) return instantly
  without opening a write transaction.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>