Skip to content

Stage 3: price TimeShift as free and TimeRange by the rows it keeps - #632

Draft
zzylol wants to merge 2 commits into
stack/windows-2-segmentsfrom
stack/windows-3-pricing
Draft

zzylol wants to merge 2 commits into
stack/windows-2-segmentsfrom
stack/windows-3-pricing

Conversation

@zzylol

@zzylol zzylol commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Stack: #574 → #620 → #618 → #621 → #627 → #625 → #628 → #632 → #634 → #616 → #617 → #622 → #624 → #629 → #630 → #631 → #633 → #635 → #636 → #637

Problem

Stage 3 (crates/plan-selection/src/lib.rs) charged every query-time TimeShift (as a pass-through) and every query-time TimeRange (as a filter) one operation per row of the scan below them. Over Example 3a's shared 5-year scan, each shift and each range cost 2,920/s, the same as the whole scan. The shared-segment plan from #628 has 10 of them and the selected shared-input plan has 8, so segments cost 44,384/s against 39,128/s.

Changes (Q66)

  • TimeShift is free at both timings. It only re-labels time. The executor does not rewrite timestamps row by row: the PromQL path folds the offset into the selector's window bounds (promql_fallback.rs, series_window(.., offset, ..)), and the per-entity summary path reads the shifted raw rows from the deployment (physical_planner/mod.rs, Pass 2 sharing and Stage 1 coverage parity with the retired MajorPass #580). So pricing it at zero is not an approximation for this executor.
  • A query-time TimeRange is charged on its output rows, meaning the rows it keeps. Rows are ordered by time, so the range seeks to its span. An ingestion-time range keeps every arriving row, so its charge (1 s of rows) is unchanged.
  • Pane roles (PaneRole) are unchanged. Retained panes and the shifts and ranges that feed only them still cost 0. The newest pane's shift is now free too. The ingestion-time pane test now expects 3 charged nodes instead of 4.
  • docs/design_docs/proposals/stage3-cost-model.md: new "Time shifts and ranges (Q66)" paragraph. Example 1's materialized runner-up changes from 47.407 to 47.340.
  • tools/dag-viewer/examples/planner-layering-example1.json is regenerated (--max-candidates 160). The selection (P82, 4.620) is the same. The 32 plans that use tumbling panes get cheaper because their shifts are now free and their ranges charge only kept rows (for example, P14 goes from 12.84 to 8.44).
  • tools/dag-viewer/examples.json: the 3a, 3b and 4a stories now describe the new selections.
  • Pattern B (Examples 3b/4b): rebuilding the five 1-min panes at query time now takes 130 ms per evaluation instead of 310 ms, so it is under the 200 ms bound. B2 is now valid but costs more than the selected plan (2.167 vs 2.083). Two integration tests that expected the latency rejection were updated (stage3_b_tumbling_candidates_are_valid, stage3_b_rebuilding_a_million_series_meets_the_latency_bound). The unit test latency_bound_rejects_slow_query_time_work still covers the rejection.

Selections (stage_pipeline --example planner-layering-<e>)

Ex Before After
1 P82 shared input, 4.62011 P82, 4.62011 (unchanged)
2 P86 shared summary, 0.0236667 P86, 0.0236667 (unchanged)
3a P365 Kll×5 · shared input, 39,128.0 (segments P487: 44,384.0) P487 Kll · 365d segments · shared segments, 18,104.0 (shared input: 25,112.0)
3b P2 Q1 Kll, 2.08333 P2, 2.08333 (unchanged; P4 is now valid at 2.167)
4a P365 shared input, 54.344 (segments: 61.644) P487 shared segments (query time), 25.144 (shared input: 34.878)
4b P4-m1 tumbling 10m · ingestion ×6, 0.90321 P4-m1, 0.90221 (shift now free)

Test plan

  • New unit test time_shift_is_free_and_time_range_pays_for_the_rows_it_keeps: over a shared 5-year scan, the shift costs 0, and the 1-year and 5-year ranges are charged the same price per kept row.
  • cargo fmt --all --check, cargo clippy --workspace --all-targets -D warnings
  • cargo test --workspace: 1690 passed, 0 failed, 13 ignored (after the rebase on main d4869a7; was 1679 passed, 13 ignored)
  • tools/dag-viewer: python3 -m unittest test_viewer: 25 OK

Stacked on #628 (stack/windows-2-segments).

🤖 Generated with Claude Code

@zzylol
zzylol force-pushed the stack/windows-2-segments branch from 7148bf3 to 14921d0 Compare October 5, 2026 03:07
@zzylol zzylol mentioned this pull request Oct 5, 2026
4 tasks done
zzylol and others added 2 commits October 5, 2026 05:58
…eps (Q66)

Stage 3 charged every query-time TimeShift (as a pass-through) and every
query-time TimeRange (as a filter) one operation per row of the scan
below it. Over Example 3a's shared 5-year scan each shift and range cost
as much as the scan's rows, so the shared-segment plan (10 of them) lost
to the shared-input plan (8).

A TimeShift now costs nothing at either timing: it only re-labels time,
and the executor folds the offset into the time bounds of the read below
it. A query-time TimeRange is charged one operation per row it keeps;
at ingestion time it keeps every arriving row, so nothing changes there.
Pane roles are unchanged.

Example 3a and 4a now select the shared 1-year segments (18,104/s vs
25,112/s for shared input in 3a). Example 3b/4 Pattern B's query-time
panes now take 130 ms per evaluation, within the 200 ms bound, so B2 is
valid (still costlier than the selected plan). Example 1's selection is
unchanged; its plans with tumbling panes get cheaper.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Example 3a and 4a now select the shared 1-year segments, and 3b's
query-time panes meet the latency bound.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol
zzylol force-pushed the stack/windows-3-pricing branch from 44291c9 to 9cf2956 Compare October 5, 2026 06:21
This was referenced Oct 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant