Skip to content

fix(pass1): offer no per-group sketch for SQL COUNT(*) - #620

Draft
zzylol wants to merge 2 commits into
stack/509-viewer-stagesfrom
stack/fix-sql-count-sketches
Draft

zzylol wants to merge 2 commits into
stack/509-viewer-stagesfrom
stack/fix-sql-count-sketches

Conversation

@zzylol

@zzylol zzylol commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Stack: #574 → #620 → #618 → #621 → #627 → #625 → #628 → #632 → #634 → #616 → #617 → #622 → #624 → #629 → #630 → #631 → #633 → #635 → #636 → #637

Rebased on main d4869a7 (DF 54). Added fix: integrate with #614: DF 54 sorts ORDER BY COUNT(*) by the aggregate's own column, so canonicalize promotes the ranked test query to TopK over the Count (as it already did ORDER BY c); the executor has no native TopK, so sql_count_star_candidates_compose_and_execute now executes every choice only for the grouped and ungrouped queries. priced_sql_count_star_candidates_bind still covers all three.

Problem

For SQL COUNT(*) (grouped, ungrouped, or ORDER BY COUNT(*) DESC LIMIT k), Pass 1 offered per-group Count-Min, Count Sketch and UnivMon realizations. Each one hashes the grouping column with a unit weight (sql_row_count_update). All of them lower to physical plans, but none binds in the executor:

  • Count-Min and UnivMon fail with keyed summary weight must be a finalized value column.
  • Count Sketch fails with summary family has no native DAG state implementation.

Stage 3's capabilities rejected Count-Min and Count Sketch, but it priced the UnivMon candidate as valid, nearly tied with the selected exact plan. So Stage 3 could select a plan that cannot run.

These sketches also do nothing useful here. Every row in a group hashes that group's key, so the sketch is just a counter that takes more memory. HydraCms is the only grouped-count sketch that helps.

Changes

  • enumerate_local_logical_candidates: when the intent is Count and every child is SQL rows (closed, no PromQL series identity), drop the Realization::Sketch alternatives. HydraCms is added afterwards and stays. PromQL count and count_over_time keep Count-Min, Count Sketch and UnivMon.
  • Tests:
    • pass1_sql_coverage::sql_count_star_candidates_compose_and_execute (replaces the grouped-only test): for the grouped, ungrouped and ORDER BY COUNT(*) DESC LIMIT 2 forms, every Pass 1 choice composes, compiles, binds and executes to the exact counts. HydraCms is included and is exact on this tiny data.
    • Invariant: priced_sql_count_star_candidates_bind and priced_example2_candidates_bind check that every physical candidate plan_stages prices with executor_models() binds in the executor. Without the fix, both new SQL count tests fail on the UnivMon candidate.
    • Unit test sql_count_star_offers_no_per_group_sketch: SQL COUNT(*) gets exactly pass-through, exact Count and HydraCms. PromQL count by and count_over_time still get Count-Min, Count Sketch and UnivMon.
    • Updated the alternative counts in filtered_aggregates::pass1_offers_filtered_count_alternatives (now every alternative executes), sql_hydra_count_dp_equals_exhaustive and sql_filtered_aggregates_dp_equals_exhaustive (18 → 9 combinations).

Notes:

  • The Example 2 invariant uses Q3's floating product (as in planner_layering_example2's Q3_FLOAT). With the integer Q3, Stage 3 prices an exact Sum over the Int64 c * c that does not bind, because the executor's summary_build only takes Float64 updates. That is a separate gap and is not fixed here.
  • Example 2's selected plan is unchanged ("Q1 exact · Q2 exact (Count acc) · Q3 exact (Count acc) · shared summary"). The committed tools/dag-viewer/examples fixture is byte-identical.
  • Related: SQL top-k ordered by an alias (ORDER BY c DESC LIMIT k) does not bind #619 (SQL top-k ordered by an alias).

Test plan

  • cargo fmt --all -- --check
  • cargo clippy --workspace --all-targets -- -D warnings
  • cargo test --workspace: 1671 passed, 0 failed, 21 ignored (re-run after the rebase on main d4869a7)

🤖 Generated with Claude Code

zzylol and others added 2 commits October 5, 2026 04:48
A SQL COUNT(*) group's rows all hash its grouping column, so a per-group
Count-Min, Count Sketch or UnivMon is a counter with extra memory, and the
executor cannot bind them. Stage 3 priced the UnivMon one as valid, so it
could select a plan that cannot run. Pass 1 now drops the per-group sketch
alternatives of a count over SQL rows; HydraCms stays. PromQL counts are
unchanged.

Tests: every Pass 1 choice for grouped, ungrouped and ranked COUNT(*)
executes to the exact counts, and every candidate Stage 3 prices for those
queries and for Example 2 binds in the executor.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…OUNT(*) queries

DataFusion 54 plans ORDER BY COUNT(*) against the aggregate itself, so
canonicalize promotes the ranked query to TopK over the Count (as it
already did for ORDER BY c). The executor has no native TopK, so the
compose-and-execute test runs every Pass 1 choice only for the grouped
and ungrouped queries; the ranked one's priced candidates still bind.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant