Conversation
zzylol
force-pushed
the
stack/fix-sql-count-sketches
branch
from
October 5, 2026 03:07
093840a to
1721efd
Compare
This was referenced Oct 5, 2026
A SQL COUNT(*) group's rows all hash its grouping column, so a per-group Count-Min, Count Sketch or UnivMon is a counter with extra memory, and the executor cannot bind them. Stage 3 priced the UnivMon one as valid, so it could select a plan that cannot run. Pass 1 now drops the per-group sketch alternatives of a count over SQL rows; HydraCms stays. PromQL counts are unchanged. Tests: every Pass 1 choice for grouped, ungrouped and ranked COUNT(*) executes to the exact counts, and every candidate Stage 3 prices for those queries and for Example 2 binds in the executor. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…OUNT(*) queries DataFusion 54 plans ORDER BY COUNT(*) against the aggregate itself, so canonicalize promotes the ranked query to TopK over the Count (as it already did for ORDER BY c). The executor has no native TopK, so the compose-and-execute test runs every Pass 1 choice only for the grouped and ungrouped queries; the ranked one's priced candidates still bind. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol
force-pushed
the
stack/fix-sql-count-sketches
branch
from
October 5, 2026 06:21
1721efd to
be12a6b
Compare
This was referenced Oct 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack: #574 → #620 → #618 → #621 → #627 → #625 → #628 → #632 → #634 → #616 → #617 → #622 → #624 → #629 → #630 → #631 → #633 → #635 → #636 → #637
Rebased on main d4869a7 (DF 54). Added
fix: integrate with #614: DF 54 sortsORDER BY COUNT(*)by the aggregate's own column, socanonicalizepromotes the ranked test query toTopKover the Count (as it already didORDER BY c); the executor has no nativeTopK, sosql_count_star_candidates_compose_and_executenow executes every choice only for the grouped and ungrouped queries.priced_sql_count_star_candidates_bindstill covers all three.Problem
For SQL
COUNT(*)(grouped, ungrouped, orORDER BY COUNT(*) DESC LIMIT k), Pass 1 offered per-group Count-Min, Count Sketch and UnivMon realizations. Each one hashes the grouping column with a unit weight (sql_row_count_update). All of them lower to physical plans, but none binds in the executor:keyed summary weight must be a finalized value column.summary family has no native DAG state implementation.Stage 3's capabilities rejected Count-Min and Count Sketch, but it priced the UnivMon candidate as valid, nearly tied with the selected exact plan. So Stage 3 could select a plan that cannot run.
These sketches also do nothing useful here. Every row in a group hashes that group's key, so the sketch is just a counter that takes more memory. HydraCms is the only grouped-count sketch that helps.
Changes
enumerate_local_logical_candidates: when the intent isCountand every child is SQL rows (closed, no PromQL series identity), drop theRealization::Sketchalternatives. HydraCms is added afterwards and stays. PromQLcountandcount_over_timekeep Count-Min, Count Sketch and UnivMon.pass1_sql_coverage::sql_count_star_candidates_compose_and_execute(replaces the grouped-only test): for the grouped, ungrouped andORDER BY COUNT(*) DESC LIMIT 2forms, every Pass 1 choice composes, compiles, binds and executes to the exact counts. HydraCms is included and is exact on this tiny data.priced_sql_count_star_candidates_bindandpriced_example2_candidates_bindcheck that every physical candidateplan_stagesprices withexecutor_models()binds in the executor. Without the fix, both new SQL count tests fail on the UnivMon candidate.sql_count_star_offers_no_per_group_sketch: SQLCOUNT(*)gets exactly pass-through, exactCountand HydraCms. PromQLcount byandcount_over_timestill get Count-Min, Count Sketch and UnivMon.filtered_aggregates::pass1_offers_filtered_count_alternatives(now every alternative executes),sql_hydra_count_dp_equals_exhaustiveandsql_filtered_aggregates_dp_equals_exhaustive(18 → 9 combinations).Notes:
planner_layering_example2'sQ3_FLOAT). With the integer Q3, Stage 3 prices an exactSumover the Int64c * cthat does not bind, because the executor'ssummary_buildonly takes Float64 updates. That is a separate gap and is not fixed here.tools/dag-viewer/examplesfixture is byte-identical.ORDER BY c DESC LIMIT k) does not bind #619 (SQL top-k ordered by an alias).Test plan
cargo fmt --all -- --checkcargo clippy --workspace --all-targets -- -D warningscargo test --workspace: 1671 passed, 0 failed, 21 ignored (re-run after the rebase on main d4869a7)🤖 Generated with Claude Code