Skip to content

executor: HydraCms kernel for count and point queries - #600

Draft
zzylol wants to merge 2 commits into
stack/509-x1-stage1-tumblingfrom
stack/509-x8a-hydra-kernel
Draft

zzylol wants to merge 2 commits into
stack/509-x1-stage1-tumblingfrom
stack/509-x8a-hydra-kernel

Conversation

@zzylol

@zzylol zzylol commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Rebased on main d4869a7 (DF 54).

Why

#580 item F / W7: Hydra for count and point queries first. The types already model GroupingStrategy::SharedMultiSubpopulation { kind: HydraCms, .. }, but the executor rejected every shared grouping. Its only Hydra kernel was HydraKLL, which automatic search never emits. Part of #509.

What

  • Before this PR: a SummaryAgg with HydraCms grouping failed at binding (invalid Count-Min dimensions or grouping strategy, or keyed summary weight must be a finalized value column). family_merges ignored grouping, so HydraKll merges type-checked even though they have no accuracy model.

  • After this PR: count by (job) and count of service="checkout" by (job) run against one shared grid, not one CMS per job:

    job count (Int64) checkout (Float64)
    api 4 2
    batch 2 1
    db 3 1

    Each estimate is at least the exact value and within e·N·(1/shared_columns + 1/width) of it. Two builds over a split stream, merged with SummaryMerge, return the same answers as one build over the whole stream.

How

  • summary_kernels/hydra_cms.rs reuses asap_sketchlib::Hydra, whose cells are HydraCounter::CM, with a one-column schema: the whole group tuple is one subpopulation, so rows do not fan out to sub-groupings. Readouts:
    • Item point count: Hydra's median estimator.
    • Group count: the minimum over rows of the cell's Count-Min row mass. With non-negative weights every row overestimates, so the minimum is sound, at most the median, and an integer. sketchlib has no group-mass query, so this mirrors its private subkey encoding. The kernel tests fail if the two encodings drift apart.
  • HydraCmsGroup is the row state: an Arc to the shared grid, plus the group it reads.
  • Factory: create_hydra_cms(family). create_planner_accumulator still rejects shared grouping, because it builds one state per group.
  • Capability checks for the update, the native family, the evaluation and the state payload.
  • New operator SharedSummaryBuild, plus the planner branch in bind_operation.
  • Merge soundness: family_merges is now grouping-aware.
    • HydraCms and HydraCountSketch → true. Same-shape grids add cell-wise under the same hashes, so the result is the grid of the union, and Theorem 2 still holds.
    • HydraKll → false, since it has no accuracy model.
    • The kernel rejects shape mismatches, because sketchlib's CMS merge panics on them. It also caps total weight at i32::MAX, the limit of sketchlib's i32 counters.
  • Tests first: before the wiring, tests/hydra_cms_execution.rs failed 2 of 3 (the binding errors above). After it, they pass.

Planner contract (for the later Pass 1/Stage 1 PR; not emitted yet)

SummaryAgg {
  family:   Sketch(SketchKind(Cms, Cms { width, depth }), G),
  grouping: G = SharedMultiSubpopulation { kind: HydraCms,
              params: HydraCms { width, depth, shared_rows, shared_columns } },
              // width/depth must equal the family's; all four > 0
  input: SummaryUpdate {
    item:   Some(Column(<one item column: Utf8 | Int64 | Bool, non-null>)),
    weight: Constant(1.0)                      // count
          | Column(<Float64, integral, ≥ 0>),  // weighted count
    weight_domain: NonNegative { .. },         // required
  },
  reduction: Reduce(GroupKeys::by(<group columns>)),
  filter: None,
}
SummaryEstimate { query: PointCount { key: SampleValue, value: None } }      // group count → Int64
SummaryEstimate { query: PointCount { key: Named(item), value: Some(v) } }   // item freq in group → Float64
SummaryMerge                                                                  // allowed (family_merges)

Output rows: [group columns…, state], one per group. default_hydra_params already gives this shape from a per-group SketchParams::Cms.

Not in this PR

  • Top-k (W7 defers it).
  • HydraCountSketch and HydraKll kernels.
  • The precompute (raw-sample) path, which still rejects shared grouping.
  • Pass 1 / Stage 1 offering Hydra.
  • Memory accounting: each group row reports the whole grid's size, so reservations are conservative. A SummaryMerge also merges the grid once per group.

Gates

🤖 Generated with Claude Code

@zzylol
zzylol changed the base branch from stack/509-w7b-sql-frequency to stack/509-x1-stage1-tumbling October 4, 2026 19:01
@zzylol
zzylol force-pushed the stack/509-x8a-hydra-kernel branch from 9af256d to 09fa09e Compare October 4, 2026 19:01
zzylol added a commit that referenced this pull request Oct 4, 2026
A count with `by` groups and an approximate target gets one more
alternative (#580 W7): one shared Count-Min grid for every group, as the
SummaryAgg contract of #600 states. The update is a unit weight per row,
hashed by a non-null Utf8/Int64/Bool column (a grouping column first, else
the PromQL series identity). Grouping is a new per-alternative axis on
LocalLogicalTarget, carried through tumbling forms.

Stage 3's analytical guarantee is now grouping-aware: for HydraCms it is
pass1::grouping::hydra_guarantee over the inner Count-Min, with collision
term e/shared_columns and failure probability e^-shared_rows (relative to
the whole input's weight). Pass 1 sizes the inner sketch and the grid for
ε/2 and δ/2 each so the sum meets the target. Other shared groupings have
no model.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol and others added 2 commits October 5, 2026 04:48
One shared grid of Count-Min cells serves every group (#580 item F, W7).
Reads a group's total (min over rows of the cell's mass) and an item's
frequency within a group (Hydra's median). Merge adds cells and is the
grid of the union.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SummaryAgg with SharedMultiSubpopulation { kind: HydraCms } now compiles to
a shared summary build that emits one row per group over one grid, and
SummaryEstimate answers PointCount per group (bare: group count; valued:
item frequency). family_merges admits linear Hydra grids (CMS and
Count-Sketch cells) and rejects HydraKll.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol
zzylol force-pushed the stack/509-x8a-hydra-kernel branch from 09fa09e to f97037a Compare October 5, 2026 06:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant