Skip to content

feat(ir): read summary coverage source and population from the subtree - #646

Open
zzylol wants to merge 2 commits into
stack/528-02b-merge-structurefrom
stack/528-02c-coverage-derivation
Open

zzylol wants to merge 2 commits into
stack/528-02b-merge-structurefrom
stack/528-02c-coverage-derivation

Conversation

@zzylol

@zzylol zzylol commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Closes #570.

Why

#567 records each summary state's coverage (source, time range, population), and #560 allows a SummaryMerge only when its inputs' coverage is provably disjoint. But a SummaryAgg's coverage is a declaration that nothing checks against the state's own subtree, so a wrong one merges silently.

Before this PR:

A = SummaryAgg(filter: region = 'us')   declared {region: eu} × [0,1)   ← wrong, A holds US data
B = SummaryAgg(filter: region = 'us')   declared {region: us} × [0,1)

SummaryMerge(A, B) → accepted ("eu" ≠ "us" looks disjoint)
merged state       → every US observation in [0,1) counted twice

After this PR: A.with_coverage(...) fails with PopulationMismatch, because A's filters say {region: us}. Producers no longer write coverage by hand: SummaryCoverage::for_summary(&agg, time_ms) builds the declaration that passes.

What is read from the subtree

Part Where it comes from Checked?
source the one Scan below the SummaryAgg (OperatorNode::scanned_source) yes, must be equal
population field = 'text' conjuncts of SummaryAgg.filter, Filter nodes and Scan.predicates, on a path that passes only through Filter and TimeRange (OperatorNode::derived_population) yes, every region must equal it exactly
time_ms the producer: a pane's bounds are a materialization fact, not in the logical plan (#601 sets them) no, still declared
  • Exact equality, not a subset: declaring fewer conditions than the filters apply overstates the data, and declaring more lets a merge with another value pass.
  • Columns are named from the Scan operator's own schema. Filter and TimeRange keep their input columns, so every column id on the path is a Scan column. A node on the path that renames a field (with_schema) makes the population unknown, because a name could then stand for a different source column.
  • Unknown population (another operator on the path, a renamed field, or a predicate that is not one text equality per field: >, regex, IN, OR, region = 'us' AND region = 'eu'): the declaration must be unrestricted, and SummaryMerge accepts such states only when every input has the same input and filter below its SummaryAgg (time panes of one computation). Otherwise it fails with UnprovenPopulation. This also stops a merge of states that read different columns under one name.
  • Rebuilds keep coverage: OperatorNode::map_children keeps a SummaryAgg's declared time bounds and reads its source and population again from the new input, so rebuilding a merge (e.g. in canonicalize) keeps working.
  • Several sources: a SummaryAgg whose input reads two sources (a join, or an IN subquery over another table) has no single source. It is valid without coverage and cannot be merged (UnknownInput).

Key code interfaces

crates/types/src/ir/summary_coverage.rs

impl SummaryCoverage {
    /// Source and population from `summary`'s subtree, over `time_ms`.
    /// None unless `summary` is a SummaryAgg reading exactly one source.
    pub fn for_summary(summary: &OperatorNode, time_ms: Option<Range<i64>>) -> Option<Self>;
}

impl OperatorNode {
    pub fn scanned_source(&self) -> Option<Source>;
    /// None when the path holds another operator, renames a field, or a
    /// predicate is not `field = 'text'`. Empty map = no restriction.
    pub fn derived_population(&self) -> Option<BTreeMap<String, String>>;
}

pub enum CoverageError {
    // ... existing variants
    PopulationMismatch,   // declared population differs from the filters
    UnprovenPopulation,   // merge of unknown-population states with different inputs
}

Tests

summary_coverage_examples.rs now builds every #560 example with a real region = '…' filter matching its declared population. summary_coverage_derivation.rs adds:

Test Behaviour
population_combines_every_filter_on_the_path SummaryAgg.filter, a Filter node and Scan.predicates, through a TimeRange, give {job, region, tier}
unsupported_predicates_leave_the_population_unknown >, OR, and one field equal to two values
renamed_field_does_not_fake_a_disjoint_population regression: a renamed tier column cannot pass as region
renamed_update_column_does_not_merge regression: Named("latency") reading size on a renamed scan is rejected
nested_merges_compose merges of merges; unknown-population inputs stay rejected through nesting
rebuilding_keeps_coverage regression: map_children keeps time bounds and re-reads population; a merge rebuilds
summary_over_two_sources_has_no_coverage regression: a join of two sources validates without coverage and cannot merge

The four regression tests fail on the code before the fix commit.

Out of scope

  • Checking time bounds: TimeRange holds a relative duration, so absolute pane bounds cannot be checked in the logical IR.
  • Reading populations through Project or Join; widen the path when a real plan needs it.

Stack and validation

Order: #645 (merged) → #567 (merged) → #560 → this PR → #539 → #540 → #541 → #542 → #543.

#542 builds planned coverage with for_summary. This PR passes the workspace tests (1,645), cargo fmt --check, and workspace/all-target/all-feature Clippy with warnings denied. The fixes come from an independent review of #560 and #646.

🤖 Generated with Claude Code

@zzylol
zzylol force-pushed the stack/528-02b-merge-structure branch from 52b4003 to 8f58489 Compare October 6, 2026 20:45
@zzylol
zzylol force-pushed the stack/528-02c-coverage-derivation branch from 996294c to 764088d Compare October 6, 2026 20:48
@zzylol
zzylol marked this pull request as ready for review October 6, 2026 20:48
@zzylol
zzylol requested a review from Selvomega October 6, 2026 20:49
zzylol added a commit that referenced this pull request Oct 6, 2026
…low multi-source summaries

Review of #560/#646 found four problems:

1. Population names came from each node's schema, which `with_schema` may
   rename. A scan whose `tier` column is named "region" made `tier = 'eu'`
   read as `{region: eu}`, so a merge with a real `{region: us}` state was
   accepted and double-counted. Columns are now named from the Scan
   operator's own schema, and a path that renames a field leaves the
   population unknown.
2. For the same reason a merge could mix states of different columns
   (`Named("latency")` reading `size` on a renamed scan). An unknown
   population only merges with the same input, so this is rejected too.
3. `OperatorNode::map_children` dropped a SummaryAgg's coverage, so
   rebuilding a merge (e.g. in canonicalize) failed. A rebuild now keeps the
   declared time bounds and reads source and population again.
4. A SummaryAgg over two sources (a join, an IN subquery over another
   table) could never validate. It now carries no coverage and cannot be
   merged.

Adds summary_coverage_derivation.rs; the four regression tests fail
before this change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol zzylol changed the title feat(ir): check summary coverage source and population against the subtree feat(ir): read summary coverage source and population from the subtree Oct 6, 2026
zzylol and others added 2 commits October 6, 2026 21:37
…btree

A SummaryAgg's declared coverage is no longer trusted. The source must be
the one scanned below it, and the population must equal the `field = 'text'`
conjuncts read from SummaryAgg.filter, Filter nodes and Scan.predicates on a
Filter/TimeRange-only path to the Scan. When the path holds anything else,
the population is unknown and the declaration must be unrestricted; such
states merge only as time panes of the same input and filter. Time bounds
stay declared: a pane's bounds are a materialization fact.

SummaryCoverage::for_summary builds a matching declaration for producers.

Closes #570.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…low multi-source summaries

Review of #560/#646 found four problems:

1. Population names came from each node's schema, which `with_schema` may
   rename. A scan whose `tier` column is named "region" made `tier = 'eu'`
   read as `{region: eu}`, so a merge with a real `{region: us}` state was
   accepted and double-counted. Columns are now named from the Scan
   operator's own schema, and a path that renames a field leaves the
   population unknown.
2. For the same reason a merge could mix states of different columns
   (`Named("latency")` reading `size` on a renamed scan). An unknown
   population only merges with the same input, so this is rejected too.
3. `OperatorNode::map_children` dropped a SummaryAgg's coverage, so
   rebuilding a merge (e.g. in canonicalize) failed. A rebuild now keeps the
   declared time bounds and reads source and population again.
4. A SummaryAgg over two sources (a join, an IN subquery over another
   table) could never validate. It now carries no coverage and cannot be
   merged.

Adds summary_coverage_derivation.rs; the four regression tests fail
before this change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
/// scanned below it and the population its filters restrict to, over
/// `time_ms`. `None` when `summary` is not a `SummaryAgg` or does not
/// read exactly one source.
pub fn for_summary(summary: &OperatorNode, time_ms: Option<Range<i64>>) -> Option<Self> {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The naming of time_ms seems not good. I assume you want conceptually a "range" here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, I will rename to "time_range"

source: summary.scanned_source()?,
regions: vec![CoverageRegion {
time_ms,
population: summary.derived_population().unwrap_or_default(),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug here: If derived_population() returns None, it will be converted to the default legal value of population.
The problem is in derived_population() semantic, None means I cannot handle this population so it should not be used. While this treacherous unwrap_or_default() silently convert this into a legal value!

Spotting this bug actually makes me feel better: At least it proves reading code is still somehow useful.

if summary.scanned_source().as_ref() != Some(&self.source) {
return Err(CoverageError::SourceMismatch);
}
let derived = summary.derived_population().unwrap_or_default();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same. Check if there is bug here

}
if let (Some(coverage), Some(ASAPOp::SummaryAgg { .. })) = (&self.coverage, rebuilt.asap())
{
let population = rebuilt.derived_population().unwrap_or_default();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seemingly a bug. Check other comments related

/// scanned below it and the population its filters restrict to, over
/// `time_ms`. `None` when `summary` is not a `SummaryAgg` or does not
/// read exactly one source.
pub fn for_summary(summary: &OperatorNode, time_ms: Option<Range<i64>>) -> Option<Self> {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function is also treacherous and shady. It is called for_summary but practically only works for SummaryAgg nodes. Some problems in PR #560 is related to this.

zzylol added a commit that referenced this pull request Oct 7, 2026
…sketches

SummaryMerge no longer derives or checks coverage: of_merge,
UnknownInput and MergeOutputMismatch are removed and summary_coverage.rs
matches main. Coverage for all summary nodes will be derived by one
function in #646. The heap-based sketch restriction is also dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Oct 7, 2026
SummaryCoverage now records which columns a state summarizes as well as
which rows: `input` (the SummaryAgg update expression) and `group_by`
(its reduction). with_coverage rejects a SummaryAgg declaration whose
columns differ from the node's own (ColumnMismatch), and SummaryMerge
requires coverage on every input (UnknownInput) with identical columns.
merge_disjoint checks columns too. summary_input_data is removed. A
nested SummaryMerge carries no coverage until #646, so it is rejected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol added a commit that referenced this pull request Oct 7, 2026
SummaryCoverage records which columns a state summarizes (input,
group_by) as well as which rows. Update the SummaryAgg and SummaryMerge
examples to #560: merges compare coverage columns, carry no coverage
until #646, and merged_coverage is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants