Repository navigation
Conversation
52b4003 to
8f58489
Compare
996294c to
764088d
Compare
…low multi-source summaries Review of #560/#646 found four problems: 1. Population names came from each node's schema, which `with_schema` may rename. A scan whose `tier` column is named "region" made `tier = 'eu'` read as `{region: eu}`, so a merge with a real `{region: us}` state was accepted and double-counted. Columns are now named from the Scan operator's own schema, and a path that renames a field leaves the population unknown. 2. For the same reason a merge could mix states of different columns (`Named("latency")` reading `size` on a renamed scan). An unknown population only merges with the same input, so this is rejected too. 3. `OperatorNode::map_children` dropped a SummaryAgg's coverage, so rebuilding a merge (e.g. in canonicalize) failed. A rebuild now keeps the declared time bounds and reads source and population again. 4. A SummaryAgg over two sources (a join, an IN subquery over another table) could never validate. It now carries no coverage and cannot be merged. Adds summary_coverage_derivation.rs; the four regression tests fail before this change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…btree A SummaryAgg's declared coverage is no longer trusted. The source must be the one scanned below it, and the population must equal the `field = 'text'` conjuncts read from SummaryAgg.filter, Filter nodes and Scan.predicates on a Filter/TimeRange-only path to the Scan. When the path holds anything else, the population is unknown and the declaration must be unrestricted; such states merge only as time panes of the same input and filter. Time bounds stay declared: a pane's bounds are a materialization fact. SummaryCoverage::for_summary builds a matching declaration for producers. Closes #570. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…low multi-source summaries Review of #560/#646 found four problems: 1. Population names came from each node's schema, which `with_schema` may rename. A scan whose `tier` column is named "region" made `tier = 'eu'` read as `{region: eu}`, so a merge with a real `{region: us}` state was accepted and double-counted. Columns are now named from the Scan operator's own schema, and a path that renames a field leaves the population unknown. 2. For the same reason a merge could mix states of different columns (`Named("latency")` reading `size` on a renamed scan). An unknown population only merges with the same input, so this is rejected too. 3. `OperatorNode::map_children` dropped a SummaryAgg's coverage, so rebuilding a merge (e.g. in canonicalize) failed. A rebuild now keeps the declared time bounds and reads source and population again. 4. A SummaryAgg over two sources (a join, an IN subquery over another table) could never validate. It now carries no coverage and cannot be merged. Adds summary_coverage_derivation.rs; the four regression tests fail before this change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| /// scanned below it and the population its filters restrict to, over | ||
| /// `time_ms`. `None` when `summary` is not a `SummaryAgg` or does not | ||
| /// read exactly one source. | ||
| pub fn for_summary(summary: &OperatorNode, time_ms: Option<Range<i64>>) -> Option<Self> { |
There was a problem hiding this comment.
The naming of time_ms seems not good. I assume you want conceptually a "range" here?
There was a problem hiding this comment.
yes, I will rename to "time_range"
| source: summary.scanned_source()?, | ||
| regions: vec![CoverageRegion { | ||
| time_ms, | ||
| population: summary.derived_population().unwrap_or_default(), |
There was a problem hiding this comment.
Bug here: If derived_population() returns None, it will be converted to the default legal value of population.
The problem is in derived_population() semantic, None means I cannot handle this population so it should not be used. While this treacherous unwrap_or_default() silently convert this into a legal value!
Spotting this bug actually makes me feel better: At least it proves reading code is still somehow useful.
| if summary.scanned_source().as_ref() != Some(&self.source) { | ||
| return Err(CoverageError::SourceMismatch); | ||
| } | ||
| let derived = summary.derived_population().unwrap_or_default(); |
There was a problem hiding this comment.
Same. Check if there is bug here
| } | ||
| if let (Some(coverage), Some(ASAPOp::SummaryAgg { .. })) = (&self.coverage, rebuilt.asap()) | ||
| { | ||
| let population = rebuilt.derived_population().unwrap_or_default(); |
There was a problem hiding this comment.
Seemingly a bug. Check other comments related
b727827 to
34c43f0
Compare
| /// scanned below it and the population its filters restrict to, over | ||
| /// `time_ms`. `None` when `summary` is not a `SummaryAgg` or does not | ||
| /// read exactly one source. | ||
| pub fn for_summary(summary: &OperatorNode, time_ms: Option<Range<i64>>) -> Option<Self> { |
There was a problem hiding this comment.
This function is also treacherous and shady. It is called for_summary but practically only works for SummaryAgg nodes. Some problems in PR #560 is related to this.
…sketches SummaryMerge no longer derives or checks coverage: of_merge, UnknownInput and MergeOutputMismatch are removed and summary_coverage.rs matches main. Coverage for all summary nodes will be derived by one function in #646. The heap-based sketch restriction is also dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SummaryCoverage now records which columns a state summarizes as well as which rows: `input` (the SummaryAgg update expression) and `group_by` (its reduction). with_coverage rejects a SummaryAgg declaration whose columns differ from the node's own (ColumnMismatch), and SummaryMerge requires coverage on every input (UnknownInput) with identical columns. merge_disjoint checks columns too. summary_input_data is removed. A nested SummaryMerge carries no coverage until #646, so it is rejected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SummaryCoverage records which columns a state summarizes (input, group_by) as well as which rows. Update the SummaryAgg and SummaryMerge examples to #560: merges compare coverage columns, carry no coverage until #646, and merged_coverage is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Closes #570.
Why
#567 records each summary state's coverage (source, time range, population), and #560 allows a
SummaryMergeonly when its inputs' coverage is provably disjoint. But aSummaryAgg's coverage is a declaration that nothing checks against the state's own subtree, so a wrong one merges silently.Before this PR:
After this PR:
A.with_coverage(...)fails withPopulationMismatch, because A's filters say{region: us}. Producers no longer write coverage by hand:SummaryCoverage::for_summary(&agg, time_ms)builds the declaration that passes.What is read from the subtree
sourceScanbelow theSummaryAgg(OperatorNode::scanned_source)populationfield = 'text'conjuncts ofSummaryAgg.filter,Filternodes andScan.predicates, on a path that passes only throughFilterandTimeRange(OperatorNode::derived_population)time_msScanoperator's own schema.FilterandTimeRangekeep their input columns, so every column id on the path is aScancolumn. A node on the path that renames a field (with_schema) makes the population unknown, because a name could then stand for a different source column.>, regex,IN,OR,region = 'us' AND region = 'eu'): the declaration must be unrestricted, andSummaryMergeaccepts such states only when every input has the same input and filter below itsSummaryAgg(time panes of one computation). Otherwise it fails withUnprovenPopulation. This also stops a merge of states that read different columns under one name.OperatorNode::map_childrenkeeps aSummaryAgg's declared time bounds and reads its source and population again from the new input, so rebuilding a merge (e.g. in canonicalize) keeps working.SummaryAggwhose input reads two sources (a join, or anINsubquery over another table) has no single source. It is valid without coverage and cannot be merged (UnknownInput).Key code interfaces
crates/types/src/ir/summary_coverage.rsTests
summary_coverage_examples.rsnow builds every #560 example with a realregion = '…'filter matching its declared population.summary_coverage_derivation.rsadds:population_combines_every_filter_on_the_pathSummaryAgg.filter, aFilternode andScan.predicates, through aTimeRange, give{job, region, tier}unsupported_predicates_leave_the_population_unknown>,OR, and one field equal to two valuesrenamed_field_does_not_fake_a_disjoint_populationtiercolumn cannot pass asregionrenamed_update_column_does_not_mergeNamed("latency")readingsizeon a renamed scan is rejectednested_merges_composerebuilding_keeps_coveragemap_childrenkeeps time bounds and re-reads population; a merge rebuildssummary_over_two_sources_has_no_coverageThe four regression tests fail on the code before the fix commit.
Out of scope
TimeRangeholds a relative duration, so absolute pane bounds cannot be checked in the logical IR.ProjectorJoin; widen the path when a real plan needs it.Stack and validation
Order: #645 (merged) → #567 (merged) → #560 → this PR → #539 → #540 → #541 → #542 → #543.
#542 builds planned coverage with
for_summary. This PR passes the workspace tests (1,645),cargo fmt --check, and workspace/all-target/all-feature Clippy with warnings denied. The fixes come from an independent review of #560 and #646.🤖 Generated with Claude Code