docs: propose workload-wide planning, summary sharing, and materialization by zzylol · Pull Request #509 · ProjectASAP/ASAPPlanner

zzylol · 2026-09-30T17:59:24Z

Summary

Propose ASAPPlanner's planning stages, from language-specific frontends through logical and physical optimization to selection of one physical plan for the whole workload. The deployment supplies empirical cost and accuracy models and execution capabilities, then executes the selected plan.

Add docs/design_docs/proposals/planner-layering.md and link it from the proposals index.

Before this PR

The planning boundary needs an explicit contract for workload-wide candidates, summary sharing, materialization, and plan selection.

After this PR

Frontends produce CandidateLogicalDAGs; logical optimization produces CandidateLogicalASAPDAGs; physical optimization produces CandidatePhysicalASAPDAGs; selection returns one PhysicalASAPDAG.
Logical optimization generates local exact and summary alternatives, then applies ASAP-aware common-subexpression elimination across computations and windows.
Physical optimization explores materialization, retention and operator implementations. Outputs may be maintained at ingestion time, retained at query time, or rebuilt without materialization.
Selection evaluates complete workload candidates using deployment cost and accuracy models, rejects candidates that fail requirements or capabilities, and chooses the cheapest remaining plan.

Four worked examples cover candidate growth (3 local, 54 logical and 156 physical alternatives), UnivMon sharing across three computations, KLL sharing across windows, and ingestion/query-time materialization choices.

Scope and validation

Documentation-only proposal; no runtime or API changes. Deployment-input types, summary subtract/delete design, and physical parallelism, partitioning and resource management remain TODO.

The branch's file contents have been restored to commit 8c29c2e376800ad0823d4917e121eed3b60ed3ab and verified identical with git diff. No runtime tests run.

Add the layering proposal, its proposals-index entry, and an Output layers section in the input/output/workflow design. Every Planner layer outputs all legal candidates; the deployment keeps every summary-family candidate and selects with its own costs. Design DAG names are marked as the target API with the current main type alongside. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Split physical planning into physical design (summary materialization, workload-level, like materialized-view selection) and physical implementation (per-node lowering and cutting). The annotated graph stays a Post-ASAP DAG; materialization is decided in design and realized by the cut. State node-by-node correspondence and the Fallback exception that the operator-flattening proposal removes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Describe current Planner behaviour (no family pruning) and the backend gap, and replace the Binary 'exception' with the general rule that a timing- sensitive node needs one compilation per distinct timing, with an example. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ng proposal Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Planner takes the query and data workloads plus the deployment's cost model, accuracy requirements and capabilities, and returns one optimal PhysicalDAG; candidate sets stay internal. Name the annotated stage MaterializedPostASAPDAG, tabulate what each DAG encodes, fold timing and selection into the layer descriptions, rename physical design to summary materialization, and make the example trace one query through every DAG. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…g row A lifecycle fixes materialization, timing, maintenance, retention and window framework together, so the stage and its DAG are named after the lifecycle (LifecyclePostASAPDAG) rather than one of those aspects. The deployment row no longer mentions ranking or selection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Describe the two plain candidate collections, the view-based compile, the caller-typed physical candidate errors, lifecycle_guarantee, and PhysicalExecution as an execution handle. Remove APIs the docs said were removed but never existed (compile_timed_candidates, PhysicalDAGCandidate), the agent instructions in the alignment proposal's baseline, and the ingestion-time Binary exception from design docs, where it is an implementation detail (the developer migration guide keeps it). Restore #485's statement that candidates do not choose placement and #508's CandidatePostASAPDAGs<Id> names in input-output-workflow.md, rejoin the split test table in physical-planning-and-deployment.md, and take planner-backend-layering.md verbatim from #509. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copies docs/design_docs/proposals/planner-backend-layering.md from #509's head 3607270 so this file merges cleanly after #509. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

The layering proposal (#509) names a single annotated DAG, LifecyclePostASAPDAG: a PostASAPDAG plus its lifecycle assignment. The code kept a SummaryMaintenanceLifecyclePlan beside the DAG and named the timed collection CandidatePostASAPDAGsWithTiming. - Rename SummaryMaintenanceLifecyclePlan to LifecyclePostASAPDAG and its error to LifecyclePostASAPDAGError. The type already held the root and each state's lifecycle, retention and window framework; per-node timing stays derived by execution_assignment as the existing PostASAPDAGAssignment overlay, so no timed graph is stored beside it and no new type is added. - Rename CandidatePostASAPDAGsWithTiming to CandidateLifecyclePostASAPDAGs; CandidatePostASAPDAGs stays the logical collection. - Call layer 2 "summary lifecycle planning" and reword comments that had the deployment choose or rank; selection is Planner's, over the deployment's cost model. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Several docs still said the deployment or "downstream" selects, ranks or chooses the plan, and the glossary said physical plans are owned by downstream systems. Per the layering proposal (#509), Planner compiles and selects the physical plan using the deployment's cost model; the deployment supplies prices and executes the selected plan. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

#509 carries the proposal; #480's copy now matches its head a095ce5 exactly, so merging either PR first leaves no conflict. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Every stage after the frontend is a Post-ASAP DAG; the prefix says how far planning has gone: Logical, Lifecycle, Physical. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…tive weights DataWorkload gains `metric_types` (metric name -> Counter | Gauge, the Prometheus TYPE metadata). plan_stages passes it to Stage 1, and Pass 1 attaches the new `CounterSamples` non-negativity proof to a keyed summary update weighted by the sample value when the target's input is samples of a declared counter, through a time range and plain sums (sum_over_time). Any other operator, a gauge or an undeclared metric (whatever its suffix) gets no proof. Stage 3 and the runtime already accept any NonNegative proof for Count-Min, so their rule is unchanged. Example 1 declares http_requests_total a counter: all 64 candidates are valid and compile (24 Count-Min + heap were invalid). Stage 3 now selects P60 (Count-Min + heap over the exact sum_over_time accumulator, shared input) at 46.201 against P58 (all exact) at 52.201, now 8th. Refs #509, #580. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…umers The summary-capability rule gains a frequency-moments key: a distinct count (one column), an L2 norm and an entropy over the same input and window share one UnivMon (#509 Example 2). UnivMon's shape does not depend on the requirement, so the three alternatives are the same state. The UnivMon accuracy model still certifies only the exact total. The published UnivMon bounds (Liu et al., SIGCOMM 2016, via Braverman and Ostrovsky) are asymptotic. The shipped kernel's heuristic recurrence, a fixed top-heap per layer with an L2/sqrt(heap) cut, does not meet their per-layer cover premise. Distinct count, L2 and entropy therefore stay uncertified. The reasoning is documented, and a test pins it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ntities Port of the parked runtime PRs for #509 Example 2 onto asap-executor: - native UnivMon build and Cardinality / FrequencyL2 / FrequencyEntropy readouts (the old stack's 14c8ac8, a prerequisite missing here); - #557: a UnivMon build accepts Utf8, Int64 and Bool identities as type-tagged keys; heap key bytes count toward state memory; - #559: exact FrequencyL2 / FrequencyEntropy reducers (bits, NULL skipped, 0 for an empty population, memory-accounted, cooperative); - #563: exact Cardinality reducer over typed tuples. The legacy raw-cost adapter lists the three intents as hash aggregates. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Port of the parked #562 and #564 for #509 Example 2. The SQL frontend names `SQRT(SUM(c*c))` over Float64 grouped unit counts as FrequencyL2, and `-SUM(p*LN(p))` with `p = COUNT(*)*1.0 / SUM(COUNT(*)) OVER ()` as FrequencyEntropy converted to nats. An exact population count guards SQL's empty-input NULL. Integer products, nullable keys, filters, HAVING, other log bases and partial windows are refused. The rules are the old Pass 1 SemanticEquivalentRewriteStrategy rules, run by lower_sql at the query root so the Stage 1 pipeline sees the intents; the exact candidate is the native exact reducer. The executor gains SQL sqrt (from #562). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Port of the parked #565: SQL LN and a complete, unordered `SUM(column) OVER ()` window (SQLWindowSum) run natively, so the original Q2 SQL of #509 Example 2 executes and matches its recognized entropy form. Partitioned, ordered and finite frames stay refused. planner_layering_example2 records Example 2's status: the three queries lower to Cardinality / FrequencyEntropy / FrequencyL2 (the design's integer Q3 does not), the time filter is a scan predicate, plan_stages plans the workload with UnivMon offered to each statistic, and the exact candidates execute (without the time filter: the runtime has no now() yet). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Load an asap-stage-pipeline/v1 document (file picker or ?doc=<path>) and show Logical, Logical ASAP and Physical ASAP lanes side by side, with candidate switchers, per-node timing and cost, a cost-ranked list of physical candidates, and the stage-3 selection and rejection reasons. Selecting a physical candidate shows its from_logical candidate in lane 2. Pre/Post-ASAP stays the default unless a stage document is loaded. stages.js holds the DOM-free validation, lane construction and ranking, tested headless against a hand-written #509 Example 1 Q2 fixture. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Contract v2 (Q28-Q30) gives each DAG one root per workload query and moves cost out of Stage 2 into stage3_selection.costs. - Read `roots` (still accepting a single `root`); mark every root and label it with its query id. - Take node badges, totals and ranking only from Stage 3 costs, labelled as a Stage 3 result; without Stage 3 the physical lane has no cost. - Load partial documents; missing later stages render "not produced". - Require every Stage 2 candidate to be selected or rejected, and show valid-but-costlier apart from invalid. - Show optional per-query requirements in the query list and on roots. - Rewrite the sample as #509 Example 1 with both queries, and label sort/limit physical operators. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

stage_pipeline now prices every plan and writes the cheapest --max-candidates (#613); the Stages view says "showing N of M plans" from the document's shown_of section. The README shows how to generate all six #509 examples into an ignored out/ directory. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Pre/post-ASAP DAGs no longer exist after #509/#572, so the viewer now reads only asap-stage-pipeline/v1 documents and uses the layout of the published Stage Viewer: example tabs with what each shows, the workload's queries and the deployment inputs on top, the Stage 3 plans ranked by cost beside three DAG lanes (Stage 0, the chosen Stage 1 candidate, the chosen Stage 2 candidate with costs), and node and edge details below. - index.html and app.js replace viewer.js; stages.js is unchanged and node-style.js keeps only the kind table the contract test reads, plus the three node groups. - examples.json lists the six #509 examples; server.py writes any missing document into out/ with stage_pipeline. - The query editor (editor.js) plans PromQL queries with optional ε/δ through /api/plan, which now runs stage_pipeline. - Removed: the Pre/Post-ASAP view, WorkloadDAG loading, render.py's standalone HTML, the dag_export-based sample and fixtures, screenshots. - test_viewer.py replaces test_render.py: the stage-document tests, the page and server checks, and app.js run against a stub DOM in V8. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Library API, architecture overview, planner pipeline and the accuracy and cost developer docs now describe the #509 stage pipeline. Docs that only describe removed APIs (legacy search, CostModel, explanation, physical-plan cost adapter) are marked historical. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

zzylol and others added 3 commits September 30, 2026 17:59

docs: treat source binding as a PhysicalDAG execution step

f1c13d4

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

zzylol mentioned this pull request Sep 30, 2026

docs: take node timing from lifecycle assignment in the flattening proposal #481

Closed

zzylol and others added 5 commits September 30, 2026 18:27

docs: point per-operator export at #481

6702d3a

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs: drop Binary lowering detail from the layering proposal

1e84c15

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs: drop mixed-placement and compatibility sections from the layeri…

cd6a18d

…ng proposal Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs: drop the implementation-status section from the layering proposal

aa0b959

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Selvomega self-requested a review September 30, 2026 18:36

docs: leave input-output-workflow unchanged in the layering PR

62ae3e7

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Selvomega requested changes Sep 30, 2026

View reviewed changes

Comment thread docs/design_docs/architecture/input-output-workflow.md Outdated

Comment thread docs/design_docs/architecture/input-output-workflow.md Outdated

Comment thread docs/design_docs/proposals/planner-backend-layering.md Outdated

docs: move selection into ASAPPlanner; deployment supplies prices

3607270

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Selvomega requested changes Sep 30, 2026

View reviewed changes

Comment thread docs/design_docs/proposals/planner-backend-layering.md Outdated

Comment thread docs/design_docs/proposals/planner-backend-layering.md Outdated

Comment thread docs/design_docs/proposals/planner-backend-layering.md Outdated

zzylol mentioned this pull request Sep 30, 2026

refactor!: align DAG APIs and share candidate graphs #480

Closed

zzylol requested a review from Selvomega September 30, 2026 20:37

Selvomega previously approved these changes Sep 30, 2026

View reviewed changes

Comment thread docs/design_docs/proposals/planner-layering.md Outdated

zzylol requested a review from milindsrivastava1997 September 30, 2026 20:51

docs: rename layering proposal to planner-layering.md

a3c6dbc

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

zzylol dismissed Selvomega’s stale review via a3c6dbc September 30, 2026 21:54

zzylol added a commit that referenced this pull request Sep 30, 2026

docs: follow #509's rename to planner-layering.md

67d01c9

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs: name the DAG stages LogicalPostASAPDAG and PhysicalPostASAPDAG

41076ca

Every stage after the frontend is a Post-ASAP DAG; the prefix says how far planning has gone: Logical, Lifecycle, Physical. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This was referenced Oct 5, 2026

Coverage lost with the legacy candidate_selection tests (follow-up of #580) #623

Open

Exact Sum over an Int64 column is priced by Stage 3 but does not bind in the executor #626

Open

test: select plans through plan_stages instead of global_selection #629

Draft

zzylol mentioned this pull request Oct 5, 2026

docs: ASAPPlanner layering MVP status report #642

Draft

2 tasks done

Navigation Menu

docs: propose workload-wide planning, summary sharing, and materialization - #509

docs: propose workload-wide planning, summary sharing, and materialization#509
zzylol merged 56 commits into
mainfrom
docs/planner-layering-contract

zzylol commented Sep 30, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

zzylol commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Before this PR

After this PR

Scope and validation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

zzylol commented Sep 30, 2026 •

edited

Loading