Skip to content

devtools: stage_pipeline examples planner-layering-2 and 4b - #613

Draft
zzylol wants to merge 2 commits into
stack/509-q49-example4-invariantsfrom
stack/509-demo-examples
Draft

zzylol wants to merge 2 commits into
stack/509-q49-example4-invariantsfrom
stack/509-demo-examples

Conversation

@zzylol

@zzylol zzylol commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Rebased on main d4869a7 (DF 54).

Problem

The DAG viewer's stage_pipeline built in #509 Examples 1, 3a, 3b and 4a only. It could not show Example 2, which is SQL, because the tool lowered only PromQL and labeled every query "promql". It also could not show the Q49 crossover from #612, where B1 wins. That example needs a deployment that does not keep raw data, but the tool always used asap_executor::capabilities() unchanged.

Before this PR: --example planner-layering-2 and --example planner-layering-4b fail with "unknown example".
After this PR: both write valid asap-stage-pipeline/v1 documents. 4b selects P4-m1 "Q1 Kll · tumbling 10m panes · ingestion time: Kll ×6 panes", and its deployment.capabilities.raw_data_retained is false.

Changes

  • stage_pipeline lowers SQL workloads with asap_frontend_sql::lower_sql_batch over Example 2's flows catalog, using a current-thread tokio runtime. PromQL workloads go through asap_frontend_promql as before. workload.queries[].language is now "sql" or "promql".
  • planner-layering-2: Q1 is COUNT DISTINCT (ε=0.02), Q2 is entropy (ε=0.05) and Q3_FLOAT is L2 (ε=0.01), all with δ=0.01 as one-time batch entries. The data is ContinuouslyIngesting at 100,000/s with cardinality 10M, as in planner_layering_example2.rs.
  • planner-layering-4b: Pattern B's quantile_over_time(0.99, latency_ms[60m]) every 10 min, over 1,000 series sampled every 1 s. It runs on DeploymentCapabilities { raw_data_retained: false, ..asap_executor::capabilities() }. main picks the capabilities for the example and passes them to stage_pipeline.
  • The header comment and usage string cover both new examples.

Example 2 selects the exact plan, P86 "Q1 exact · Q2 exact (Count acc) · Q3 exact (Count acc) · shared summary". The built-in models have no UnivMon accuracy model ("no accuracy model for UnivMon"), so every UnivMon candidate is rejected. The test records this outcome but the design does not require it.

  • Stage 3 prices every candidate (up to 4096 combinations). --max-candidates now only limits how many plans the document carries: the cheapest first, then the invalid ones. Before this, the selection was made over the first N combinations only, so at 128 of 486, Examples 3a and 4a selected P122 (60,853 and 84.5 cost/s) instead of the cheapest plan, P365 (39,128 and 54.3). When not every plan is written, a shown_of section gives the totals. candidate_cap_is_recorded now checks that the selection with --max-candidates 5 matches the uncapped run.

Test plan

  • After the cap fix (e3d77b00): fmt and clippy are clean; cargo test --workspace 1,668 passed, 0 failed; at the default cap, 3a and 4a select P365 and Example 2 selects P86.

  • New example2_plans_the_sql_workload: the document validates, every query's language is sql, and the selected label is recorded.

  • New example4b_selects_ingestion_time_panes_without_raw_data: the document validates, raw_data_retained is false, and the selected label contains "ingestion time" and "tumbling 10m panes".

  • cargo fmt --all --check, cargo clippy --workspace --all-targets -- -D warnings, cargo test -p asap-devtools (45 passed) and cargo test --workspace (1668 passed, 0 failed, 21 ignored) all pass.

🤖 Generated with Claude Code

zzylol added a commit that referenced this pull request Oct 4, 2026
stage_pipeline now prices every plan and writes the cheapest
--max-candidates (#613); the Stages view says "showing N of M plans" from
the document's shown_of section. The README shows how to generate all six
#509 examples into an ignored out/ directory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
zzylol and others added 2 commits October 5, 2026 04:48
Example 2 is the SQL workload over `flows` (distinct, entropy, L2),
lowered with the SQL frontend; workload queries report their language.
Example 4b is the Q49 crossover: Pattern B's hourly p99 every 10 min over
1,000 series sampled every second, on a deployment that does not keep raw
data, where the ingestion-time tumbling KLL (B1) is selected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e plans written

stage_pipeline selected among the first --max-candidates combinations, so
a cap below the combination count could select a costlier plan (Examples
3a and 4a selected P122 at 128 of 486; the cheapest is P365). Stage 3 now
prices up to 4096 combinations, and the document carries the cheapest
--max-candidates plans with their logical candidates, then invalid ones,
and a shown_of section with the totals when not every plan is written.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol
zzylol force-pushed the stack/509-q49-example4-invariants branch from d28b006 to a23ab1a Compare October 5, 2026 06:21
@zzylol
zzylol force-pushed the stack/509-demo-examples branch from e3d77b0 to 727cf89 Compare October 5, 2026 06:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant