perf: make transformation validation and hashing linear

[TonisOrmisson]

Summary

Phase 6a of the approved performance plan: reduce repeated transformation-plan validation and canonical serialization work without changing public behavior. Dataset-label query work remains a separate deferred Phase 6b.

• Validate a plan with insertion-ordered case-folded variable lookup and one last qualifying-create index instead of repeated full scans. Preserve sequential semantics, delete/recreate order, error precedence, and the special rule that RECODE-create does not permit deleting the last variable.
• Compute canonical plan bytes and SHA-256 once at each successful public apply boundary, reuse them for the audit JSON/hash/result, and preserve canonical/SPSS source provenance. No model fields, public signatures, cache, dependency, configuration, or persistent artifact changes.
• Leave SPSS prefix rebinding, parser/token position work, SQL statements/rows, and dataset-label reads unchanged; those are separate measured work.

RED/GREEN and checks

• RED 1cdcc18: five intentional failures on base: 10k EXECUTE suffix bookkeeping, 10k label case-fold lookup, generic object/mapping apply encoding/hash counts, and SPSS apply encoding/hash counts.
• GREEN ebe4637: production only; tests immutable after RED.
• Focused GREEN: 102 passed across frontend/in-place/conditional tests.
• Full non-service run: 393 passed, 9 skipped, 53 deselected in 198s using Python 3.13.2, pinned specification checkout and no pytest cache.
• Independent read-only full-diff review: No findings.
• PR CI: all 20 service/package jobs successful across Python 3.11–3.14 and artifact-install paths; 29/29 GitHub checks successful, merge state CLEAN. Selected service checks executed rather than being all skipped.
• Latest-head Codex review: no major issues. CodeRabbit: no actionable comments/minimal merge risk; no review threads.
• CodeRabbit's generic docstring-percentage warning was assessed as non-actionable: touched private functions retain focused type/doc comments where needed, repository has no docstring gate, and adding boilerplate would not improve the contract. No configuration or unrelated documentation added.

Measured evidence

Immutable base 66566529e86019b815f0bd3bae241b37930316df versus ebe4637a7faef5c3d9a3bab1f785695cd6b7c33a, same Python 3.13.2/SQLAlchemy/RFC8785 environment. Five fresh timed processes per revision/workload, alternating revision order; one warmup excluded.

Workload	Base median	After median	Ratio
10,000 variables + 10,000 EXECUTEs	7.361 s	0.009 s	804×
10,000 variables + 10,000 labels	23.766 s	0.075 s	318×
10,000 EXECUTE SPSS compile	9.798 s	1.505 s	6.5×
10,000-label generic apply	10.700 s	1.839 s	5.8×
10,000-label SPSS apply	28.611 s	11.512 s	2.5×

Exact counters: binder suffix entries 49,995,000→0; casefold on label workload 200,010,000→30,000; canonical plan encodings/hashes generic 4/3→1/1, SPSS 3/2→1/1. Actual SQL statements, fetched rows, affected rows, audit JSON, hashes, metadata, identity and provenance were byte/semantic-equivalent in isolated SQLite applies; no SQL reduction is claimed.

Full evidence, raw profiles, checksums and exact commands: ../python-query-planning-evidence.md and /tmp/python-query-planning-phase6a-zX3o6a (outside repository). Measurements include shared-host noise and a bounded 1k cProfile series; no timing assertions were added.

Known limits

• Frontend prefix-position processing and per-create prefix binding remain intentionally unchanged. Dataset-filtered label reads are deferred to Phase 6b.
• The conformance manifest case reject-descending-range already returns invalid_transformation_plan while the manifest expects invalid_numeric_range on both base and after; this PR does not touch that code and does not claim to fix it.

Next phase

After user merge: Phase 6b, separately characterize and optimize dataset-filtered label reads only if returned-row/query measurements justify it. Overall phases 0–9 remain unfinished. Do not auto-merge.

[TonisOrmisson]

@codex review

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-09T05:50:19.688037Z	ebe4637	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 19c1d04c-470d-4d98-a045-eea00d3be0b3

📥 Commits

Reviewing files that changed from the base of the PR and between 6656652 and ebe4637.

📒 Files selected for processing (4)

• src/openstatspec/sql/inplace_transform.py
• src/openstatspec/transform/validation.py
• tests/test_inplace_transform.py
• tests/test_transform_frontend.py

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

The change replaces list scans with casefolded dictionary lookups during plan binding. In-place application now computes canonical plan bytes and its SHA-256 hash once, then reuses both values for auditing and returned results. Tests cover performance, binding behavior, Unicode serialization, and hash call counts.

Changes

Transformation pipeline

Layer / File(s)	Summary
Case-insensitive variable binding src/openstatspec/transform/validation.py, tests/test_transform_frontend.py	Binding now stores variables in a casefolded dictionary. Create, delete, lookup, and predicate paths use dictionary operations. Delete validation precomputes the last create operation. Tests cover linear bookkeeping, error ordering, delete-and-recreate behavior, and Unicode canonicalization.
Canonical plan threading src/openstatspec/sql/inplace_transform.py, tests/test_inplace_transform.py	In-place application computes canonical plan bytes and the SHA-256 hash once. The values pass through submission, audit insertion, and the returned result. Tests verify canonical JSON, hashes, and single invocation counts.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to ebe46

This change makes transformation binding and plan hashing linear while preserving transformation semantics and audit outputs. Current coverage supports merge readiness with no actionable risk identified.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 11.54% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 26 functions across 4 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the primary changes: linear transformation validation and single-pass plan hashing.

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch perf/query-planning

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Another round soon, please!

Reviewed commit: ebe4637a7f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".