perf: group export metadata reads by dataset

[TonisOrmisson]

Summary

Phase 5: remove metadata N+1 reads from canonical and all three live legacy exporters (also used by migration backfill).

• Dataset-scoped grouped reads, local to each export; no cache, production class, dependency, configuration, schema changes or new user steps.
• Preserve ordered metadata/rows, missing-value diagnostics, empty sets, dataset isolation, same-instance freshness and database-read-only export. Existing conversions and public signatures retained.
• Canonical reads are bounded at 14 SQL executions; legacy SQLite/MySQL/PostgreSQL at 18, independently of variable/set counts. Adapter readiness remains unchanged.

Verification

• RED 3222e63: five genuine count failures as dictionaries grow. GREEN 5e00c74: production only, tests immutable.
• Actual Dolt CI exposed reserved SQL alias attribute; corrected to attr in 67534e8, without dialect branching. Latest push CI 34268697340: all 20 jobs successful across PHP 8.4/8.5 and nine server versions.
• composer check: 615 tests, 10,468 assertions, 112 service/environment skips; validation/lint/style/PHPStan pass. Focused exporter/adapter tests: 76 pass.
• Independent read-only full-diff review at latest head: no findings. Latest-head Codex review: no major issues. CodeRabbit: no actionable comments and minimal merge risk. PR CI: all 20 jobs successful; 41/41 GitHub checks successful, merge state CLEAN.
• CodeRabbit's generic docstring-percentage warning was assessed as non-actionable: grouped-row array shapes are already annotated, repository lint/style/PHPStan pass, and repetitive private-method docblocks would add no missing contract.

Measured evidence

Actual SQLite 3.45.1 / PHP 8.5.9, same installed engine 3.1.1, immutable base c4a697e and AFTER 67534e8. One warmup + five timed exports per workload/path/format; setup and read-back excluded from timing. Public adapter uses the real SAV/ZSAV writer, with query-only database access.

200 variables, 100 variable sets, 100 MR sets, two cases, labels/attributes/missing values:

Path	SQL executions before → after	Median ms before → after
Canonical reconstruction, SAV	809 → 14	23.165 → 10.088
SQLite legacy, SAV	1410 → 18	16.991 → 7.830
Full adapter SAV	820 → 25	51.888 → 45.518
Full adapter ZSAV	820 → 25	60.717 → 31.025

Counts are actual PDO SQL execution calls, not prepares or wire commands. 180/180 before/after typed output pairs matched, including metadata, ordering, diagnostics, binary64 bits and provenance; no fields excluded. All 360 captured traces contain SELECTs only. Small and existing mixed numeric/string fixtures also verified.

Limits: shared-host timing noise; tiny workloads showed some regressions (mixed-fixture full SAV median 1.449 → 2.078 ms). No universal speedup or timing SLA claimed. Server SQL correctness is covered by CI; SQLite execution of legacy server exporters is not server performance evidence. Full ranges, commands and immutable-source checksums are retained outside the product repository in php-export-metadata-evidence.md; no benchmark framework added.

Scope / next

No new malformed foreign-label ownership policy, cache, snapshot redesign, streaming, or unrelated cleanup. After user merge: Phase 6, Python dataset-filtered label reads and linear/reduced-repeat transformation planning. Overall plan remains unfinished; do not auto-merge.

[TonisOrmisson]

@codex review

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-08T19:40:25.888123Z	67534e8	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 58619646-a30e-4f22-a4a6-29351cff951c

📥 Commits

Reviewing files that changed from the base of the PR and between c4a697e and 67534e8.

📒 Files selected for processing (10)

• src/Sql/CanonicalWideTableExporter.php
• src/Sql/MySqlWideTableExporter.php
• src/Sql/PostgreSqlWideTableExporter.php
• src/Sql/SqliteWideTableExporter.php
• tests/Integration/MySqlFamilySpssRoundTripTestCase.php
• tests/Integration/PostgreSqlSpssRoundTripTest.php
• tests/Spss/SpssAdapterTest.php
• tests/Sql/MySqlWideTableExporterTest.php
• tests/Sql/PostgreSqlWideTableExporterTest.php
• tests/Support/ExportCountingPdo.php

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

The exporters now batch-load catalogue metadata and set members before building output metadata. Tests add SQL execution counters and verify bounded query counts, metadata freshness, read-only behavior, and malformed-catalogue diagnostics across exporter implementations.

Changes

Catalogue batching

Layer / File(s)	Summary
Canonical exporter batching src/Sql/CanonicalWideTableExporter.php	The canonical exporter groups labels, attributes, missing rules, and set members before variable and set processing.
Dialect exporter batching src/Sql/*WideTableExporter.php	SQLite, MySQL, and PostgreSQL exporters pass pre-fetched catalogue rows to metadata builders and batch-load set members.
SQL execution measurement tests/Support/ExportCountingPdo.php, tests/Integration/*	Test PDO wrappers capture SQL operations. Round-trip tests compare exporter output and enforce execution-count limits.
Grouped export validation tests/Spss/SpssAdapterTest.php, tests/Sql/*WideTableExporterTest.php	Tests cover query scaling, ordered metadata freshness, read-only exports, malformed metadata diagnostics, and grouped-row mocks.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Merge Risk: ⚪ Minimal · up to 67534

The batched exporter changes preserve tested metadata and output behavior while removing per-variable and per-set catalogue reads. No unresolved merge-blocking risk remains.

Sequence Diagram(s)

sequenceDiagram
  participant WideTableExporter
  participant PDO
  participant MetadataBuilders
  WideTableExporter->>PDO: Execute grouped catalogue queries
  PDO-->>WideTableExporter: Return grouped rows
  WideTableExporter->>MetadataBuilders: Pass labels, rules, roles, attributes, and members
  MetadataBuilders-->>WideTableExporter: Build export metadata

Loading

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 68.42% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 76 functions across 10 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: grouping export metadata reads by dataset to improve performance.

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch perf/bulk-export-metadata

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. 🎉

Reviewed commit: 67534e8af7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".