perf: batch server case imports within native limits

[TonisOrmisson]

Summary

Phase 4 of the approved integrity/performance plan: replace per-row server case INSERTs with bounded prepared multirow batches. No public API, dependencies, configuration, catalogue/schema batching, or user steps change.

• One internal sender shared by PostgreSQL and MySQL/MariaDB/Dolt importers: 256-row / 1 MiB soft targets, 65,535 parameters including ordinals, active MySQL-family packet budget. A preflight-valid oversized singleton is still supported.
• Generate encoded rows incrementally, buffer only a bounded batch, reuse only the last prepared shape. Preserve ordered ordinals, NULL/empty/escaped UTF-8 strings, and existing binary64 encoding.
• Leave preflight, error-mode restoration, transaction ownership, finalization, and rollback/compensation unchanged. MySQL-family DDL still uses existing compensating cleanup, not transactional DDL atomicity.
• Multiple-row MySQL imports add one packet-budget query before DDL, not per batch; zero/one-row imports add none. Two fitting rows save one execute but total MySQL SQL execution API count stays unchanged.

Tests and reviews

• RED commit 04f2530: six targeted failures on the base (single-row executions and missing second-batch prepare/full-batch rollback behavior); broader focused run had 24 intended failures and five singleton-preservation passes.
• GREEN commit 6a753ce: production only; tests unchanged after RED.
• composer check: 600 tests, 9,824 assertions, 112 service/environment skips; validation, lint, style, PHPStan pass.
• Public importer tests cover empty/one/two rows, full/tail batches, maximum profile width/parameter count, soft byte and packet limits, preflight-boundary singleton, exact reconstructed values, second-batch prepare/execute failures and scoped rollback.
• Existing live CI suites extended for native/emulated prepares, ordered binary64/text/NULL values, wide parameter-limited imports, late server CHECK failure after a successful batch, preservation of prior datasets, and PostgreSQL multi-batch finalization failure. No new workflow/configuration.
• Dedicated local PostgreSQL/MySQL: eight native/emulated wide/late-failure cases passed. Separate MySQL max_allowed_packet=262144: six escaped-string cases passed; observed [6,6,1] batches for 10,000-byte text, singleton batches for valid 65,000/65,520-byte text.
• Independent read-only full-diff local review: no findings.
• Push CI 34220366922 and PR CI 34221832624: all 20 jobs each successful, PHP 8.4/8.5 across both PostgreSQL/MySQL/Dolt and all three MariaDB versions. Actual service test assertions executed, not all-skipped suites.
• Latest-head Codex review: completed, no major issues. CodeRabbit: completed, no actionable comments. No review threads; 41/41 GitHub checks successful, merge state CLEAN.
• CodeRabbit's generic docstring-percentage warning was assessed as non-actionable: necessary internal row-shape annotations and packet arithmetic comments are present, repository style/static checks pass, and redundant PHPUnit method docblocks would add boilerplate without documenting a missing contract. No configuration change or generated documentation added.

Measured complete-import performance

Same fixed 10,000-row uncompressed SAV, actual SpssAdapter::import/installed engine, four mixed numeric/text columns, same isolated servers/runtime/dependencies, ready empty catalogue. One warmup + five measured imports per workload/revision; nearby baseline repeat and two AFTER runs. Every ordered scalar checked using NULL tags, exact text and binary64 bits.

Server / PDO mode	Nearby baseline median	AFTER medians (two runs)	Speedup
PostgreSQL / native	7,021 ms	460 / 532 ms	13.2–15.3×
PostgreSQL / emulated	6,244 ms	508 / 450 ms	12.3–13.9×
MySQL / native	6,410 ms	492 / 487 ms	13.0–13.2×
MySQL / emulated	6,736 ms	477 / 500 ms	13.5–14.1×

Case executes: 10,000 → 40. Total SQL execution PDO API calls: PostgreSQL 10,132 → 172, MySQL 10,101 → 142 (excludes two unchanged transaction calls; not wire-command counts). Prepare API calls increase by one for the tail shape. Actual versions: PHP 8.5.9, MySQL 8.4.9, PostgreSQL 17.11; local pgsql extension 8.5.10/libpq 16.15 held identical across revisions.

Limits: dedicated loopback containers use tmpfs and a shared host, so this is not a production SLA. Tiny 1/2-row follow-ups showed no reproducible regression but do not prove formal equivalence. SAV byte checksum: b2e921455d9d6bc129148e89731602d35d6f7d6ec70a654968b5a572e7f65d37. Benchmark harness/results remain outside the product repository; no persistent benchmark framework or dataset snapshots added. Existing installed writer compression=1 precision observation recorded separately, outside this batching change; fixed uncompressed fixtures verified exactly.

Next phase

After user merge: Phase 5, grouped PHP export metadata reads to remove N+1 queries without a new cache/layer. The overall plan is not complete. Do not auto-merge.

[TonisOrmisson]

@codex review

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-08T11:42:54.982374Z	6a753ce	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: e2648b93-dc47-4b34-ab11-8acbadfbc421

📥 Commits

Reviewing files that changed from the base of the PR and between 46cbdae and 6a753ce.

📒 Files selected for processing (7)

• src/Sql/MySqlWideTableImporter.php
• src/Sql/PostgreSqlWideTableImporter.php
• src/Sql/PreparedCaseBatch.php
• tests/Integration/MySqlFamilySpssRoundTripTestCase.php
• tests/Integration/PostgreSqlSpssRoundTripTest.php
• tests/Sql/MySqlWideTableImporterTest.php
• tests/Sql/PostgreSqlWideTableImporterTest.php

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

The importers now use PreparedCaseBatch to insert validated case rows in bounded batches. Unit and integration tests cover row, parameter, byte, packet, prepared-statement, rollback, and round-trip behavior.

Changes

Bounded case-row import

Layer / File(s)	Summary
Shared batching and importer integration src/Sql/PreparedCaseBatch.php, src/Sql/MySqlWideTableImporter.php, src/Sql/PostgreSqlWideTableImporter.php	The importers generate ordered case rows and pass them to PreparedCaseBatch, which limits batches by parameter count and payload size.
MySQL batch behavior and rollback tests tests/Sql/MySqlWideTableImporterTest.php	Tests cover MySQL batch boundaries, packet and byte limits, statement probes, and failures during preparation or execution of a later batch.
PostgreSQL batch behavior and rollback tests tests/Sql/PostgreSqlWideTableImporterTest.php	Tests cover empty and bounded imports, ordered values, oversized rows, later-batch failures, rollback, and PDO error-mode restoration.
Server round-trip validation tests/Integration/MySqlFamilySpssRoundTripTestCase.php, tests/Integration/PostgreSqlSpssRoundTripTest.php	Integration tests cover native and emulated prepares, wide and escaped data, late failures, rollback, and preservation of existing datasets.

Estimated code review effort: 4 (Complex) | ~45 minutes

Merge Risk: ⚪ Minimal · up to 6a753

The bounded import implementation has no substantiated merge-blocking issue at the current head.

Sequence Diagram(s)

sequenceDiagram
  participant Importer
  participant PreparedCaseBatch
  participant PDO
  Importer->>PreparedCaseBatch: provide validated caseRows
  PreparedCaseBatch->>PDO: prepare bounded INSERT statement
  PreparedCaseBatch->>PDO: execute batched parameters
  PDO-->>Importer: return success or batch failure

Loading

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 30 functions across 7 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: batching server case imports within database limits.

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch perf/batched-server-import

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. 👍

Reviewed commit: 6a753ce995

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".