Read export data, fidelity and validation from one native snapshot

[TonisOrmisson]

Phase 3 — consistent, database-read-only export

Keeps existing public commands, signatures and result shapes. Uses native database snapshots, not dataset copies, application locks, new configuration or dependencies.

Changes

• One private read boundary starts before catalog verification and owns connection/transaction cleanup.
• SQLite uses ordinary BEGIN; PostgreSQL/MySQL/MariaDB use repeatable-read isolation; Dolt additionally starts its native transaction explicitly despite retained autocommit.
• SAV export reads values, dictionary and import fidelity diagnostics on the same snapshot, resolving fidelity by the dataset UUID.
• Validation reflects its physical columns on that same connection rather than another engine.
• Connections are closed and owned engines disposed before SAV writing/publication and on failures. Removing the second export read also removes repeated profile/catalog verification.

Validation

• RED 90a8350: deterministic SQLite WAL interleaving reproduces mixed descriptor, later-loss contamination, and validation schema race (3 failures).
• GREEN cbde051: all three pass; full local non-service suite 385 passed, 9 skipped, 53 deselected; focused checks and git diff --check pass.
• ca08100 corrects only the service test's native-state observation: PyMySQL caches an OK-packet status that SELECT EOF packets do not update. MySQL/MariaDB now record disabled autocommit honestly; the real concurrent-commit/old-and-fresh export semantics remain asserted. Reproduced the probe failure and pass on a disposable MySQL 8.4.9 instance.
• Independent full-diff local review at latest head: No findings.
• Latest-head push and PR CI both passed all 14 jobs: Python 3.11–3.14, package build/install, PostgreSQL 17.10/18.4, MySQL 8.4.11/9.7.2, MariaDB 11.4.12/11.8.8/12.3.2, and Dolt 2.2.2/2.2.3 interleaving. SELECT-only Dolt read/export tests also passed. No service claim is inferred from mocks.
• Existing Dolt propagation tests now observe the unchanged public read-profile gate rather than internal nested reader calls.

Boundary

Data/metadata consistency uses each engine's native snapshot guarantees. SQLite transactional schema rename is covered. This does not add writer/DDL coordination for non-MVCC server operations such as PostgreSQL TRUNCATE. Database transactions are not held during filesystem work.

Progress

Phase 2 #20 is merged. Next: phase 4 — bounded batched PHP server imports, in a separate PR.

Final review and handoff

• Full-diff local subagent review: No findings.
• @codex review completed for latest head ca08100, no findings and confirmed 👍.
• CodeRabbit completed: No actionable comments.
• All GitHub checks pass; merge state CLEAN. Three logical commits are pushed; worktree clean.

Phase 3 is ready for handoff. Do not merge automatically; the overall plan is not yet complete.

[TonisOrmisson]

@codex review

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-08T08:20:45.524233Z	ca08100	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: ee172ec2-3ce4-4fe3-8aa9-34fa1d24018e

📥 Commits

Reviewing files that changed from the base of the PR and between 790051c and ca08100.

📒 Files selected for processing (5)

• src/openstatspec/spss/sav.py
• src/openstatspec/sql/wide.py
• tests/test_dolt_conformance.py
• tests/test_read_only_export.py
• tests/test_sql_services.py

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

The SQL layer now provides verified, connection-scoped read snapshots. SAV export and validation use the same snapshot for related reads. Tests cover snapshot consistency, isolation, cleanup, failure handling, and concurrent writes.

Changes

Consistent SQL read snapshots

Layer / File(s)	Summary
Snapshot lifecycle and SQL readers src/openstatspec/sql/wide.py	Adds _read_snapshot with dialect-specific transaction handling. Dataset, fidelity-event, and validation reads use the snapshot connection.
SAV export snapshot integration src/openstatspec/spss/sav.py	SAV export reads dataset data, metadata, rows, and fidelity events within one snapshot.
SQLite snapshot and failure tests tests/test_read_only_export.py	Tests consistent reads, validation schema snapshots, read failures, WAL mode, transaction handling, and connection cleanup.
Cross-database export and conformance tests tests/test_dolt_conformance.py, tests/test_sql_services.py	Tests profile propagation, isolation, transaction state, concurrent writes, fidelity events, reader cleanup, and failed export preservation.

Estimated code review effort: 4 (Complex) | ~45 minutes

Merge Risk: ⚪ Minimal · up to ca081

The export and validation paths consistently use one database snapshot and release database resources before filesystem work. No actionable merge risk was identified.

Sequence Diagram(s)

sequenceDiagram
  participant export_sav
  participant _read_snapshot
  participant SQL readers
  participant Catalog database
  export_sav->>_read_snapshot: Open one read snapshot
  _read_snapshot->>Catalog database: Begin dialect-specific transaction
  export_sav->>SQL readers: Read dataset, rows, metadata, and fidelity events
  SQL readers->>Catalog database: Execute snapshot-scoped queries
  _read_snapshot-->>export_sav: Return consistent export data
  _read_snapshot->>Catalog database: Close connection and dispose engine

Loading

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 8.57% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 35 functions across 5 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: reading export data, fidelity information, and validation data from one native snapshot.

• Fix all pre-merge checks with AI

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/consistent-read-snapshot

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. 🎉

Reviewed commit: ca0810079f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".