diff --git a/CHANGELOG.md b/CHANGELOG.md index 9f8492a..83a4d0b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,18 @@ All notable changes to this reference implementation are documented here. ## Unreleased +### Added + +- Implement explicitly selected official SPSS Frontend 0.3 over unchanged + Plan 0.1/0.2 through the shared parser/binder, with a strict + `compile_spss_request` boundary and opt-in live apply provenance. Support + comments, dictionary-order `TO`, grouped commands, finite open RECODE ranges, + additive ordered typed labels, inequality aliases, and canonical NOT lowering. + Precedence is comparisons → NOT → AND → OR; parentheses override it. + Tests cover all 90 effective normative cases plus request boundaries and + in-place SQLite identity/metadata/audit/no-artifact checks. Default APIs and + Python extension support/rejections remain unchanged. + ### Fixed - Namespace new Python schema-change plans and SPSS output as @@ -12,9 +24,9 @@ All notable changes to this reference implementation are documented here. The former `openstatspec-transformation-plan-v0.3` remains accepted as a legacy Python extension with unchanged semantics, canonical JSON, and hashes; stored audits are not migrated. New compilation changes schema-plan hashes. -- Clarify that specification `v0.5.0` has no official Plan 0.3 and that its - syntax-only Frontend 0.3 over Plan 0.1/0.2 is not implemented here. Legacy - acceptance and Python extension tests do not claim official conformance. +- Clarify that specification `v0.5.0` has no official Plan 0.3. Its syntax-only + Frontend 0.3 over Plan 0.1/0.2 is separate from Python schema extensions. + Legacy acceptance and Python extension tests do not claim official conformance. See [compatibility and migration](docs/transformations.md#contract-ownership-and-legacy-compatibility). ## 0.8.1 - 2026-09-10 diff --git a/docs/transformations.md b/docs/transformations.md index 448c5be..fe22288 100644 --- a/docs/transformations.md +++ b/docs/transformations.md @@ -34,7 +34,7 @@ they are never treated as arbitrary SQL. Published OpenStatSpec `v0.5.0` defines Transformation Plan 0.1/0.2, not Plan 0.3. Its optional SPSS Frontend 0.3 is a syntax-only expansion emitting -Plan 0.1/0.2; this adapter does **not** implement that frontend yet. +Plan 0.1/0.2, implemented through the explicit official request boundary below. Explicit `create_variable` / `delete_variable` operations (including SPSS `STRING` / `DELETE VARIABLES`) belong to a **Python extension**, not official @@ -46,7 +46,7 @@ OpenStatSpec conformance. New schema-changing compilations emit: The exported names `TRANSFORMATION_PLAN_SCHEMA_CHANGE_CONTRACT` and `SPSS_FRONTEND_SCHEMA_CHANGE_CONTRACT` remain unchanged; their values now use these Python-owned identifiers. Plans without these explicit schema operations -retain their existing Plan 0.1/0.2 selection and frontend identifier. Official +retain their existing Plan 0.1/0.2 selection and default frontend identifier. Official Plan 0.1/0.2 still reject explicit create/delete operations. The loader and executor continue to accept the old Python plan identifier @@ -64,6 +64,78 @@ when its operations are identical. An intentional change of a saved plan's contract likewise creates a new artifact with a new hash, not an audit migration. Older adapter versions cannot load the new IDs. +## Official Frontend 0.3 + +Use `openstatspec.compile_spss_request(request)` for the official JSON request +boundary. It accepts **only** `openstatspec-spss-syntax-frontend-v0.3`, with +exact required fields `contract`, `input_alias`, `input_schema`, and +`source_text`. Unknown fields, missing fields, wrong types, noncanonical or +nonfinite typed codes, ambiguous variable names, and mismatched label types +fail closed. Request-shape failures report `invalid_spss_request`; source +failures retain the normative diagnostics. It returns no partial compilation. + +```python +compilation = openstatspec.compile_spss_request({ + "contract": "openstatspec-spss-syntax-frontend-v0.3", + "input_alias": "parent", + "input_schema": {"variables": [ + {"name": "age", "storage_kind": "numeric"}, + ]}, + "source_text": "COMMENT finite ages. RECODE age (LOWEST THRU 17 = 0).", +}) +``` + +The ordered request dictionary accepts only `name`, `storage_kind`, +`variable_label`, ordered typed `value_labels`, `format_family`, `width`, +`decimals`, and `measurement_level`. In particular, `width`/`decimals` are +request field names, not `format_width`/`format_decimals`. Descriptive input +format metadata is preserved, including partial metadata and string formats; +`FORMATS` operations still accept only bounded numeric F formats. Physical +identifiers and Python `declared_string_width` are not request fields. + +The same parser/binder implements command-boundary star and `COMMENT` +comments, non-nested block comments, dictionary-order inclusive `TO` ranges, +grouped recodes/labels/formats/levels, finite `LOWEST`/`HIGHEST` RECODE bounds, +`NE`/`<>`/`~=`, `NOT`, and `ADD VALUE LABELS`. `TO` cannot generate `INTO` +targets. Additive labels update an existing typed code at its ordinal and +append new codes in source order, using the preceding label state. Numeric +zero has positive-zero identity; string codes retain exact contents. +Comments remain in the LF-normalized source hash but emit no operation; +comment-only input is invalid. Python `STRING` and `DELETE VARIABLES` are +rejected under the official selector. + +**Explicit implementation decision:** precedence is standard SPSS order, +comparisons → `NOT` → `AND` → `OR` (tightest first). Thus `NOT a = 1 AND +b = 1` means `(NOT (a = 1)) AND b = 1`, not negation of the conjunction. +Parentheses override precedence. Comparison complements and De Morgan lowering +preserve SQL UNKNOWN, and maximal same-operator nodes flatten in source order. +This decision has adapter regression tests; normative fixtures are unchanged. + +Programs using only Plan 0.1 operations retain their exact Plan 0.1 object; +any Plan 0.2-only operation selects Plan 0.2. Both source and complete canonical +plan hashes remain independent. Official compilations record the 0.3 frontend +identifier regardless of output-plan version. + +For typed-schema or live-database callers, explicitly pass +`frontend_contract="openstatspec-spss-syntax-frontend-v0.3"` to +`compile_spss_syntax` or `apply_spss_in_place`. The latter compiles against the +live schema within the existing transaction and records official frontend and +source/plan provenance. JSON callers should use `compile_spss_request` rather +than building a typed schema from untrusted fields. Omitting the selector +retains all old support, rejection behavior, and Python extension selection; +the existing CLI remains on that compatibility path. + +Evidence: `tests/test_frontend_v03.py` runs the 35 declared and all 90 effective +cases from specification commit +`864e84479f554b8ee250ffed44c4dfb963750d4a`, applying only the published inherited +contract overrides and two comment supersessions. It checks exact Plan 0.1/0.2 +objects/hashes, source hashes, diagnostics, and declared output metadata. +Local SQLite integration checks UNKNOWN, sequential data and metadata, +identity, audit provenance, and absence of copy/history artifacts. Existing +frontend, plan, and in-place suites remain in the gate. This is not new service +execution evidence: MySQL/MariaDB/Dolt provisioning restrictions and caller-owned +Dolt commits are unchanged. + ## Install the audit schema Install the compact audit relation once before the first apply: diff --git a/src/openstatspec/__init__.py b/src/openstatspec/__init__.py index eb73a94..54be92c 100644 --- a/src/openstatspec/__init__.py +++ b/src/openstatspec/__init__.py @@ -10,7 +10,7 @@ remove_derived_physical_relation, retire_derived, validate, validate_derived, ) from .core import CapabilityDeclaration, LossReport, UnsupportedOperationError -from .frontends.spss import SpssFrontendCompilation, compile_spss_syntax +from .frontends.spss import SpssFrontendCompilation, compile_spss_syntax, compile_spss_request from .sql import DoltConformanceSource from .sql.workflow import TransformationError from .transform import ( @@ -37,7 +37,7 @@ "TransformationPlan", "TypedValue", "ValueLabel", "VariableDefinition", "VariableSchema", "transformation_plan_from_dict", "apply_spss_in_place", "apply_transformation_plan_in_place", - "compile_spss_syntax", "install_in_place_transformation_schema", + "compile_spss_syntax", "compile_spss_request", "install_in_place_transformation_schema", "UnsupportedOperationError", "capabilities", "capability_matrix", "derive_sql_dataset", "dolt_state_snapshot", "execute_sql_transformation", "export_sav", "get_dataset", "import_sav", "initialize_catalog", "inspect", "list_datasets", diff --git a/src/openstatspec/api.py b/src/openstatspec/api.py index ff54612..2ceff4d 100644 --- a/src/openstatspec/api.py +++ b/src/openstatspec/api.py @@ -226,7 +226,8 @@ def derive_sql_dataset(*, database_url: Any, **options: Any) -> Mapping[str, Any def apply_spss_in_place( *, database_url: Any, dataset_id: str, source_text: str, - actor: str, expected_branch: str | None = None, + actor: str, frontend_contract: str | None = None, + expected_branch: str | None = None, expected_head: str | None = None, dolt_conformance_source: DoltConformanceSource | None = None, ) -> Mapping[str, Any]: @@ -239,6 +240,7 @@ def apply_spss_in_place( dataset_id=dataset_id, source_text=source_text, actor=actor, + frontend_contract=frontend_contract, expected_branch=expected_branch, expected_head=expected_head, dolt_conformance_source=dolt_conformance_source, diff --git a/src/openstatspec/frontends/spss/__init__.py b/src/openstatspec/frontends/spss/__init__.py index 58ed87e..7975b6d 100644 --- a/src/openstatspec/frontends/spss/__init__.py +++ b/src/openstatspec/frontends/spss/__init__.py @@ -3,9 +3,11 @@ from .binding import bind_spss_syntax from .compiler import ( SPSS_FRONTEND_CONTRACT, + SPSS_FRONTEND_V03_CONTRACT, SPSS_FRONTEND_SCHEMA_CHANGE_CONTRACT, SpssFrontendCompilation, compile_spss_syntax, + compile_spss_request, ) from .syntax import ( SpssSyntaxProgram, @@ -18,11 +20,13 @@ __all__ = [ "SPSS_FRONTEND_CONTRACT", + "SPSS_FRONTEND_V03_CONTRACT", "SPSS_FRONTEND_SCHEMA_CHANGE_CONTRACT", "SpssFrontendCompilation", "SpssSyntaxProgram", "bind_spss_syntax", "compile_spss_syntax", + "compile_spss_request", "normalize_spss_source", "parse_spss_syntax", "spss_source_hash", diff --git a/src/openstatspec/frontends/spss/binding.py b/src/openstatspec/frontends/spss/binding.py index 2913115..1183f7b 100644 --- a/src/openstatspec/frontends/spss/binding.py +++ b/src/openstatspec/frontends/spss/binding.py @@ -24,7 +24,7 @@ from .syntax import ( BooleanSyntax, ComparisonSyntax, ComputeCommandSyntax, ExecuteCommandSyntax, DeleteVariablesCommandSyntax, FormatsCommandSyntax, IfCommandSyntax, - OperandSyntax, PredicateSyntax, + NotSyntax, OperandSyntax, PredicateSyntax, Token, VariableRangeSyntax, RecodeCommandSyntax, RecodeMatchSyntax, RecodeResultSyntax, SpssSyntaxProgram, SyntaxLiteral, ValueLabelsCommandSyntax, StringCommandSyntax, VariableLabelsCommandSyntax, VariableLevelCommandSyntax, @@ -74,9 +74,16 @@ def _bind_operand( def _bind_predicate( - syntax: PredicateSyntax, variables: list[VariableDefinition], + syntax: PredicateSyntax, variables: list[VariableDefinition], *, negated: bool = False, ) -> PredicateExpression: + if isinstance(syntax, NotSyntax): + return _bind_predicate(syntax.operand, variables, negated=not negated) if isinstance(syntax, ComparisonSyntax): + if syntax.operator == "ne" or (negated and syntax.operator == "="): + expanded = BooleanSyntax("or", tuple( + replace(syntax, operator=operator) for operator in ("<", ">") + ), syntax.span) + return _bind_predicate(expanded, variables, negated=negated and syntax.operator == "ne") left, left_type = _bind_operand(syntax.left, variables) right, right_type = _bind_operand(syntax.right, variables) if left_type != right_type: @@ -91,19 +98,21 @@ def _bind_predicate( "Ordered comparisons require numeric operands.", span=syntax.span, operator=syntax.operator, ) - return ComparisonExpression(left, syntax.operator, right) + operator = ({"=": "=", "<": ">=", "<=": ">", ">": "<=", ">=": "<"}[syntax.operator] if negated else syntax.operator) + return ComparisonExpression(left, operator, right) assert isinstance(syntax, BooleanSyntax) + operator = ("or" if syntax.operator == "and" else "and") if negated else syntax.operator operands: list[PredicateExpression] = [] for operand in syntax.operands: - bound = _bind_predicate(operand, variables) + bound = _bind_predicate(operand, variables, negated=negated) if ( isinstance(bound, BooleanExpression) - and bound.operator == syntax.operator + and bound.operator == operator ): operands.extend(bound.operands) else: operands.append(bound) - return BooleanExpression(syntax.operator, tuple(operands)) + return BooleanExpression(operator, tuple(operands)) def _assignment( @@ -151,7 +160,7 @@ def _assignment( def _match( - syntax: RecodeMatchSyntax, source: VariableDefinition, + syntax: RecodeMatchSyntax, source: VariableDefinition, *, official_v03: bool = False, ) -> RecodeMatch: expected = _expected_type(source.storage_kind) if syntax.kind == "system_missing": @@ -172,7 +181,7 @@ def _match( ) if lower.number() > upper.number(): raise frontend_error( - "invalid_numeric_range", + "invalid_variable_range" if official_v03 else "invalid_numeric_range", "THRU lower endpoint exceeds its upper endpoint.", span=syntax.span, variable=source.name, ) @@ -252,10 +261,12 @@ def _validate_recode_string_width( def _bind_recode( - command: RecodeCommandSyntax, variables: list[VariableDefinition], + command: RecodeCommandSyntax, variables: list[VariableDefinition], *, official_v03: bool = False, ) -> tuple[list[RecodeOperation], list[SourceSpan]]: sources = [_resolve(variables, token.text, token.span)[1] for token in command.sources] targets = command.targets + if targets is not None and len(targets) != len(sources): + raise frontend_error("spss_syntax_error", "RECODE INTO requires one target per source.", span=command.span) target_mode: Literal["create", "replace"] = "create" if targets is not None else "replace" target_names = ( [token.text for token in targets] if targets is not None @@ -292,7 +303,7 @@ def _bind_recode( if clause.match.kind == "else": else_result = result continue - rules.append(RecodeRule(_match(clause.match, source), result)) + rules.append(RecodeRule(_match(clause.match, source, official_v03=official_v03), result)) unmatched = else_result or RecodeResult( "system_missing" if target_mode == "create" else "copy" ) @@ -332,6 +343,28 @@ def _bind_recode( # replace intentionally preserves the existing variable metadata. A later # VALUE LABELS command replaces value labels explicitly. return operations, spans + + +def _expand_variables( + tokens: tuple[Token | VariableRangeSyntax, ...], variables: list[VariableDefinition], +) -> tuple[Token, ...]: + expanded: list[Token] = [] + for token in tokens: + if isinstance(token, VariableRangeSyntax): + first, _ = _resolve(variables, token.first.text, token.first.span) + last = first + for endpoint in (token.last, *token.continuations): + next_index, _ = _resolve(variables, endpoint.text, endpoint.span) + if last > next_index: + raise frontend_error("invalid_variable_range", "TO endpoints are reversed in dictionary order.", span=token.span) + last = next_index + expanded.extend(Token("identifier", v.name, v.name, token.span) for v in variables[first:last + 1]) + else: + _resolve(variables, token.text, token.span) + expanded.append(token) + return tuple(expanded) + + def bind_spss_syntax( program: SpssSyntaxProgram, schema: VariableSchema, *, input_alias: str = "parent", ) -> BoundTransformation: @@ -345,6 +378,22 @@ def bind_spss_syntax( operations: list[PlanOperation] = [] spans: list[SourceSpan] = [] for command in program.commands: + if program.official_v03: + if isinstance(command, (StringCommandSyntax, DeleteVariablesCommandSyntax)): + raise frontend_error("unsupported_spss_command", "Python schema commands are outside official Frontend 0.3.", span=command.span) + if isinstance(command, RecodeCommandSyntax): + command = replace(command, sources=_expand_variables(command.sources, variables)) + elif isinstance(command, (FormatsCommandSyntax, VariableLevelCommandSyntax, VariableLabelsCommandSyntax)): + command = replace(command, assignments=tuple( + replace(assignment, variable=token) + for assignment in command.assignments + for token in _expand_variables((assignment.variable,), variables) + )) + elif isinstance(command, ValueLabelsCommandSyntax): + command = replace(command, groups=tuple( + replace(group, variables=_expand_variables(group.variables, variables)) + for group in command.groups + )) if isinstance(command, StringCommandSyntax): for variable_token in command.variables: if variable_token.text.startswith("__"): @@ -383,7 +432,7 @@ def bind_spss_syntax( del variables[index] continue if isinstance(command, RecodeCommandSyntax): - recodes, recode_spans = _bind_recode(command, variables) + recodes, recode_spans = _bind_recode(command, variables, official_v03=program.official_v03) operations.extend(recodes) spans.extend(recode_spans) continue @@ -515,6 +564,10 @@ def bind_spss_syntax( "VALUE LABELS contains duplicate canonical codes.", span=group.span, variable=variable.name, ) + if command.additive: + merged = {label.value.canonical_key(): label for label in variable.value_labels} + merged.update((label.value.canonical_key(), label) for label in labels) + labels = tuple(merged.values()) operation = ReplaceValueLabelsOperation(variable.name, labels) operations.append(operation) spans.append(group.span) diff --git a/src/openstatspec/frontends/spss/compiler.py b/src/openstatspec/frontends/spss/compiler.py index a2fb8e2..6897089 100644 --- a/src/openstatspec/frontends/spss/compiler.py +++ b/src/openstatspec/frontends/spss/compiler.py @@ -2,13 +2,16 @@ from __future__ import annotations +from collections.abc import Mapping from dataclasses import dataclass +from typing import Any +from ...transform.errors import TransformationFrontendError, frontend_error from ...transform.plan import ( TRANSFORMATION_PLAN_SCHEMA_CHANGE_CONTRACT, - TransformationPlan, + TransformationPlan, TypedValue, ValueLabel, ) -from ...transform.schema import BoundTransformation, VariableSchema +from ...transform.schema import BoundTransformation, VariableDefinition, VariableSchema from .binding import bind_spss_syntax from .syntax import ( normalize_spss_source, @@ -18,6 +21,7 @@ SPSS_FRONTEND_CONTRACT = "openstatspec-spss-syntax-frontend-v0.2" +SPSS_FRONTEND_V03_CONTRACT = "openstatspec-spss-syntax-frontend-v0.3" # Python-owned schema extension, not the official syntax-only Frontend 0.3. SPSS_FRONTEND_SCHEMA_CHANGE_CONTRACT = "openstatspec-python-schema-change-spss-v0.1" @@ -29,6 +33,7 @@ class SpssFrontendCompilation: source_text_lf: str source_hash: str bound: BoundTransformation + selected_contract: str | None = None @property def plan(self) -> TransformationPlan: @@ -40,6 +45,8 @@ def plan_hash(self) -> str: @property def frontend_contract(self) -> str: + if self.selected_contract is not None: + return self.selected_contract if self.plan.contract == TRANSFORMATION_PLAN_SCHEMA_CHANGE_CONTRACT: return SPSS_FRONTEND_SCHEMA_CHANGE_CONTRACT return SPSS_FRONTEND_CONTRACT @@ -50,16 +57,81 @@ def compile_spss_syntax( schema: VariableSchema, *, input_alias: str = "parent", + frontend_contract: str | None = None, ) -> SpssFrontendCompilation: """Parse and bind source without SQL generation or database mutation.""" + if frontend_contract not in (None, SPSS_FRONTEND_V03_CONTRACT): + raise frontend_error("invalid_spss_request", "Select the exact official Frontend 0.3 contract or omit the selector for legacy behavior.") normalized = normalize_spss_source(source) - bound = bind_spss_syntax( - parse_spss_syntax(normalized), - schema, - input_alias=input_alias, - ) + program = parse_spss_syntax(normalized, official_v03=frontend_contract is not None) + try: + bound = bind_spss_syntax(program, schema, input_alias=input_alias) + except TransformationFrontendError as error: + if frontend_contract is not None and error.span is None: + error.span = program.span + raise return SpssFrontendCompilation( source_text_lf=normalized, source_hash=spss_source_hash(normalized), bound=bound, + selected_contract=frontend_contract, ) + + +def compile_spss_request(request: Mapping[str, Any]) -> SpssFrontendCompilation: + """Validate the exact official Frontend 0.3 request, then use the shared frontend.""" + def invalid(): + raise frontend_error("invalid_spss_request", "Request must conform to the official Frontend 0.3 schema.") + + if not isinstance(request, Mapping) or set(request) != {"contract", "input_alias", "input_schema", "source_text"}: + invalid() + if request["contract"] != SPSS_FRONTEND_V03_CONTRACT: + invalid() + if any(not isinstance(request[key], str) or not request[key] for key in ("input_alias", "source_text")): + invalid() + raw_schema = request["input_schema"] + if not isinstance(raw_schema, Mapping) or set(raw_schema) != {"variables"}: + invalid() + raw_variables = raw_schema["variables"] + if not isinstance(raw_variables, list) or not raw_variables: + invalid() + variables = [] + allowed = {"name", "storage_kind", "variable_label", "value_labels", "format_family", "width", "decimals", "measurement_level"} + try: + for raw in raw_variables: + if not isinstance(raw, Mapping) or not {"name", "storage_kind"} <= set(raw) <= allowed: + invalid() + if not isinstance(raw["name"], str) or not raw["name"] or raw["storage_kind"] not in ("numeric", "string"): + invalid() + for key in ("variable_label", "format_family"): + if raw.get(key) is not None and not isinstance(raw[key], str): + invalid() + for key, minimum in (("width", 1), ("decimals", 0)): + value = raw.get(key) + if value is not None and (isinstance(value, bool) or not isinstance(value, (int, float)) or value < minimum or value != int(value)): + invalid() + if raw.get("measurement_level") not in (None, "nominal", "ordinal", "scale"): + invalid() + raw_labels = raw.get("value_labels", []) + if not isinstance(raw_labels, list): + invalid() + labels = [] + for label in raw_labels: + if not isinstance(label, Mapping) or set(label) != {"value", "label"} or not isinstance(label["label"], str): + invalid() + labels.append(ValueLabel(TypedValue.from_dict(label["value"]), label["label"])) + keys = [label.value.canonical_key() for label in labels] + if len(keys) != len(set(keys)): + invalid() + variables.append(VariableDefinition( + raw["name"], raw["storage_kind"], + variable_label=raw.get("variable_label"), value_labels=tuple(labels), + format_family=raw.get("format_family"), + format_width=int(raw["width"]) if raw.get("width") is not None else None, + format_decimals=int(raw["decimals"]) if raw.get("decimals") is not None else None, + measurement_level=raw.get("measurement_level"), _validate_format=False, + )) + schema = VariableSchema(tuple(variables)) + except (ValueError, TypeError, OverflowError) as error: + raise frontend_error("invalid_spss_request", "Invalid Frontend 0.3 input schema.") from error + return compile_spss_syntax(request["source_text"], schema, input_alias=request["input_alias"], frontend_contract=SPSS_FRONTEND_V03_CONTRACT) diff --git a/src/openstatspec/frontends/spss/execution.py b/src/openstatspec/frontends/spss/execution.py index 108bed0..4ea3367 100644 --- a/src/openstatspec/frontends/spss/execution.py +++ b/src/openstatspec/frontends/spss/execution.py @@ -19,6 +19,7 @@ def apply_spss_in_place( dataset_id: str, source_text: str, actor: str, + frontend_contract: str | None = None, expected_branch: str | None = None, expected_head: str | None = None, dolt_conformance_source: DoltConformanceSource | None = None, @@ -26,11 +27,12 @@ def apply_spss_in_place( """Compile SPSS syntax and apply its canonical plan in one transaction.""" def prepare(connection: Any, live_dataset_id: str) -> InPlacePlanSubmission: - schema = load_transformation_schema(connection, live_dataset_id) + schema = load_transformation_schema(connection, live_dataset_id, lock_dataset=True) compilation = compile_spss_syntax( source_text, schema, input_alias="parent", + frontend_contract=frontend_contract, ) return InPlacePlanSubmission( plan=compilation.plan, diff --git a/src/openstatspec/frontends/spss/syntax.py b/src/openstatspec/frontends/spss/syntax.py index 5aadd44..febb499 100644 --- a/src/openstatspec/frontends/spss/syntax.py +++ b/src/openstatspec/frontends/spss/syntax.py @@ -14,7 +14,7 @@ TokenKind = Literal[ "identifier", "number", "string", "left_paren", "right_paren", "equals", "less", "less_equal", "greater", "greater_equal", - "comma", "plus", "minus", "slash", "period", "eof", + "comma", "plus", "minus", "slash", "period", "eof", "not_equal", ] @@ -26,6 +26,14 @@ class Token: span: SourceSpan +@dataclass(frozen=True) +class VariableRangeSyntax: + first: Token + last: Token + span: SourceSpan + continuations: tuple[Token, ...] = () + + @dataclass(frozen=True) class SyntaxLiteral: kind: Literal["numeric", "string"] @@ -58,7 +66,7 @@ class RecodeClauseSyntax: @dataclass(frozen=True) class RecodeCommandSyntax: - sources: tuple[Token, ...] + sources: tuple[Token | VariableRangeSyntax, ...] clauses: tuple[RecodeClauseSyntax, ...] targets: tuple[Token, ...] | None span: SourceSpan @@ -74,7 +82,7 @@ class OperandSyntax: @dataclass(frozen=True) class ComparisonSyntax: left: OperandSyntax - operator: Literal["=", "<", "<=", ">", ">="] + operator: Literal["=", "<", "<=", ">", ">=", "ne"] right: OperandSyntax span: SourceSpan @@ -86,7 +94,13 @@ class BooleanSyntax: span: SourceSpan -PredicateSyntax = ComparisonSyntax | BooleanSyntax +@dataclass(frozen=True) +class NotSyntax: + operand: "PredicateSyntax" + span: SourceSpan + + +PredicateSyntax = ComparisonSyntax | BooleanSyntax | NotSyntax @dataclass(frozen=True) @@ -106,7 +120,7 @@ class IfCommandSyntax: @dataclass(frozen=True) class FormatAssignmentSyntax: - variable: Token + variable: Token | VariableRangeSyntax family: str width: int decimals: int @@ -121,7 +135,7 @@ class FormatsCommandSyntax: @dataclass(frozen=True) class VariableLevelAssignmentSyntax: - variable: Token + variable: Token | VariableRangeSyntax level: Literal["nominal", "ordinal", "scale"] span: SourceSpan @@ -153,7 +167,7 @@ class DeleteVariablesCommandSyntax: @dataclass(frozen=True) class VariableLabelSyntax: - variable: Token + variable: Token | VariableRangeSyntax label: Token span: SourceSpan @@ -173,7 +187,7 @@ class ValueLabelSyntax: @dataclass(frozen=True) class ValueLabelsGroupSyntax: - variables: tuple[Token, ...] + variables: tuple[Token | VariableRangeSyntax, ...] labels: tuple[ValueLabelSyntax, ...] span: SourceSpan @@ -182,6 +196,7 @@ class ValueLabelsGroupSyntax: class ValueLabelsCommandSyntax: groups: tuple[ValueLabelsGroupSyntax, ...] span: SourceSpan + additive: bool = False SyntaxCommand = ( @@ -196,6 +211,7 @@ class ValueLabelsCommandSyntax: class SpssSyntaxProgram: commands: tuple[SyntaxCommand, ...] span: SourceSpan + official_v03: bool = False _NUMBER = re.compile( @@ -234,7 +250,7 @@ def spss_source_hash(source: str) -> str: return hashlib.sha256(normalized.encode("utf-8")).hexdigest() -def tokenize_spss(source: str) -> tuple[Token, ...]: +def tokenize_spss(source: str, *, official_v03: bool = False) -> tuple[Token, ...]: """Tokenize supported SPSS text without consulting a dataset catalog.""" if not isinstance(source, str): raise TypeError("source must be text") @@ -250,6 +266,20 @@ def tokenize_spss(source: str) -> tuple[Token, ...]: if character.isspace(): offset += 1 continue + if official_v03 and source.startswith("/*", offset): + end = source.find("*/", offset + 2) + nested = source.find("/*", offset + 2) + if end < 0 or (nested >= 0 and nested < end): + raise frontend_error("spss_syntax_error", "Unterminated or nested block comment.", span=_span(source, offset, len(source))) + offset = end + 2 + continue + boundary = not tokens or tokens[-1].kind == "period" + if official_v03 and boundary and character == "*": + end = source.find(".", offset) + if end < 0: + raise frontend_error("spss_syntax_error", "Expected '.' after comment.", span=_span(source, offset, len(source))) + offset = end + 1 + continue if character in {"'", '"'}: start = offset quote = character @@ -315,6 +345,12 @@ def tokenize_spss(source: str) -> tuple[Token, ...]: continue break text = source[start:offset] + if official_v03 and boundary and text.isascii() and text.casefold() == "comment": + end = source.find(".", offset) + if end < 0: + raise frontend_error("spss_syntax_error", "Expected '.' after comment.", span=_span(source, start, len(source))) + offset = end + 1 + continue if text.casefold() in {"nan", "infinity"}: raise frontend_error( "spss_syntax_error", @@ -325,6 +361,11 @@ def tokenize_spss(source: str) -> tuple[Token, ...]: "identifier", text, text, _span(source, start, offset), )) continue + if official_v03 and source[offset:offset + 2] in {"<>", "~="}: + text = source[offset:offset + 2] + tokens.append(Token("not_equal", text, text, _span(source, offset, offset + 2))) + offset += 2 + continue if source.startswith("<=", offset) or source.startswith(">=", offset): text = source[offset:offset + 2] tokens.append(Token( @@ -350,9 +391,10 @@ def tokenize_spss(source: str) -> tuple[Token, ...]: class _Parser: - def __init__(self, source: str) -> None: + def __init__(self, source: str, *, official_v03: bool = False) -> None: self.source = source - self.tokens = tokenize_spss(source) + self.official_v03 = official_v03 + self.tokens = tokenize_spss(source, official_v03=official_v03) self.index = 0 @property @@ -372,7 +414,8 @@ def accepts(self, kind: TokenKind) -> Token | None: def accepts_keyword(self, keyword: str) -> Token | None: token = self.current - if token.kind == "identifier" and token.text.casefold() == keyword.casefold(): + if (token.kind == "identifier" and token.text.casefold() == keyword.casefold() + and (not self.official_v03 or token.text.isascii())): return self.advance() return None @@ -391,12 +434,21 @@ def expects_keyword(self, keyword: str) -> Token: ) return token - def variable_list(self, *, stop_kinds: frozenset[str]) -> tuple[Token, ...]: - variables: list[Token] = [] + def variable_list(self, *, stop_kinds: frozenset[str]) -> tuple[Token | VariableRangeSyntax, ...]: + variables: list[Token | VariableRangeSyntax] = [] while self.current.kind not in stop_kinds: - if self.accepts("comma") is not None: + if not self.official_v03 and self.accepts("comma") is not None: continue - variables.append(self.expects("identifier", "Expected a variable name.")) + first = self.expects("identifier", "Expected a variable name.") + if self.official_v03 and self.accepts_keyword("TO") is not None: + last = self.expects("identifier", "Expected a TO endpoint.") + continuations = [] + while self.accepts_keyword("TO") is not None: + continuations.append(self.expects("identifier", "Expected a TO endpoint.")) + end = continuations[-1] if continuations else last + variables.append(VariableRangeSyntax(first, last, _joined_span(first.span, end.span), tuple(continuations))) + else: + variables.append(first) if not variables: raise frontend_error( "spss_syntax_error", "Expected at least one variable name.", @@ -437,9 +489,13 @@ def comparison(self) -> PredicateSyntax: "greater": ">", "greater_equal": ">=", } token = self.current + if self.official_v03: + operators["not_equal"] = "ne" + if token.kind == "identifier" and token.text.isascii() and token.text.casefold() == "ne": + operators["identifier"] = "ne" if token.kind not in operators: raise frontend_error( - "spss_syntax_error", "Expected a comparison operator.", + "expression_type_unsupported" if self.official_v03 and token.kind == "right_paren" else "spss_syntax_error", "Expected a comparison operator.", span=token.span, ) self.advance() @@ -448,10 +504,16 @@ def comparison(self) -> PredicateSyntax: left, operators[token.kind], right, _joined_span(left.span, right.span), ) + def negation(self) -> PredicateSyntax: + if self.official_v03 and (token := self.accepts_keyword("NOT")) is not None: + operand = self.negation() + return NotSyntax(operand, _joined_span(token.span, operand.span)) + return self.comparison() + def conjunction(self) -> PredicateSyntax: - operands = [self.comparison()] + operands = [self.negation()] while self.accepts_keyword("AND") is not None: - operands.append(self.comparison()) + operands.append(self.negation()) if len(operands) == 1: return operands[0] return BooleanSyntax( @@ -500,8 +562,13 @@ def if_command(self, start: Token) -> IfCommandSyntax: def formats(self, start: Token) -> FormatsCommandSyntax: assignments: list[FormatAssignmentSyntax] = [] while self.current.kind not in {"period", "eof"}: - self.accepts("slash") - variable = self.expects("identifier", "FORMATS requires a variable name.") + if assignments or not self.official_v03: + self.accepts("slash") + variables = ( + self.variable_list(stop_kinds=frozenset({"left_paren", "period", "eof", "slash"})) + if self.official_v03 else + (self.expects("identifier", "FORMATS requires a variable name."),) + ) self.expects("left_paren", "Expected '(' before an SPSS format.") format_token = self.expects("identifier", "Expected an SPSS format such as F1.0.") match = re.fullmatch(r"([A-Za-z]+)([0-9]+)(?:[.]([0-9]+))?", format_token.text) @@ -513,7 +580,7 @@ def formats(self, start: Token) -> FormatsCommandSyntax: if (family != "F" or width < 1 or width > 40 or decimals > 16 or (decimals != 0 and width < decimals + 2)): raise frontend_error("invalid_format", "Only valid numeric F formats are supported.", span=format_token.span, format=format_token.text) - assignments.append(FormatAssignmentSyntax(variable, family, width, decimals, _joined_span(variable.span, right.span))) + assignments.extend(FormatAssignmentSyntax(variable, family, width, decimals, _joined_span(variable.span, right.span)) for variable in variables) if not assignments: raise frontend_error("spss_syntax_error", "FORMATS requires an assignment.", span=self.current.span) end = self.expects("period", "Expected '.' after FORMATS.") @@ -523,14 +590,15 @@ def variable_level(self, start: Token) -> VariableLevelCommandSyntax: self.expects_keyword("LEVEL") assignments: list[VariableLevelAssignmentSyntax] = [] while self.current.kind not in {"period", "eof"}: - self.accepts("slash") + if assignments or not self.official_v03: + self.accepts("slash") variables = self.variable_list( stop_kinds=frozenset({"left_paren", "period", "eof", "slash"}), ) self.expects("left_paren", "Expected '(' before a measurement level.") level = self.expects("identifier", "Expected NOMINAL, ORDINAL, or SCALE.") normalized = level.text.casefold() - if normalized not in {"nominal", "ordinal", "scale"}: + if normalized not in {"nominal", "ordinal", "scale"} or (self.official_v03 and not level.text.isascii()): raise frontend_error( "spss_syntax_error", "Expected NOMINAL, ORDINAL, or SCALE.", @@ -631,19 +699,26 @@ def recode_clause(self) -> RecodeClauseSyntax: elif (token := self.accepts_keyword("SYSMIS")) is not None: match = RecodeMatchSyntax("system_missing", token.span) else: - first = self.literal() + lowest = self.accepts_keyword("LOWEST") if self.official_v03 else None + first = SyntaxLiteral("numeric", -float.fromhex("0x1.fffffffffffffp+1023"), lowest.span) if lowest else self.literal() if self.accepts_keyword("THRU") is not None: - upper = self.literal() + highest = self.accepts_keyword("HIGHEST") if self.official_v03 else None + upper = SyntaxLiteral("numeric", float.fromhex("0x1.fffffffffffffp+1023"), highest.span) if highest else self.literal() match = RecodeMatchSyntax( "range", _joined_span(first.span, upper.span), lower=first, upper=upper, ) else: + if lowest: + raise frontend_error("spss_syntax_error", "LOWEST requires THRU.", span=first.span) values = [first] while self.current.kind != "equals": - self.accepts("comma") - if self.current.kind == "equals": - break + if self.official_v03: + self.expects("comma", "Expected ',' between RECODE selectors.") + else: + self.accepts("comma") + if self.current.kind == "equals": + break values.append(self.literal()) match = RecodeMatchSyntax( "values", _joined_span(values[0].span, values[-1].span), @@ -681,14 +756,16 @@ def recode(self, start: Token) -> RecodeCommandSyntax: ) targets = None if self.accepts_keyword("INTO") is not None: - targets = self.variable_list(stop_kinds=frozenset({"period", "eof"})) - if len(targets) != len(sources): + targets = self.variable_list(stop_kinds=frozenset({"period", "eof", "slash"}) if self.official_v03 else frozenset({"period", "eof"})) + if self.official_v03 and any(isinstance(target, VariableRangeSyntax) or target.text.casefold() == "to" for target in targets): + raise frontend_error("spss_syntax_error", "INTO targets cannot use TO.", span=targets[0].span) + if not self.official_v03 and len(targets) != len(sources): raise frontend_error( "spss_syntax_error", "RECODE INTO requires one target for every source variable.", span=_joined_span(targets[0].span, targets[-1].span), ) - end = self.expects("period", "Expected '.' after RECODE.") + end = self.current if self.official_v03 and self.current.kind in {"slash", "period"} else self.expects("period", "Expected '.' after RECODE.") return RecodeCommandSyntax( sources, tuple(clauses), targets, _joined_span(start.span, end.span), ) @@ -697,12 +774,17 @@ def variable_labels(self, start: Token) -> VariableLabelsCommandSyntax: self.expects_keyword("LABELS") assignments: list[VariableLabelSyntax] = [] while self.current.kind not in {"period", "eof"}: - self.accepts("slash") - variable = self.expects("identifier", "Expected a variable name.") + if assignments or not self.official_v03: + self.accepts("slash") + variables = ( + self.variable_list(stop_kinds=frozenset({"string", "period", "eof", "slash"})) + if self.official_v03 else + (self.expects("identifier", "Expected a variable name."),) + ) label = self.expects("string", "Expected a quoted variable label.") - assignments.append(VariableLabelSyntax( + assignments.extend(VariableLabelSyntax( variable, label, _joined_span(variable.span, label.span), - )) + ) for variable in variables) if not assignments: raise frontend_error( "spss_syntax_error", "VARIABLE LABELS requires an assignment.", @@ -713,11 +795,12 @@ def variable_labels(self, start: Token) -> VariableLabelsCommandSyntax: tuple(assignments), _joined_span(start.span, end.span), ) - def value_labels(self, start: Token) -> ValueLabelsCommandSyntax: + def value_labels(self, start: Token, *, additive: bool = False) -> ValueLabelsCommandSyntax: self.expects_keyword("LABELS") groups: list[ValueLabelsGroupSyntax] = [] while self.current.kind not in {"period", "eof"}: - self.accepts("slash") + if groups or not self.official_v03: + self.accepts("slash") group_start = self.current variables = self.variable_list( stop_kinds=frozenset({"number", "string", "period", "slash", "eof"}) @@ -738,18 +821,26 @@ def value_labels(self, start: Token) -> ValueLabelsCommandSyntax: variables, tuple(labels), _joined_span(group_start.span, labels[-1].span), )) + if self.official_v03 and not groups: + raise frontend_error("spss_syntax_error", "VALUE LABELS requires a group.", span=self.current.span) end = self.expects("period", "Expected '.' after VALUE LABELS.") return ValueLabelsCommandSyntax( - tuple(groups), _joined_span(start.span, end.span), + tuple(groups), _joined_span(start.span, end.span), additive, ) def parse(self) -> SpssSyntaxProgram: commands: list[SyntaxCommand] = [] while self.current.kind != "eof": start = self.expects("identifier", "Expected an SPSS command.") - command = start.text.casefold() + command = start.text.casefold() if not self.official_v03 or start.text.isascii() else start.text + if self.official_v03 and command in {"string", "delete"}: + raise frontend_error("unsupported_spss_command", "Python schema commands are outside official Frontend 0.3.", span=start.span, command=start.text) if command == "recode": commands.append(self.recode(start)) + while self.official_v03 and self.accepts("slash") is not None: + commands.append(self.recode(self.current)) + if self.official_v03: + self.expects("period", "Expected '.' after RECODE.") elif command == "compute": commands.append(self.compute(start)) elif command == "if": @@ -769,6 +860,9 @@ def parse(self) -> SpssSyntaxProgram: commands.append(self.variable_labels(start)) elif command == "value": commands.append(self.value_labels(start)) + elif self.official_v03 and command == "add": + self.expects_keyword("VALUE") + commands.append(self.value_labels(start, additive=True)) else: raise frontend_error( "unsupported_spss_command", @@ -779,18 +873,18 @@ def parse(self) -> SpssSyntaxProgram: program_span = _joined_span(commands[0].span, commands[-1].span) else: program_span = self.current.span - return SpssSyntaxProgram(tuple(commands), program_span) + return SpssSyntaxProgram(tuple(commands), program_span, self.official_v03) -def parse_spss_syntax(source: str) -> SpssSyntaxProgram: +def parse_spss_syntax(source: str, *, official_v03: bool = False) -> SpssSyntaxProgram: """Parse the supported command subset into a catalog-independent AST.""" normalized = normalize_spss_source(source) comment = re.search(r"(?m)^[ \t]*\*", normalized) - if comment is not None: + if comment is not None and not official_v03: raise frontend_error( "unsupported_spss_command", "SPSS comment statements are outside the v0.1 subset.", span=_span(normalized, comment.start(), comment.start() + 1), command="*", ) - return _Parser(normalized).parse() + return _Parser(normalized, official_v03=official_v03).parse() diff --git a/src/openstatspec/sql/inplace_transform.py b/src/openstatspec/sql/inplace_transform.py index baeee98..eee3566 100644 --- a/src/openstatspec/sql/inplace_transform.py +++ b/src/openstatspec/sql/inplace_transform.py @@ -908,9 +908,11 @@ def install_in_place_transformation_schema( engine.dispose() -def load_transformation_schema(connection: Any, dataset_id: str) -> VariableSchema: - """Read the live canonical variable schema within the caller's transaction.""" - return _input_schema(connection, dataset_id)[2] +def load_transformation_schema( + connection: Any, dataset_id: str, *, lock_dataset: bool = False, +) -> VariableSchema: + """Read the live schema, optionally locking its dataset until transaction end.""" + return _input_schema(connection, dataset_id, lock_dataset=lock_dataset)[2] def _compensate_failed_apply( @@ -1009,7 +1011,12 @@ def _run_in_place_submission( ) sqlite_version = (*sqlite_version_parts, 0, 0)[:3] allow_delete_variable = sqlite_version >= (3, 35, 0) - engine = create_engine(database_url) + # Catalog validation precedes the dataset lock: post-lock schema reads must + # see commits made while waiting, not MySQL/MariaDB's earlier RR snapshot. + engine = create_engine( + database_url, + isolation_level="READ COMMITTED" if profile.name in {"mysql", "mariadb"} else None, + ) if profile.name == "sqlite": @sqlalchemy_event.listens_for(engine, "connect") def enable_sqlite_foreign_keys(dbapi_connection, _connection_record): diff --git a/src/openstatspec/transform/schema.py b/src/openstatspec/transform/schema.py index 4b66135..6a871cb 100644 --- a/src/openstatspec/transform/schema.py +++ b/src/openstatspec/transform/schema.py @@ -2,7 +2,7 @@ from __future__ import annotations -from dataclasses import dataclass +from dataclasses import dataclass, field from typing import Literal from .plan import TransformationPlan, ValueLabel @@ -22,6 +22,8 @@ class VariableDefinition: format_decimals: int | None = None measurement_level: Literal["nominal", "ordinal", "scale"] | None = None declared_string_width: int | None = None + # Official requests carry descriptive format metadata, not only writable F formats. + _validate_format: bool = field(default=True, repr=False, compare=False, kw_only=True) def __post_init__(self) -> None: if not isinstance(self.name, str) or not self.name: @@ -46,7 +48,7 @@ def __post_init__(self) -> None: ) format_parts = (self.format_family, self.format_width, self.format_decimals) - if any(part is not None for part in format_parts): + if self._validate_format and any(part is not None for part in format_parts): if any(part is None for part in format_parts): raise ValueError("Format family, width, and decimals must be set together.") if self.storage_kind != "numeric" or self.format_family != "F": diff --git a/tests/test_conditional_inplace_transform.py b/tests/test_conditional_inplace_transform.py index f4c62f3..d27991e 100644 --- a/tests/test_conditional_inplace_transform.py +++ b/tests/test_conditional_inplace_transform.py @@ -69,6 +69,101 @@ def conditional_catalog(tmp_path): return url, path, dataset_id, table_name +def test_official_v03_apply_preserves_unknown_metadata_identity_and_provenance(conditional_catalog): + from sqlalchemy import create_engine + + url, path, dataset_id, table_name = conditional_catalog + contract = "openstatspec-spss-syntax-frontend-v0.3" + source = ("COMMENT official frontend.\r\nCOMPUTE target = 0. " + "IF (NOT source_a = 1 AND source_b >= 1 OR source_b NE 2) target = 1. " + "IF (NOT (source_a = 1 OR source_b = 1)) target = 2. " + "RECODE source_a (LOWEST THRU HIGHEST = COPY). " + "VARIABLE LABELS target 'Official'. VALUE LABELS target 0 'No' 1 'Yes'. " + "ADD VALUE LABELS target 2 'Other' 1 'Updated'. " + "FORMATS source_a TO target (F8.0). VARIABLE LEVEL target (NOMINAL). EXECUTE.") + with sqlite3.connect(path) as connection: + connection.execute(f'UPDATE "{table_name}" SET source_a = NULL, source_b = NULL WHERE __case_ordinal = 2') + tables = connection.execute("SELECT name FROM sqlite_master ORDER BY name").fetchall() + identity = connection.execute("SELECT * FROM dataset").fetchall() + original_variables = connection.execute("SELECT variable_id, source_ordinal, physical_name FROM variable ORDER BY source_ordinal").fetchall() + engine = create_engine(url) + try: + with engine.connect() as connection: + schema = inplace_transform.load_transformation_schema(connection, dataset_id) + finally: + engine.dispose() + expected = openstatspec.compile_spss_syntax(source, schema, frontend_contract=contract) + result = openstatspec.apply_spss_in_place( + database_url=url, dataset_id=dataset_id, source_text=source, + actor="official-test", frontend_contract=contract, + ) + assert (result["dataset_id"], result["physical_table_name"], result["frontend_contract"]) == (dataset_id, table_name, contract) + assert result["dolt_commit_performed"] is False + with sqlite3.connect(path) as connection: + assert connection.execute(f'SELECT source_a, source_b, target FROM "{table_name}" ORDER BY __case_ordinal').fetchall() == [ + (1, 1, 1), (None, None, 0), (0, 1, 1), (2, 2, 2), + ] + assert connection.execute("SELECT * FROM dataset").fetchall() == identity + assert connection.execute("SELECT name FROM sqlite_master ORDER BY name").fetchall() == tables + assert connection.execute("SELECT variable_id, source_ordinal, physical_name FROM variable ORDER BY source_ordinal").fetchall()[:2] == original_variables + assert connection.execute("SELECT variable_label, print_format_family, print_format_width, print_format_decimals, measurement_level FROM variable WHERE source_name = 'target'").fetchone() == ("Official", "F", 8, 0, "nominal") + assert connection.execute("SELECT numeric_code, label FROM value_label ORDER BY ordinal").fetchall() == [(0, "No"), (1, "Updated"), (2, "Other")] + assert connection.execute("SELECT source_kind, frontend_contract, source_hash, plan_hash, canonical_plan_json, actor FROM transformation_apply").fetchone() == ( + "spss_syntax", contract, expected.source_hash, expected.plan_hash, expected.plan.canonical_json(), "official-test", + ) + before_failure = tuple(connection.iterdump()) + for invalid in ("COMPUTE target = 99. STRING note (A4).", "COMPUTE target = 99. DELETE VARIABLES source_a."): + with pytest.raises(openstatspec.TransformationFrontendError, match="unsupported_spss_command"): + openstatspec.apply_spss_in_place(database_url=url, dataset_id=dataset_id, source_text=invalid, actor="official-test", frontend_contract=contract) + with sqlite3.connect(path) as connection: + assert tuple(connection.iterdump()) == before_failure + + +def test_live_add_labels_locks_before_schema_read(conditional_catalog, monkeypatch): + from sqlalchemy import event + from sqlalchemy.engine import Engine + + url, path, dataset_id, _ = conditional_catalog + order = [] + locked_connection = None + original_apply = inplace_transform._apply_plan_on_connection + + def observe_lock(connection, clause, *args): + nonlocal locked_connection + if getattr(clause, "_for_update_arg", None) is not None: + assert connection.in_transaction() + locked_connection = connection + order.append("lock") + if getattr(clause, "is_select", False) and any( + getattr(table, "name", None) == "variable" for table in clause.get_final_froms() + ): + assert locked_connection is connection, "schema read before dataset lock" + order.append("read") + + def apply(connection, **kwargs): + assert locked_connection is connection + assert connection.in_transaction() + order.append("execute") + return original_apply(connection, **kwargs) + + monkeypatch.setattr(inplace_transform, "_apply_plan_on_connection", apply) + event.listen(Engine, "before_execute", observe_lock) + try: + for code, label in ((1, "One"), (2, "Two")): + order.clear() + locked_connection = None + openstatspec.apply_spss_in_place( + database_url=url, dataset_id=dataset_id, + source_text=f"ADD VALUE LABELS source_a {code} '{label}'.", + actor="lock-test", frontend_contract="openstatspec-spss-syntax-frontend-v0.3", + ) + assert order.index("lock") < order.index("read") < order.index("execute") + finally: + event.remove(Engine, "before_execute", observe_lock) + with sqlite3.connect(path) as connection: + assert connection.execute("SELECT numeric_code, label FROM value_label ORDER BY ordinal").fetchall() == [(1, "One"), (2, "Two")] + + def test_exact_bounded_program_compiles_to_stable_v02_plan() -> None: schema = openstatspec.VariableSchema(( openstatspec.VariableDefinition("source_a", "numeric"), diff --git a/tests/test_frontend_v03.py b/tests/test_frontend_v03.py new file mode 100644 index 0000000..8772221 --- /dev/null +++ b/tests/test_frontend_v03.py @@ -0,0 +1,227 @@ +"""Official Frontend 0.3 evidence; fixtures remain specification-owned.""" +from copy import deepcopy +import json + +import pytest + +import openstatspec +from openstatspec.frontends.spss import compiler +from openstatspec.frontends.spss import spss_source_hash +from test_transform_frontend import _frontend_conformance_manifest + +CONTRACT = "openstatspec-spss-syntax-frontend-v0.3" + + +def _cases(): + root = _frontend_conformance_manifest().parent + manifest = json.loads((root / "spss-syntax-frontend-0.3.json").read_text()) + cases = [("0.3", case) for case in manifest["cases"]] + assert len(cases) == 35 + for inherited in manifest["inherited_manifests"]: + parent = json.loads((root / inherited["manifest"]).read_text()) + for case in parent["cases"]: + if case["id"] not in inherited["superseded_cases"]: + case = deepcopy(case) + case["request"]["contract"] = inherited["request_contract_override"] + cases.append((parent["manifest_version"], case)) + assert len(cases) == 90 + return cases + + +@pytest.mark.parametrize("version,case", _cases(), ids=lambda x: x["id"] if isinstance(x, dict) else x) +def test_effective_official_frontend_v03(version, case): + request = case["request"] + assert request["contract"] == CONTRACT + assert spss_source_hash(request["source_text"]) == case["expected_source_hash"] + if case["expected_error"]: + with pytest.raises(openstatspec.TransformationFrontendError) as caught: + compiler.compile_spss_request(request) + assert caught.value.code == case["expected_error"] + assert caught.value.span is not None + return + compilation = compiler.compile_spss_request(request) + expected = case.get("expected_plan", case.get("expected_plan_0_1")) + expected_hash = case.get("expected_plan_hash") + reference = case.get("expected_plan_case", case.get("expected_plan_case_0_1")) + if reference: + plan_version = "0.1" if "expected_plan_case_0_1" in case else version + manifest = json.loads((_frontend_conformance_manifest().parent / f"transformation-plan-{plan_version}.json").read_text()) + plan_case = next((c for c in manifest["cases"] if c["id"] == reference), None) + if plan_case is None: + frontend = json.loads((_frontend_conformance_manifest().parent / f"spss-syntax-frontend-{plan_version}.json").read_text()) + plan_case = next(c for c in frontend["cases"] if c["id"] == reference) + expected = plan_case.get("plan", plan_case.get("expected_plan")) + assert expected_hash is None or expected_hash == plan_case["expected_plan_hash"] + expected_hash = plan_case["expected_plan_hash"] + assert expected is not None + assert compilation.plan.as_dict() == expected + assert compilation.plan_hash == expected_hash + assert compilation.frontend_contract == CONTRACT + assert compilation.source_hash == case["expected_source_hash"] + if "expected_output_metadata" in case: + assert {v.name: {"variable_label": v.variable_label, "value_labels": [x.as_dict() for x in v.value_labels]} + for v in compilation.bound.output_schema.variables} == case["expected_output_metadata"] + + +def _request(source="EXECUTE."): + return {"contract": CONTRACT, "input_alias": "parent", "source_text": source, + "input_schema": {"variables": [{"name": n, "storage_kind": "numeric"} for n in ("a", "b", "c", "target")]}} + + +@pytest.mark.parametrize("source,explicit", [ + ("NOT a = 1 AND b = 1 OR c = 1", "((NOT (a = 1)) AND b = 1) OR c = 1"), + ("NOT NOT a = 1", "a = 1"), + ("NOT (a NE 1 OR b <= 2)", "(a >= 1 AND a <= 1) AND b > 2"), +]) +def test_resolved_not_precedence_and_flattening(source, explicit): + def compile_predicate(predicate): + return compiler.compile_spss_request(_request(f"IF ({predicate}) target = 1.")).plan + assert compile_predicate(source) == compile_predicate(explicit) + if source.startswith("NOT a"): + assert compile_predicate(source) != compile_predicate("NOT (a = 1 AND b = 1 OR c = 1)") + + +@pytest.mark.parametrize("source", ["STRING note (A4).", "DELETE VARIABLES a."]) +def test_official_rejects_python_extensions(source): + with pytest.raises(openstatspec.TransformationFrontendError) as caught: + compiler.compile_spss_request(_request(source)) + assert caught.value.code == "unsupported_spss_command" + + +@pytest.mark.parametrize("path,value", [ + (("contract",), "openstatspec-spss-syntax-frontend-v0.2"), + (("contract",), []), (("extra",), True), (("input_alias",), 1), + (("source_text",), ""), (("input_schema", "extra"), 1), + (("input_schema", "variables"), []), + (("input_schema", "variables"), ()), + (("input_schema", "variables", 0, "physical_name"), "a"), + (("input_schema", "variables", 0, "declared_string_width"), 4), + (("input_schema", "variables", 0, "width"), True), + (("input_schema", "variables", 0, "decimals"), -1), + (("input_schema", "variables", 0, "variable_label"), 1), + (("input_schema", "variables", 0, "format_family"), []), + (("input_schema", "variables", 0, "measurement_level"), {}), + (("input_schema", "variables", 0, "storage_kind"), []), + (("input_schema", "variables", 0, "value_labels"), [{"value": {"type": "binary64", "bits": "8000000000000000"}, "label": "zero"}]), + (("input_schema", "variables", 0, "value_labels"), [{"value": {"type": "binary64", "bits": "7ff0000000000000"}, "label": "infinity"}]), + (("input_schema", "variables", 0, "value_labels"), [{"value": {"type": "string", "value": "x"}, "label": 2}]), +]) +def test_request_boundary_rejects_wrong_fields_and_types(path, value): + request = _request() + target = request + for key in path[:-1]: + target = target[key] + target[path[-1]] = value + with pytest.raises(openstatspec.TransformationFrontendError) as caught: + compiler.compile_spss_request(request) + assert caught.value.code == "invalid_spss_request" + + +def test_grouped_commands_and_comments_preserve_order_and_metadata(): + source = ("/* before */ COMMENT command. * star.\r\n" + "RECODE a TO b (LOWEST THRU 1 = 0) / c (1 THRU HIGHEST = 2).\r\n" + "VARIABLE LABELS a TO b '/* literal */ O''Brien' / c 'Third'. " + "FORMATS a b (F8.2) / c (F4.0). " + "VARIABLE LEVEL a TO b (ORDINAL) / c (SCALE). " + "VALUE LABELS a TO b -0 'Zero' 1 'One'. " + "ADD VALUE LABELS a TO b 2 'Two' 0 'Updated' / c 3 'Three'.") + compilation = openstatspec.compile_spss_request(_request(source)) + operations = compilation.plan.operations + assert [op.op for op in operations] == ( + ["recode"] * 3 + ["set_variable_label"] * 3 + ["set_format"] * 3 + + ["set_measurement_level"] * 3 + ["replace_value_labels"] * 5 + ) + a, b, c, _ = compilation.bound.output_schema.variables + for variable in (a, b): + assert variable.variable_label == "/* literal */ O'Brien" + assert (variable.format_family, variable.format_width, variable.format_decimals, variable.measurement_level) == ("F", 8, 2, "ordinal") + assert [(x.value.number(), x.label) for x in variable.value_labels] == [(0, "Updated"), (1, "One"), (2, "Two")] + assert c.variable_label == "Third" + assert (c.format_width, c.format_decimals, c.measurement_level) == (4, 0, "scale") + assert compilation.source_text_lf == source.replace("\r\n", "\n") + assert compilation.source_hash == spss_source_hash(source) + + +@pytest.mark.parametrize("source", [ + "COMMENT text. EXECUTE.", "* text. EXECUTE.", "/* text */ EXECUTE.", + "FORMATS a TO c (F8.0).", "FORMATS a b (F8.0).", + "VARIABLE LABELS a b 'Both'.", "RECODE a (1 = 0) / b (1 = 0).", + "IF (a NE 1) target = 1.", "IF (a <> 1) target = 1.", + "IF (a ~= 1) target = 1.", "IF (NOT a = 1) target = 1.", + "RECODE a (LOWEST THRU HIGHEST = 0).", "ADD VALUE LABELS a 1 'One'.", +]) +def test_official_expansion_does_not_change_default_rejections(source): + request = _request(source) + schema = openstatspec.VariableSchema(tuple(openstatspec.VariableDefinition(v["name"], v["storage_kind"]) for v in request["input_schema"]["variables"])) + with pytest.raises(openstatspec.TransformationFrontendError): + openstatspec.compile_spss_syntax(source, schema) + assert openstatspec.compile_spss_request(request).frontend_contract == CONTRACT + + +@pytest.mark.parametrize("source,code", [ + ("RECODE a (01=9).", "spss_syntax_error"), + ("RECODE a (1-2=9).", "spss_syntax_error"), + ("RECODE a (1 2=9).", "spss_syntax_error"), + ("RECODE a (1,=9).", "spss_syntax_error"), + ("RECODE a (1=2). / b (1=3).", "spss_syntax_error"), + ("COMMENT$bogus. EXECUTE.", "unsupported_spss_command"), + ("COMMENTé. EXECUTE.", "unsupported_spss_command"), + ("COMMENT.foo. EXECUTE.", "unsupported_spss_command"), + ("FORMATS ,a,,b, (F8.0).", "spss_syntax_error"), + ("ADD VALUE LABELS / a 1 'One'.", "spss_syntax_error"), + ("VALUE LABELS / a 1 'One'.", "spss_syntax_error"), + ("FORMATS / a (F8.0).", "spss_syntax_error"), + ("VARIABLE LABELS / a 'One'.", "spss_syntax_error"), + ("VARIABLE LEVEL / a (NOMINAL).", "spss_syntax_error"), + ("FORMATS a TO c TO b (F8.0).", "invalid_variable_range"), + ("/* outer /* inner */ */ EXECUTE.", "spss_syntax_error"), + ("COMPUTE a = 1 * not a comment.", "spss_syntax_error"), + ("COMMENT unterminated", "spss_syntax_error"), + ("EXECUTE. VALUE LABELS.", "spss_syntax_error"), + ("EXECUTE. ADD VALUE LABELS.", "spss_syntax_error"), + ("RECODE a (LOWEST = 0).", "spss_syntax_error"), + ("RECODE a (2 THRU 1 = 0).", "invalid_variable_range"), + ("RECODE a TO c (1 = 0) INTO only_one.", "spss_syntax_error"), + ("RECODE a (1 = 0) INTO a.", "target_already_exists"), + ("FORMATS c TO missing (F8.0).", "unknown_variable"), + ("FORMATS a (F8.0). VARIABLE LABELS missing 'No'.", "unknown_variable"), + ("ADD VALUE LABELS a -0 'Minus' 0 'Plus'.", "duplicate_value_label"), + ("EXECUTE. FORMATſ a (F8.0).", "unsupported_spss_command"), +]) +def test_official_source_boundary(source, code): + with pytest.raises(openstatspec.TransformationFrontendError) as caught: + openstatspec.compile_spss_request(_request(source)) + assert caught.value.code == code + assert caught.value.span is not None + + +@pytest.mark.parametrize("source,explicit", [ + ("FORMATS a TO b TO c (F8.0).", "FORMATS a b c (F8.0)."), + ("RECODE a (1,-2=9).", "RECODE a (1, -2 = 9)."), +]) +def test_official_selector_and_range_continuations(source, explicit): + assert compiler.compile_spss_request(_request(source)).plan == compiler.compile_spss_request(_request(explicit)).plan + + +def test_request_metadata_uses_schema_fields_without_coercion_or_loss(): + request = _request("VARIABLE LABELS a 'Changed'. EXECUTE.") + request["input_schema"]["variables"] = [ + {"name": "a", "storage_kind": "string", "format_family": "A", "width": 12, + "decimals": 0, "measurement_level": "nominal", "value_labels": [ + {"value": {"type": "string", "value": "x "}, "label": "Exact"}]}, + {"name": "b", "storage_kind": "numeric", "width": 8.0}, + ] + before = deepcopy(request) + a, b = openstatspec.compile_spss_request(request).bound.output_schema.variables + assert request == before + assert (a.format_family, a.format_width, a.format_decimals, a.measurement_level) == ("A", 12, 0, "nominal") + assert a.value_labels[0].value.value == "x " + assert (b.format_family, b.format_width, b.format_decimals) == (None, 8, None) + + +@pytest.mark.parametrize("missing", ["contract", "input_alias", "input_schema", "source_text"]) +def test_request_requires_every_contract_field(missing): + request = _request() + del request[missing] + with pytest.raises(openstatspec.TransformationFrontendError, match="invalid_spss_request"): + openstatspec.compile_spss_request(request) diff --git a/tests/test_inplace_transform.py b/tests/test_inplace_transform.py index baab261..9131898 100644 --- a/tests/test_inplace_transform.py +++ b/tests/test_inplace_transform.py @@ -1431,6 +1431,36 @@ def capture_profile(_database_url, *, dolt_conformance_source): assert captured == [sentinel] +@pytest.mark.parametrize("profile_name", ["mysql", "mariadb", "dolt", "sqlite", "postgresql"]) +def test_in_place_isolation_uses_real_profile(monkeypatch, profile_name) -> None: + database_url = ( + "sqlite://" if profile_name == "sqlite" + else "postgresql+psycopg://user@host/database" if profile_name == "postgresql" + else "mysql+pymysql://user@host/database" + ) + monkeypatch.setattr( + inplace_transform, "effective_profile", + lambda _url, **_kwargs: ( + SimpleNamespace(name=profile_name), {"server_version": "3.35.0"}, + ), + ) + captured = [] + + def capture_engine(url, **kwargs): + captured.append((url, kwargs.get("isolation_level"))) + raise RuntimeError("stop at engine creation") + + monkeypatch.setattr(inplace_transform, "create_engine", capture_engine) + with pytest.raises(RuntimeError, match="stop at engine creation"): + openstatspec.apply_transformation_plan_in_place( + database_url=database_url, dataset_id="synthetic", + plan=_plan("RECODE score (1 = 2)."), actor="test-agent", + ) + assert captured == [(database_url, ( + "READ COMMITTED" if profile_name in {"mysql", "mariadb"} else None + ))] + + def test_plan_apply_forwards_explicit_dolt_conformance_source(monkeypatch) -> None: sentinel = object() captured = [] diff --git a/tests/test_sql_services.py b/tests/test_sql_services.py index b945899..786740e 100755 --- a/tests/test_sql_services.py +++ b/tests/test_sql_services.py @@ -269,6 +269,79 @@ def write_after_read(*args, **kwargs): engine.dispose() +@pytest.mark.parametrize("environment_name", [ + "OPENSTATSPEC_MYSQL_URL", "OPENSTATSPEC_MARIADB_URL", +]) +def test_live_in_place_add_labels_sees_commit_after_catalog_validation( + environment_name, source_sav, monkeypatch, +): + database_url = os.environ.get(environment_name) + if not database_url: + pytest.skip(f"{environment_name} is not configured") + imported = openstatspec.import_sav( + source_sav, database_url=database_url, dataset_id="labels_" + uuid4().hex, + ) + engine = create_engine(database_url) + tables = normative_catalog(MetaData()) + verify = inplace_transform.require_verified_catalog + connections = [] + + def commit_second_apply_after_validation(connection, **kwargs): + result = verify(connection, **kwargs) + connections.append(connection) + if len(connections) == 1: + # The catalog SELECT has already established a snapshot under RR; + # the first apply has not yet acquired its prepare-time dataset lock. + assert connection.in_transaction() + writer = openstatspec.apply_spss_in_place( + database_url=database_url, dataset_id=imported["dataset_id"], + source_text="ADD VALUE LABELS age 35 'second writer'.", + actor="second-writer", + frontend_contract="openstatspec-spss-syntax-frontend-v0.3", + ) + assert writer["status"] == "succeeded" + assert connections[1] is not connection + assert connections[1].closed + return result + + try: + openstatspec.install_in_place_transformation_schema(database_url=database_url) + with monkeypatch.context() as patch: + patch.setattr( + inplace_transform, "require_verified_catalog", + commit_second_apply_after_validation, + ) + result = openstatspec.apply_spss_in_place( + database_url=database_url, dataset_id=imported["dataset_id"], + source_text="ADD VALUE LABELS age 36 'first writer'.", + actor="first-writer", + frontend_contract="openstatspec-spss-syntax-frontend-v0.3", + ) + assert result["status"] == "succeeded" + assert len(connections) == 2 + _, variables, _ = wide.read_wide_dataset( + database_url=database_url, dataset_id=imported["dataset_id"], + ) + age = next(variable for variable in variables if variable["source_name"] == "age") + assert json.loads(age["value_labels"]) == { + "34.0": "thirty-four", "35.0": "second writer", "36.0": "first writer", + } + finally: + with engine.begin() as connection: + audit = inplace_transform.apply_audit_catalog(MetaData()) + if inspect_database(connection).has_table(audit.name): + connection.execute(audit.delete().where( + audit.c.dataset_id == imported["dataset_id"], + )) + delete_dataset_representation(connection, tables, imported["dataset_id"]) + quote = connection.dialect.identifier_preparer.quote + connection.exec_driver_sql(f"DROP TABLE {quote(imported['data_table'])}") + connection.execute(tables.operation.delete().where( + tables.operation.c.operation_id == imported["operation_id"], + )) + engine.dispose() + + def test_live_unknown_dolt_is_read_only_and_rejects_default_writes( tmp_path, ) -> None: