Skip to content

fix(clickhouse): strip virtual catalogs from executed queries and inserts - #6036

Open
mday-io wants to merge 3 commits into
SQLMesh:mainfrom
mday-io:mday/fix-clickhouse-insert-virtual-catalog
Open

mday-io wants to merge 3 commits into
SQLMesh:mainfrom
mday-io:mday/fix-clickhouse-insert-virtual-catalog

Conversation

@mday-io

@mday-io mday-io commented Sep 9, 2026 •

Copy link
Copy Markdown
Collaborator

Description

When a ClickHouse gateway is used alongside catalog-aware gateways, SQLMesh injects a virtual catalog (__<gateway>__ by default, or the configured virtual_catalog) so every model has a 3-part name. ClickHouse doesn't support catalogs, so that prefix has to be stripped before SQL reaches the engine. Some DDL paths already handled this (create_view, alter_table, delete_from's target table, create_schema), but the queries themselves still carried the catalog. As a result, INSERT ... SELECT, CREATE TABLE ... AS SELECT and subqueries inside DELETE ... WHERE all failed.

This PR strips the virtual catalog centrally in ClickhouseEngineAdapter._to_sql, so it covers every rendered statement:

  • Any table or column reference whose catalog equals the injected _default_catalog has its catalog removed.
  • References to other catalogs and string literals are left untouched.
  • The caller's expression is never mutated. It is only copied when a reference actually needs rewriting, so large INSERT ... VALUES statements aren't deep-copied for no reason.
  • Stripping only happens after inject_virtual_catalog() has run. This matches the existing strip sites, so a ClickHouse-only project with virtual_catalog configured behaves exactly as before.

Test Plan

Added to tests/core/engine_adapter/test_clickhouse.py:

  • test_virtual_catalog_stripped_from_execute_queries (parametrized) covers INSERT ... SELECT, a plain SELECT, and a JOIN that mixes the virtual catalog with another catalog plus a string literal containing the catalog name. It also checks that the input expression isn't mutated.
  • test_virtual_catalog_stripped_from_ctas_and_delete checks the exact SQL emitted by ctas() and by delete_from() with a subquery.

Both tests fail without the fix. All 38 tests in test_clickhouse.py pass, and so does ruff/mypy on the changed files.

Checklist

  • I have run make style and fixed any issues
  • I have added tests for my changes (if applicable)
  • All existing tests pass (make fast-test)
  • My commits are signed off (git commit -s) per the DCO

…erts

Signed-off-by: mday-io <mdaytn@gmail.com>
Strip the injected virtual catalog from every rendered expression rather
than only queries and inserts, so CTAS and DELETE statements no longer
leak it. Drop the fallback to the configured virtual_catalog when none was
injected, matching the other catalog strip sites, and only copy the
expression when a reference actually needs rewriting.

Consolidate the tests: the unconfigured-catalog test never enabled the
feature, and the fallback test only covered the removed fallback.

Signed-off-by: mday-io <mdaytn@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant