feat(db): move from Turso/libSQL to Postgres via @profullstack/libsql-pg

[ralyodio]

What changed

MeshHook's database layer moves from Turso/libSQL to Postgres at the code level. Nothing here touches a server or deploys anything.

• packages/shared/lib/db.js builds its client with @profullstack/libsql-pg@0.1.1 (createClient({ url: process.env.DATABASE_URL })). It keeps the @libsql/client surface over a pg pool and rewrites the SQLite idioms the queries picked up on Turso (strftime('%Y-%m-%dT%H:%M:%fZ','now'), ? placeholders, INSERT OR IGNORE) per statement. The exported db object (one / oneOrNone / manyOrNone / none / tx / batch) is unchanged, so no call site moved. @libsql/client is removed from @meshhook/shared and @meshhook/web.
• Fail fast: DATABASE_URL must be postgres:///postgresql://. Unset, libsql:// or file: throws with the reason; a leftover TURSO_DATABASE_URL is named together with the copy recipe. workers/orchestrator.mjs checks it at start. No fallback to a file database.
• Postgres migrations in migrations-pg/ (5 files). The SQLite files in migrations/ stay until the cutover is proven. Design decisions, all written at the top of migrations-pg/0001_core_tables.sql:
  • every timestamp column stays ISO-8601 text, with meshhook_now_iso() (a to_char(now() at time zone 'utc', ...) wrapper) as the default. The converter suggested timestamptz, but the app issues set updated_at = strftime(...) and where expires_at <= strftime(...), and Postgres will not assign or compare text against timestamptz;
  • ids default to gen_random_uuid()::text;
  • the seven update_*_updated_at triggers become one BEFORE UPDATE trigger function, with the workflow_runs WHEN guard preserved so the orchestrator's own updated_at write is not overwritten;
  • integer primary key (rowid alias) columns are identity bigint, so lastInsertRowid keeps working (Queue.send relies on it; libsql-pg appends RETURNING msg_id);
  • 0/1 flags stay bigint with their CHECK.
• scripts/db-migrate.js applies migrations-pg one transaction per file with the same checksum ledger; scripts/verify-migration.js reads information_schema/pg_indexes instead of sqlite_master and drops the PRAGMA foreign_keys check (a PRAGMA is a no-op through the shim, and destructuring its empty result would have thrown); scripts/setup.js collects a Postgres URL and writes DATABASE_URL.
• The in-process write lock and SQLITE_BUSY retry (there for libSQL's single multiplexed connection) are gone; a retry on 40001/40P01 replaces them, so the workers get real concurrency out of the pool.
• apps/web/vite.config.js externalises @profullstack/libsql-pg, pg and pg-native instead of the libsql bindings. .env.example and README updated.

Idioms fixed by hand

• Schema: the converter carried the SQLite UUID default (lower(hex(randomblob(4)) || ... substr('89ab', abs(random()) % 4 + 1, 1) ...) over as gen_random_bytes + abs(random()) % 4, which Postgres rejects (random() is double precision, no % operator). Replaced with gen_random_uuid()::text.
• Schema: status text default 'draft' check (status in (...)) came out as default 'draft' (check) (status in (...)) (see package bug below). Written by hand.
• Schema: triggers (emitted as TODOs) rewritten as a Postgres trigger function.
• Schema: the text timestamp columns the converter promoted to timestamptz are kept as text for the reason above (workers/orchestrator.mjs, packages/shared/lib/auth.js and scripts/db-migrate.js all assign strftime(...) output to them).
• scripts/verify-migration.js: PRAGMA foreign_keys and sqlite_master queries replaced.
• packages/shared/lib/db.js: toLibsqlSql ($n to ?) is kept because workers/http-exec.mjs still uses $1/$2.
• Checked and needing no change: rowid and json_extract appear only in comments; coalesce(max(version), 0), count(*), min/max on text, limit ? offset ?, on conflict (...) do update set ... = excluded..., returning * all pass through. No ?NNN, json_each, COLLATE NOCASE, last_insert_rowid(), LIMIT -1, FTS5, GLOB or IS NOT <value>.

Tests

pnpm test passes with no database: vitest 56 passed / 85 skipped (the queue, auth and orchestrator suites need a database and are describe.skipIf(!TEST_DATABASE_URL); with it set, each test gets its own schema via options=-c search_path=... and the real migrations-pg applied), node:test 258 passed. packages/shared/lib/db.test.js (new) covers the URL guard and the placeholder rewrite. pnpm --filter @meshhook/web run build passes with the new externals.

The DB-bound suites have therefore not been run against Postgres from here (no database access on this box). The coordinator should run TEST_DATABASE_URL=postgres://... pnpm test once against a scratch database before the cutover; the suites create and drop their own schemas.

Cutover recipe (coordinator)

# 1. schema (checksummed ledger in schema_migrations)
DATABASE_URL=postgres://... pnpm db:migrate
DATABASE_URL=postgres://... pnpm db:verify
# 2. rows (+ verify counts per table)
npx libsql-pg copy --from "$TURSO_DATABASE_URL" --token "$TURSO_AUTH_TOKEN" --to "$DATABASE_URL" --verify
# 3. set DATABASE_URL in the vault, unset TURSO_DATABASE_URL / TURSO_AUTH_TOKEN, deploy web + orchestrator

Beyond the recipe: the copier resets identity sequences (workflow_events.id, audit_log.id, queue_messages.msg_id) with setval(max); queue_archive.msg_id is a plain bigint PK, nothing to reset. queue_config is seeded by the migrations, so copy it with --upsert or --exclude queue_config to avoid a conflict on queue_name. Run ANALYZE after the load. The schema_migrations table on Turso has different columns (version/checksum) from the one the runner creates; exclude it from the copy too.

Package bugs found (@profullstack/libsql-pg@0.1.1, convert-schema)

1. DEFAULT '<literal>' CHECK (...) loses the CHECK. Input: status text default 'draft' check (status in ('draft', 'published', 'archived')). Expected: unchanged. Got: status text default 'draft' (check) (status in ('draft', 'published', 'archived')) plus the note status: default check -> (check). rewriteDefault runs its \sdefault\s+ regex on the code mask, where the blanked literal reads as whitespace, so the "default expression" it finds is the word check. Same for theme_preference text default 'light' check (...).
2. random() in a default is carried over unchanged (only a warning in unsupportedIdioms, none in the converter). Input: the SQLite v4-UUID default above. Got: abs(random()) % 4 + 1 inside gen_random_bytes(...) concatenation, which Postgres rejects (operator does not exist: double precision % integer). Expected: a TODO, or gen_random_uuid()::text when the expression matches the well-known randomblob UUID idiom.
3. Text-timestamp promotion has no CLI switch. promoteTextTimestamps: false exists in ConvertOptions but convert-schema exposes no flag for it, and the promotion is wrong for any app that assigns strftime(...) to those columns (Postgres refuses text into timestamptz). Every promoted column here had to be reverted by hand.

🤖 Generated with Claude Code

[github-actions]

ThreatCrush Security Scan

11 finding(s)

HIGH/CRITICAL: 5 | MEDIUM: 3 | LOW: 3

Severity	Rule	Location
HIGH	secret-database-url	apps/web/.env.example:9
HIGH	secret-database-url	docs/Environment-Setup.md:46
HIGH	secret-database-url	docs/Environment-Setup.md:132
HIGH	secret-slack-webhook	docs/WEBHOOK_CONFIGURATION.md:145
HIGH	secret-generic-credential	src/nodes/README.md:271
MEDIUM	sql-template-interpolation	apps/web/src/routes/api/secrets/[id]/+server.js:80
MEDIUM	sql-template-interpolation	apps/web/src/routes/api/workflows/[id]/+server.js:141
MEDIUM	sql-template-interpolation	src/queue/test-helpers.js:52
LOW	sql-template-interpolation	docs/Turso-Migration.md:70
LOW	secret-generic-credential	src/nodes/webhook.test.js:287
LOW	sql-template-interpolation	workers/orchestrator.test.js:52

Snippets are redacted; ThreatCrush never prints matched credential material.

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	@​profullstack/​libsql-pg@​0.1.1

View full report