Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 53 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -255,6 +255,57 @@ jobs:
CORTEX_RERANKER_OFFLINE: "1"
run: python scripts/verify_mcp_hosts.py -- hypermnesia-mcp

# The plugin's own PostgreSQL server for real: pinned theseus + pgvector
# archives, initdb, start, vector + pg_trgm, schema, stop. macOS runs under
# sandbox-exec with nothing under /opt/homebrew or /usr/local readable (the
# pg0 dyld failure of PR #748's review); Linux loads the runner's libraries.
# source: ADR-1110
managed-pg:
needs: [changes, lint]
if: github.event_name != 'schedule' && (github.event_name != 'pull_request' || needs.changes.outputs.code == 'true' || needs.changes.outputs.deps == 'true' || needs.changes.outputs.workflows == 'true')
name: Managed PostgreSQL (${{ matrix.leg }})
runs-on: ${{ matrix.os }}
# No timeout-minutes: the server's start and stop wait on kernel events,
# never a clock (ADR-1110 D2); a hang is a defect to fix, not a verdict.
strategy:
fail-fast: false
matrix:
include:
- leg: linux-x86_64
os: ubuntu-latest
- leg: linux-aarch64
os: ubuntu-24.04-arm
- leg: macos-arm64
os: macos-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1

- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"

- name: Install the launcher base set (locked)
uses: ./.github/actions/uv-locked-install
with:
set: --only-group launcher-base
environment: ${{ env.pythonLocation }}

# The same invocation as tests_py/scripts/test_launcher_pg_e2e.py: an
# empty environment, and on macOS no readable Homebrew.
- name: Setup, a launch, vector and pg_trgm, stop
run: |
set -euo pipefail
work="$RUNNER_TEMP/managed-pg"
mkdir -p "$work/home"
locked=()
if [ "$RUNNER_OS" = macOS ]; then
locked=(sandbox-exec -f scripts/locked_mac_env/no_homebrew.sb)
fi
${locked[@]+"${locked[@]}"} env -i HOME="$work/home" PATH=/usr/bin:/bin:/usr/sbin:/sbin \
TMPDIR="$work" LANG=en_US.UTF-8 "$(command -v python)" -I \
scripts/locked_mac_env/managed_pg_e2e.py "$GITHUB_WORKSPACE" "$work"

mcp-host-config:
needs: [changes, lint]
if: github.event_name != 'schedule' && ((github.event_name != 'pull_request' || needs.changes.outputs.code == 'true' || needs.changes.outputs.deps == 'true' || needs.changes.outputs.workflows == 'true') && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.fork == false))
Expand Down Expand Up @@ -1092,6 +1143,7 @@ jobs:
- changes
- test
- test-sqlite
- managed-pg
- mcp-host-config
- test-windows
- release-deps
Expand All @@ -1111,7 +1163,7 @@ jobs:
# source: ADR-0802


ALLOWED_SKIPS: test,test-sqlite,mcp-host-config,test-windows,release-deps,craftsmanship,typecheck,build,docker-runtime-build,devcontainer-build,docker-smoke,install-e2e
ALLOWED_SKIPS: test,test-sqlite,managed-pg,mcp-host-config,test-windows,release-deps,craftsmanship,typecheck,build,docker-runtime-build,devcontainer-build,docker-smoke,install-e2e
run: python3 scripts/check_ci_gate_results.py

# source: ADR-0802
Expand Down
95 changes: 95 additions & 0 deletions .github/workflows/pgvector.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
name: pgvector

# Compiles the pgvector archive Setup installs (scripts/launcher_pg_assets.py)
# from the pinned upstream tag, against the pinned theseus PostgreSQL tree, on
# a runner of each target triple; attests each archive's build provenance and
# publishes the four archives once as assets of one prerelease of this
# repository, from main only: a branch builds and attests but never creates a
# release, so `gh attestation verify --source-ref refs/heads/main` holds for
# what Setup downloads. An existing release is never modified: a rebuild
# cannot replace the bytes the sha256 pins name.
# source: ADR-1110

on:
push:
paths:
- .github/workflows/pgvector.yml
- scripts/pgvector_build.py
- scripts/launcher_pg_assets.py # the pins: a version bump rebuilds
workflow_dispatch:

permissions:
contents: read

jobs:
build:
name: pgvector (${{ matrix.target }})
runs-on: ${{ matrix.os }}
permissions:
contents: read
id-token: write
attestations: write
strategy:
fail-fast: false
matrix:
include:
- target: aarch64-apple-darwin
os: macos-15
- target: x86_64-apple-darwin
os: macos-15-intel
# glibc 2.35 runners: the library's symbol versions stay under the
# tree's GLIBC_2.34 floor (checked by the build script).
- target: x86_64-unknown-linux-gnu
os: ubuntu-22.04
- target: aarch64-unknown-linux-gnu
os: ubuntu-22.04-arm
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false

- name: Build, prove it loads, package
run: python3 -I scripts/pgvector_build.py "${{ matrix.target }}" "$RUNNER_TEMP/pgvector"

- name: Attest build provenance
uses: actions/attest-build-provenance@4d101475d8b20a2381f78447822ac1eab6504dd8 # v4.2.2
with:
subject-path: ${{ runner.temp }}/pgvector/out/pgvector-*.tar.gz

- uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: pgvector-${{ matrix.target }}
path: ${{ runner.temp }}/pgvector/out/pgvector-*.tar.gz
if-no-files-found: error

publish:
needs: build
if: github.repository == 'cdeust/Cortex' && github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false

- uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
pattern: pgvector-*
merge-multiple: true
path: dist

- name: Publish once, never replace
env:
GH_TOKEN: ${{ github.token }}
run: |
set -euo pipefail
tag="$(python3 -I -c 'import sys; sys.path.insert(0, "scripts"); import launcher_pg_assets as a; print(a.PGVECTOR_RELEASE)')"
sha256sum dist/*.tar.gz | tee -a "$GITHUB_STEP_SUMMARY"
if gh release view "$tag" --repo "$GITHUB_REPOSITORY" --json assets --jq '.assets[].name'; then
echo "release $tag exists: nothing replaced" | tee -a "$GITHUB_STEP_SUMMARY"
exit 0
fi
gh release create "$tag" dist/*.tar.gz --repo "$GITHUB_REPOSITORY" \
--target "$GITHUB_SHA" --prerelease --title "$tag" \
--notes "pgvector built by .github/workflows/pgvector.yml (run $GITHUB_RUN_ID) from the pinned upstream tag against the pinned theseus PostgreSQL. Verify: gh attestation verify <archive> --repo $GITHUB_REPOSITORY --source-ref refs/heads/main"
33 changes: 33 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,39 @@ adheres to [Semantic Versioning](https://semver.org/).

### Added

- **The Claude Code plugin runs its own PostgreSQL server (#744, #740;
ADR-1110).** On macOS (arm64, x86_64) and Linux glibc (x86_64, aarch64),
Setup (`claude --init-only`) on a fresh install downloads the pinned
theseus-rs PostgreSQL 18.6.0 and a pgvector 0.8.7 archive (sha256-checked,
about 13 MB), runs `initdb --pwfile`, starts the server on 127.0.0.1 with
its cluster in `~/.claude/methodology/managed-pg` (uninstalling the plugin
never deletes memories), creates `vector`, `pg_trgm` and the schema, and
only then selects it. No Homebrew or system PostgreSQL is needed: the macOS
binaries carry their own OpenSSL (pg0, used by the earlier draft of this
change, loaded Homebrew's and failed in dyld on a Mac without it). The
pgvector archives are compiled by Cortex's CI (`pgvector.yml`: pinned
upstream commit, built and load-tested against the pinned PostgreSQL on each
platform, build provenance attested) and published from main only as a
release of this repository. The server is started and stopped on kernel events (ready in
`postmaster.pid`, process exit), never on a timer, and a postmaster is
adopted or signalled only when its PID and kernel start time are the ones
recorded at spawn and it authenticates with the cluster's password (a
wrong server signature, a foreign nonce, trust or a cleartext request is
refused). A postmaster that dies under live clients is started again on the
same port, and a client never takes a `running` status whose postmaster is
dead; one whose supervisor died before recording it is adopted once it
proves itself; a live postmaster of the cluster that Cortex did not start
is named and left alone, its `postmaster.pid` never removed. The Codex hook entry and `hypermnesia-mcp` stop in one line
(`Cortex not ready: ...`) when the server cannot run. The
Codex plugin, `hypermnesia-mcp` and `cortex-doctor` attach to the same
server, so Claude and Codex on one account share one store. A configured
`database_url` (`CLAUDE_PLUGIN_OPTION_DATABASE_URL` for hooks and Setup)
always wins, in Setup too; a server that cannot start leaves the SQLite
store selected, recorded as `sqlite_fallback` in the Setup outcome and shown
at session start as `SQLite fallback: <cause>`; Windows, musl and Linux
hosts missing `libxml2.so.2` or OpenSSL 3 keep SQLite. Existing SQLite
stores are not migrated. `setup_db.py` now fails when a schema statement
fails instead of reporting ready with warnings.
- **Benchmark refresh preparation: pinned datasets, a per-query journal,
offline token counting and energy per scored query.** Scores and retrieval
are unchanged; nothing here runs a scored benchmark.
Expand Down
27 changes: 24 additions & 3 deletions PRIVACY.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,13 +33,26 @@ like any other memory (locally) — avoid doing so.
configured PostgreSQL that is unreachable is an error shown at session
start ("Cortex not ready"), never a silent switch to SQLite. Nothing is
uploaded.
- **The plugin's own PostgreSQL (Claude Code plugin, macOS and Linux GNU,
fresh installs that ran `claude --init-only`):** the same data, in a
PostgreSQL server the plugin runs, listening on 127.0.0.1 only, with a
random password kept in a file only you can read. Its data (the cluster,
the password, its state) lives in `~/.claude/methodology/managed-pg/`, next
to the backend marker; only its binaries live in the plugin's data
directory. It is selected only after it started and its schema was created;
otherwise, and on Windows, the SQLite store above is used, and the cause is
shown at session start ("SQLite fallback: ..."). An existing SQLite store is
not moved. Uninstalling the plugin removes the binaries, never the
memories: delete `~/.claude/methodology/managed-pg/` to remove them. The
Codex plugin and direct `hypermnesia-mcp` launches on the same account use
this same server and store.
- **PostgreSQL + pgvector (opt-in — `install-plugin.sh --postgres`, large
stores, shared team databases):** your data is stored in **your own**
database, configured via the `DATABASE_URL` environment variable (default
`postgresql://127.0.0.1:5432/cortex`); the opt-in setup script installs
PostgreSQL locally on your machine, and an existing PostgreSQL install is
detected and kept across plugin updates. Cortex never provisions or
connects to any database you did not configure.
detected and kept across plugin updates. Apart from its own server above,
Cortex never provisions or connects to any database you did not configure.

You own this data. Deleting the database file (or the relevant rows) permanently
removes it. The `forget` tool deletes individual memories: a hard delete
Expand Down Expand Up @@ -91,7 +104,15 @@ The only outbound network activity is:
with the machine-learning set). Nothing asks a git remote
whether a release is out. These transfers fetch software only; no user
content is sent. `CORTEX_EMBEDDING_ZERO_DOWNLOAD=1` skips the
machine-learning packages and the model.
machine-learning packages and the model. When the plugin runs its own
PostgreSQL server (below), Setup also downloads, once per pinned version,
the PostgreSQL 18.6.0 archive of theseus-rs/postgresql-binaries and a
pgvector 0.8.7 archive from this repository's own GitHub releases (about
13 MB, each checked against a pinned sha256). The pgvector archives are
compiled by Cortex's CI from the pinned upstream source commit against that
same PostgreSQL, with a build-provenance attestation you can check with
`gh attestation verify <archive> --repo cdeust/Cortex --source-ref refs/heads/main`
(the workflow publishes only from main).
2. **One-time model download.** On first use, the open-source embedding model
(`sentence-transformers`, all-MiniLM-L6-v2) and the reranking model
(`flashrank`, ms-marco-MiniLM) are downloaded from the public Hugging Face
Expand Down
19 changes: 15 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,14 +107,24 @@ the same one, once done) shows `Cortex ready: <store>, <n> memories; <how recall
everything before the first session; only with it does that session start ready. To work
without Cortex, offline or not, run `claude plugin disable hypermnesia-mcp@cortex-plugins`.

The first use creates a local SQLite store under `~/.claude/methodology/`. Models are downloaded
On macOS and Linux (glibc), `claude --init-only` on a fresh install also gives the plugin its own
PostgreSQL server with pgvector and pg_trgm: pinned, sha256-checked binaries (about 13 MB; pgvector is
compiled by Cortex's CI, with build provenance attested) on 127.0.0.1, with no Homebrew or system
PostgreSQL needed. Its data lives in `~/.claude/methodology/managed-pg/`, so uninstalling the plugin
never deletes your memories. It becomes the store only after it has started and its schema exists;
if it cannot start (a Linux distribution without `libxml2.so.2` or OpenSSL 3, for example) the SQLite
store is used, as it is on Windows, and the session start says why (`SQLite fallback: ...`). The Codex
plugin on the same account attaches to the same server. An existing SQLite store is not moved
(ADR-1110).

Otherwise the first use creates a local SQLite store under `~/.claude/methodology/`. Models are downloaded
once when needed and then run offline. The embedding and reranking model files are both fetched
on first use. Optional integrations, remote PostgreSQL, and OTLP telemetry use the network only
when explicitly configured. [PRIVACY.md](PRIVACY.md) lists the exact scope.

An existing PostgreSQL setup is kept: a configured `database_url` option (or `DATABASE_URL`),
or the backend marker `install-plugin.sh --postgres` writes, always wins over the SQLite
default. An install that ran before on the local `cortex` database with nothing configured
or the backend marker `install-plugin.sh --postgres` writes, always wins over the plugin's own
server and the SQLite default. An install that ran before on the local `cortex` database with nothing configured
keeps it: on its first launch Cortex moves to SQLite only when nothing listens on
localhost:5432 and no `PG*` variable configures PostgreSQL, records the choice, and prints it. A fresh install never connects to a database nothing configured.

Expand Down Expand Up @@ -298,7 +308,8 @@ and BEAM results, the ablations and the floor gates are in [benchmarks/](benchma

## Storage

SQLite by default. PostgreSQL is one configuration field, worth it for very large stores or a
SQLite by default, or the plugin's own PostgreSQL server on a fresh macOS or Linux install (see
Install). Your own PostgreSQL is one configuration field, worth it for very large stores or a
database shared across a team.

```bash
Expand Down
2 changes: 1 addition & 1 deletion docs/adr/ADR-1104-zero-ops-plugin-install.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ title: Zero-ops plugin install

accepted

Decision 4 below (the SessionStart hook installs the base set, 300 s) is superseded by ADR-1108: Setup installs, no hook does, and sets live per lock. The other decisions stand.
Decision 4 below (the SessionStart hook installs the base set, 300 s) is superseded by ADR-1108: Setup installs, no hook does, and sets live per lock. Decision 1 (selection order) is amended by ADR-1110: a configured database still wins, then a `managed` marker (written by Setup only after the plugin's own PostgreSQL started with pgvector and pg_trgm) selects that server, then this decision's chain unchanged. The other decisions stand.

## Context

Expand Down
Loading
Loading