diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..4075c8d --- /dev/null +++ b/.gitattributes @@ -0,0 +1 @@ +bin/*.wasm filter=lfs diff=lfs merge=lfs -text diff --git a/.github/workflows/docker.yml b/.github/workflows/docker.yml index dbcfa2c..b1c7962 100644 --- a/.github/workflows/docker.yml +++ b/.github/workflows/docker.yml @@ -26,6 +26,9 @@ jobs: - uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1 with: persist-credentials: false + # The .wasm files in bin/ are Git LFS objects. Without them, the + # tests and the image get the pointer files. + lfs: true - uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1 diff --git a/.github/workflows/go.yml b/.github/workflows/go.yml index a969be0..05b1dc9 100644 --- a/.github/workflows/go.yml +++ b/.github/workflows/go.yml @@ -24,6 +24,9 @@ jobs: with: persist-credentials: false fetch-tags: true + # The .wasm files in bin/ are Git LFS objects. Without them, the + # tests and the image get the pointer files. + lfs: true - uses: actions/setup-node@2028fbc5c25fe9cf00d9f06a71cc4710d4507903 # v6.0.0 with: diff --git a/AGENTS.md b/AGENTS.md index 87c2673..f387c5a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -49,34 +49,43 @@ Notes on the tests and on configuration: flag to UPPER_SNAKE. `-allow-push` becomes `ALLOW_PUSH`, and `-bucket` becomes `BUCKET`. - `godotenv` loads a `.env` file from the working directory at startup. +- The `.wasm` files in `bin/` are Git LFS objects. Run `git lfs pull` before + you build. Without them, `TestRepoBin` fails. The image copies `bin/` to + `/usr/libexec/objgit/bin`. For a local run with hooks, set + `WASM_PATH=./bin`. +- `TestCluster` in `internal/kube` needs a real cluster. It skips itself + without the `OBJGIT_TEST_KUBE_*` variables. - Tigris client credentials come from the standard AWS SDK chain, such as `AWS_PROFILE`. ## Where the code lives -| Path | Purpose | -| --------------------------------------------------------- | --------------------------------------------------------------------------------- | -| `cmd/objgitd/main.go` | Builds the one `*daemon` and starts every listener. | -| `cmd/objgitd/git_protocol.go` | The git:// server. Also holds `operationFor` and `(*daemon).authorize`. | -| `cmd/objgitd/http.go` | Smart HTTP. `*daemon` is the `http.Handler` itself. | -| `cmd/objgitd/ssh.go` | The SSH server and its per-session dispatch. | -| `cmd/objgitd/shell.go` | The SSH `sh` command: an interactive shell in the hook sandbox. | -| `cmd/objgitd/receivepack.go` | The go-git fork that streams hook output, plus `writePack`. | -| `cmd/objgitd/hooks.go` | Ref diffing and the sandboxed hook run. | -| `cmd/objgitd/snapshots.go` | The erofs snapshot run after a push. | -| `cmd/objgitd/lfs.go` | The Git LFS HTTP handlers and `git-lfs-authenticate` over SSH. | -| `internal/auth` | The one authorization interface. | -| `internal/lfs` | Git LFS: protocol types, the bucket store, and the presigner. | -| `internal/repofs` | Maps a repository path to a `storage.Storer`. | -| `internal/storage/tigris` | Repository storage. A `storage.Storer` on the bucket. | -| `internal/bundler` | The async upload queue behind that storer. | -| `internal/s3fs` | Daemon-level state only, which is the SSH host key. | -| `internal/mountfs`, `internal/treefs`, `internal/kefkash` | The hook sandbox filesystem and shell wiring. | -| `internal/metrics` | Every Prometheus vector, plus thin helpers. | -| `internal/gittest` | `Isolate`, which keeps the real git client in tests away from this repository. | -| `internal/snapshot` | erofs images of git trees: `Ensure`, `Open`, and the `Store` interface. | -| `internal/slog.go` | JSON handler init. | -| `cmd/membench/` | Push memory benchmark harness. Not shipped; see `docs/usage/memory-benchmark.md`. | +| Path | Purpose | +| --------------------------------------------------------- | -------------------------------------------------------------------------------------- | +| `cmd/objgitd/main.go` | Builds the one `*daemon` and starts every listener. | +| `cmd/objgitd/git_protocol.go` | The git:// server. Also holds `operationFor` and `(*daemon).authorize`. | +| `cmd/objgitd/http.go` | Smart HTTP. `*daemon` is the `http.Handler` itself. | +| `cmd/objgitd/ssh.go` | The SSH server and its per-session dispatch. | +| `cmd/objgitd/shell.go` | The SSH `sh` command: an interactive shell in the hook sandbox. | +| `cmd/objgitd/receivepack.go` | The go-git fork that streams hook output, plus `writePack`. | +| `cmd/objgitd/hooks.go` | Ref diffing and the sandboxed hook run. | +| `cmd/objgitd/snapshots.go` | The erofs snapshot run after a push. | +| `cmd/objgitd/lfs.go` | The Git LFS HTTP handlers and `git-lfs-authenticate` over SSH. | +| `internal/auth` | The one authorization interface. | +| `internal/kube` | The in-cluster API client, `kube:apply`, and `tekton:pipelinerun`. | +| `internal/wasmbin` | Runs the WASI programs in `-wasm-path`, such as kustomize, as hook commands. | +| `internal/lfs` | Git LFS: protocol types, the bucket store, and the presigner. | +| `internal/repofs` | Maps a repository path to a `storage.Storer`. | +| `internal/storage/tigris` | Repository storage. A `storage.Storer` on the bucket. | +| `internal/bundler` | The async upload queue behind that storer. | +| `internal/s3fs` | Daemon-level state only, which is the SSH host key. | +| `internal/mountfs`, `internal/treefs`, `internal/kefkash` | The hook sandbox filesystem and shell wiring. | +| `internal/metrics` | Every Prometheus vector, plus thin helpers. | +| `internal/gittest` | `Isolate`, which keeps the real git client in tests away from this repository. | +| `internal/snapshot` | erofs images of git trees: `Ensure`, `Open`, and the `Store` interface. | +| `internal/slog.go` | JSON handler init. | +| `bin/` | WASI programs for hooks (Git LFS). The image copies them to `/usr/libexec/objgit/bin`. | +| `cmd/membench/` | Push memory benchmark harness. Not shipped; see `docs/usage/memory-benchmark.md`. | ## Architecture @@ -87,7 +96,7 @@ describes the daemon and links to one page for each subsystem. | ------------------------------------------------------ | ----------------------------------------------------------------------- | | [transports.md](docs/architecture/transports.md) | Any transport. It holds two protocol points that are easy to get wrong. | | [auth.md](docs/architecture/auth.md) | Credentials, decisions, or a new `Authorizer`. | -| [hooks.md](docs/architecture/hooks.md) | Push hooks, output streaming, or the sandbox. | +| [hooks.md](docs/architecture/hooks.md) | Push hooks, output streaming, the sandbox, or its Kubernetes commands. | | [metrics.md](docs/architecture/metrics.md) | Any metric or instrumentation seam. | | [snapshots.md](docs/architecture/snapshots.md) | Snapshot images, the snapshot cache, or `runSnapshots`. | | [tigris-storer.md](docs/architecture/tigris-storer.md) | Object layout, refs, packs, the pack cache, or the upload path. | diff --git a/Dockerfile b/Dockerfile index ed8f9e0..7010ba6 100644 --- a/Dockerfile +++ b/Dockerfile @@ -19,6 +19,14 @@ RUN --mount=type=cache,target=/go/pkg/mod \ COPY . . +# The .wasm files in bin/ are Git LFS objects. A checkout without LFS has +# pointer files there, and the image would ship them. Stop here when a file +# does not start with the WebAssembly magic. +RUN for f in bin/*.wasm; do \ + head -c 4 "$f" | od -An -tx1 | grep -q '00 61 73 6d' || \ + { echo "$f is not WebAssembly; run git lfs pull" >&2; exit 1; }; \ + done + # Static, stripped binary. No cgo: objgitd is pure Go and answers the git # protocol natively (no `git` binary at runtime). RUN --mount=type=cache,target=/go/pkg/mod \ @@ -33,6 +41,10 @@ FROM gcr.io/distroless/static-debian12:nonroot COPY --from=build /objgitd /objgitd +# WASI programs for hooks, such as kustomize. The default -wasm-path looks +# here after /app/wasm/bin, so a program mounted there replaces one of these. +COPY bin/ /usr/libexec/objgit/bin/ + # Smart HTTP, git://, metrics. SSH (-ssh-bind) is opt-in; publish it yourself. EXPOSE 8080 9418 9090 diff --git a/README.md b/README.md index c0b7162..1cfcfef 100644 --- a/README.md +++ b/README.md @@ -12,6 +12,7 @@ Notable features: - ssh, http, and git protocol support. - [post-receive hooks](./docs/usage/hooks.md) powered by userspace sandboxed shells and [Kefka](https://xeiaso.net/blog/2026/dancing-mad-sandboxing/). - SSH sandbox access to the environment hooks run in. +- [Kustomize, Kubernetes, and Tekton commands](./docs/usage/kubernetes-hooks.md) in hooks, so a push can start a PipelineRun. - [Git LFS](./docs/usage/lfs.md) with presigned transfers, so large files move straight between the client and Tigris instead of through the daemon. LFS object bytes are deduplicated across every repository in the bucket. - EROFS snapshots per tree (including LFS pointer resolution). - basic prometheus metrics. diff --git a/bin/kustomize.wasm b/bin/kustomize.wasm new file mode 100755 index 0000000..c9b6764 --- /dev/null +++ b/bin/kustomize.wasm @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:1724a4e906800e5af0c104e816d56213fdf1d2cb6e2cef417c4e25d4baac061b +size 24380813 diff --git a/cmd/objgitd/git_protocol.go b/cmd/objgitd/git_protocol.go index 1756bf7..cdd41a1 100644 --- a/cmd/objgitd/git_protocol.go +++ b/cmd/objgitd/git_protocol.go @@ -13,8 +13,10 @@ import ( "github.com/go-git/go-git/v6/plumbing/transport" "github.com/go-git/go-git/v6/storage" "github.com/tigrisdata/objgit/internal/auth" + "github.com/tigrisdata/objgit/internal/kube" "github.com/tigrisdata/objgit/internal/metrics" "github.com/tigrisdata/objgit/internal/repofs" + "github.com/tigrisdata/objgit/internal/wasmbin" ) // handshakeTimeout bounds how long a client has to send its git-proto-request. @@ -72,6 +74,15 @@ type daemon struct { // registers no LFS route, so the feature reads as a 404 from outside // instead of as a broken endpoint. lfs *lfsService + + // kube is the Kubernetes API client behind the kube:apply and + // tekton:pipelinerun hook commands, nil when -allow-kubernetes is unset. + // A nil kube registers stubs that name the flag. + kube *kube.Client + + // bins is the WASI programs from -wasm-path, such as kustomize, that + // hooks and the SSH sh command can run. A nil bins adds none. + bins *wasmbin.Set } // storerFor reports whether a repository already exists at st, returning st diff --git a/cmd/objgitd/hooks.go b/cmd/objgitd/hooks.go index 00cb00e..944ee01 100644 --- a/cmd/objgitd/hooks.go +++ b/cmd/objgitd/hooks.go @@ -23,10 +23,12 @@ import ( "github.com/go-git/go-git/v6/plumbing/transport" "github.com/go-git/go-git/v6/storage" "github.com/tigrisdata/objgit/internal/kefkash" + "github.com/tigrisdata/objgit/internal/kube" "github.com/tigrisdata/objgit/internal/metrics" "github.com/tigrisdata/objgit/internal/mountfs" "github.com/tigrisdata/objgit/internal/pushevents" "github.com/tigrisdata/objgit/internal/treefs" + "github.com/tigrisdata/objgit/internal/wasmbin" "github.com/tigrisdata/objgit/internal/webhook" "mvdan.cc/sh/v3/expand" "mvdan.cc/sh/v3/interp" @@ -164,7 +166,7 @@ func (d *daemon) runHook(repoPath, service string, st storage.Storer, u refUpdat } stdin := strings.NewReader(hookStdin(u)) - sh, err := newHookShell(tree, changes, hookEnv(repoPath, service, u, changes), stdin, stdout, stderr) + sh, err := newHookShell(tree, changes, hookEnv(repoPath, service, u, changes), d.bins, d.kube, hookOrigin(repoPath, u), stdin, stdout, stderr) if err != nil { log.Error("hook: build shell", "err", err) return @@ -266,6 +268,18 @@ func hookEnv(repoPath, service string, u refUpdate, c hookChanges) []string { } } +// hookOrigin describes update u to the Kubernetes commands. It carries the +// same values as the OBJGIT_* variables, but a script cannot change it. The +// SSH sh shell can target a tag or a commit, so Branch is set only for a +// branch. +func hookOrigin(repoPath string, u refUpdate) kube.Origin { + o := kube.Origin{Repo: repoPath, Ref: u.Name.String(), Commit: u.New.String()} + if u.Name.IsBranch() { + o.Branch = u.Name.Short() + } + return o +} + // hookStdin mirrors git's post-receive stdin for update u: " \n". func hookStdin(u refUpdate) string { return u.Old.String() + " " + u.New.String() + " " + u.Name.String() + "\n" @@ -274,8 +288,11 @@ func hookStdin(u refUpdate) string { // newHookShell builds the kefka sandbox a hook runs in: /src is a lazy // read-only view of tree, /tmp is writable scratch that holds hookChangesFile, // and the shell starts in /src with env. Both push hooks and the SSH sh command -// use it, so the two environments cannot drift apart. -func newHookShell(tree *object.Tree, changes hookChanges, env []string, stdin io.Reader, stdout, stderr io.Writer) (*interp.Runner, error) { +// use it, so the two environments cannot drift apart. bins adds the WASI +// programs from -wasm-path, which can replace a kefka built-in. kc backs +// kube:apply and tekton:pipelinerun, which act for origin; nil registers stubs +// that say how to turn them on. +func newHookShell(tree *object.Tree, changes hookChanges, env []string, bins *wasmbin.Set, kc *kube.Client, origin kube.Origin, stdin io.Reader, stdout, stderr io.Writer) (*interp.Runner, error) { fsys := mountfs.New(map[string]billy.Filesystem{ "src": treefs.New(tree), "tmp": memfs.New(), @@ -288,6 +305,8 @@ func newHookShell(tree *object.Tree, changes hookChanges, env []string, stdin io coreutils.Register(reg) wasmprog.Register(reg) uutils.Register(reg) + bins.Register(reg) + kube.Register(reg, kc, origin) if err := reg.Chdir(fsys, "/src"); err != nil { return nil, fmt.Errorf("chdir /src: %w", err) } diff --git a/cmd/objgitd/kube_test.go b/cmd/objgitd/kube_test.go new file mode 100644 index 0000000..d1bfbeb --- /dev/null +++ b/cmd/objgitd/kube_test.go @@ -0,0 +1,193 @@ +package main + +import ( + "context" + "net" + "os" + "os/exec" + "path/filepath" + "strings" + "sync" + "testing" + "time" + + "github.com/go-git/go-billy/v6/memfs" + "github.com/go-git/go-git/v6/plumbing" + "github.com/tigrisdata/objgit/internal/auth" + "github.com/tigrisdata/objgit/internal/kube" + "github.com/tigrisdata/objgit/internal/kube/kubetest" + "github.com/tigrisdata/objgit/internal/repofs" + "github.com/tigrisdata/objgit/internal/wasmbin" +) + +// repoBins loads the repository's bin directory once, so the tests that run +// kustomize compile it once. +var repoBins = sync.OnceValues(func() (*wasmbin.Set, error) { + return wasmbin.Load(context.Background(), []string{filepath.Join("..", "..", "bin")}, "") +}) + +// tektonHook is the hook recipe from docs/usage/kubernetes-hooks.md. +const tektonHook = "kustomize build .tekton | kube:apply && tekton:pipelinerun .tekton/testrun.yaml\n" + +// TestReceivePackHookKubernetes pushes a repository with an Xe-style .tekton +// bundle and the Tekton hook recipe, against a fake Kubernetes API server. +func TestReceivePackHookKubernetes(t *testing.T) { + if _, err := exec.LookPath("git"); err != nil { + t.Skip("git not installed") + } + + const pipelinePath = "/apis/tekton.dev/v1beta1/namespaces/ci/pipelines/xe-x-build-test" + tests := []struct { + name string + enabled bool + deny [2]string // a verb and resource the fake refuses + wantOutput []string + avoidOutput []string + wantRun bool + }{ + { + name: "applies the bundle and creates a run for the pushed commit", + enabled: true, + wantOutput: []string{ + "pipeline.tekton.dev/xe-x-build-test serverside-applied", + "pipelinerun.tekton.dev/x-m-00001 created in namespace ci", + }, + wantRun: true, + }, + { + name: "a failed apply creates no run", + enabled: true, + deny: [2]string{"create", "pipelines"}, + wantOutput: []string{`cannot create resource "pipelines"`}, + avoidOutput: []string{"created in namespace"}, + }, + { + name: "without -allow-kubernetes the commands explain themselves", + enabled: false, + wantOutput: []string{"kube:apply: Kubernetes commands are disabled; start objgitd with -allow-kubernetes"}, + avoidOutput: []string{"created in namespace"}, + }, + } + + bins, err := repoBins() + if err != nil { + t.Fatal(err) + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + srv := kubetest.New(t) + if tt.deny[0] != "" { + srv.Deny(tt.deny[0], tt.deny[1]) + } + d := &daemon{ + sysFS: memfs.New(), + resolver: repofs.BucketResolver{Base: newMemBase()}, + authz: auth.AllowAnonymous{AllowWrite: true}, + allowHooks: true, + hookTimeout: 2 * time.Minute, // the first kustomize run compiles the module + bins: bins, + } + if tt.enabled { + d.kube = kube.New(kube.Config{ + Host: srv.URL, + HTTPClient: srv.Client(), + Token: func() (string, error) { return srv.Token(), nil }, + Namespace: "objgit", + }) + } + + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + ln, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatalf("listen: %v", err) + } + go func() { _ = d.ServeGitProtocol(ctx, ln) }() + + work := t.TempDir() + runGit(t, work, "init", "-b", "main") + runGit(t, work, "config", "user.email", "test@example.com") + runGit(t, work, "config", "user.name", "Test") + for _, name := range []string{"kustomization.yaml", "x.yaml", "testrun.yaml"} { + data, err := os.ReadFile(filepath.Join("..", "..", "internal", "wasmbin", "testdata", "tekton", name)) + if err != nil { + t.Fatal(err) + } + writeFile(t, filepath.Join(work, ".tekton", name), string(data)) + } + writeFile(t, filepath.Join(work, ".objgit", "hooks", "receive-pack"), tektonHook) + runGit(t, work, "add", ".") + runGit(t, work, "commit", "-m", "tekton") + sha := strings.TrimSpace(runGit(t, work, "rev-parse", "HEAD")) + + out := runGit(t, work, "push", "git://"+ln.Addr().String()+"/xe/x.git", "main") + + for _, want := range tt.wantOutput { + if !strings.Contains(out, want) { + t.Errorf("push output lacks %q:\n%s", want, out) + } + } + for _, avoid := range tt.avoidOutput { + if strings.Contains(out, avoid) { + t.Errorf("push output has %q:\n%s", avoid, out) + } + } + + var runs []kubetest.Request + for _, r := range srv.Requests() { + if strings.HasSuffix(r.Path, "/pipelineruns") { + runs = append(runs, r) + } + } + if !tt.wantRun { + if len(runs) != 0 { + t.Errorf("%d PipelineRuns created, want 0", len(runs)) + } + return + } + if _, ok := srv.Object(pipelinePath); !ok { + t.Errorf("no Pipeline at %s", pipelinePath) + } + if len(runs) != 1 { + t.Fatalf("%d PipelineRuns created, want 1", len(runs)) + } + meta := runs[0].Body["metadata"].(map[string]any) + if got := meta["annotations"].(map[string]any)[kube.AnnotationCommit]; got != sha { + t.Errorf("run commit annotation = %v, want the pushed commit %s", got, sha) + } + for _, p := range runs[0].Body["spec"].(map[string]any)["params"].([]any) { + p := p.(map[string]any) + if p["name"] == "commit" && p["value"] != sha { + t.Errorf("commit param = %v, want %s", p["value"], sha) + } + if p["name"] == "branch" && p["value"] != "main" { + t.Errorf("branch param = %v, want main", p["value"]) + } + } + }) + } +} + +func TestHookOrigin(t *testing.T) { + sha := plumbing.NewHash("0123456789abcdef0123456789abcdef01234567") + tests := []struct { + name string + ref plumbing.ReferenceName + wantBranch string + }{ + {"a pushed branch", "refs/heads/main", "main"}, + {"a nested branch", "refs/heads/feature/x", "feature/x"}, + {"an SSH shell on a tag", "refs/tags/v1.0", ""}, + {"an SSH shell on a commit", "refs/commits/0123456789abcdef0123456789abcdef01234567", ""}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got := hookOrigin("xe/x", refUpdate{Name: tt.ref, New: sha}) + want := kube.Origin{Repo: "xe/x", Ref: tt.ref.String(), Branch: tt.wantBranch, Commit: sha.String()} + if got != want { + t.Errorf("hookOrigin = %+v, want %+v", got, want) + } + }) + } +} diff --git a/cmd/objgitd/main.go b/cmd/objgitd/main.go index 1f47dff..c3e7155 100644 --- a/cmd/objgitd/main.go +++ b/cmd/objgitd/main.go @@ -25,11 +25,13 @@ import ( "github.com/tigrisdata/objgit" "github.com/tigrisdata/objgit/internal" "github.com/tigrisdata/objgit/internal/auth" + "github.com/tigrisdata/objgit/internal/kube" "github.com/tigrisdata/objgit/internal/lfs" "github.com/tigrisdata/objgit/internal/metrics" "github.com/tigrisdata/objgit/internal/repofs" "github.com/tigrisdata/objgit/internal/s3fs" "github.com/tigrisdata/objgit/internal/storage/tigris" + "github.com/tigrisdata/objgit/internal/wasmbin" tstorage "github.com/tigrisdata/storage-go" "golang.org/x/sync/errgroup" @@ -47,6 +49,11 @@ var ( allowHooks = flag.Bool("allow-hooks", false, "run .objgit/hooks/receive-pack in a sandbox after a successful push") hookTimeout = flag.Duration("hook-timeout", 60*time.Second, "wall-clock limit for a single hook run") + wasmPath = flag.String("wasm-path", "/app/wasm/bin,/usr/libexec/objgit/bin", "comma-separated directories of WASI programs that hooks and the SSH sh command can run; each *.wasm file is a command named after the file, the first directory with a name wins, as in PATH, and a directory that does not exist is skipped") + wasmCacheDir = flag.String("wasm-cache-dir", "", "directory that keeps compiled WASI programs from -wasm-path across restarts; empty keeps them in memory only, so each start compiles each program again on its first run") + + allowKubernetes = flag.Bool("allow-kubernetes", false, "enable the kube:apply and tekton:pipelinerun commands in hooks and the SSH sh command; they act with the pod's in-cluster ServiceAccount, so anybody who can push a hook gets its Kubernetes permissions. Needs -allow-hooks") + packCacheDir = flag.String("pack-cache-dir", "", "parent directory for the local pack cache; empty uses the OS temp directory") packCacheBytes = flag.Int64("pack-cache-bytes", 2<<30, "disk budget for the local pack cache, least-recently-used eviction; 0 disables caching") @@ -116,6 +123,11 @@ func main() { } } + if *allowKubernetes && !*allowHooks { + slog.Error("-allow-kubernetes needs -allow-hooks; the Kubernetes commands only run in hooks and the SSH sh command") + os.Exit(1) + } + ctx, cancel := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) defer cancel() @@ -198,6 +210,30 @@ func main() { snapshotTmpDir: *packCacheDir, } + if *allowKubernetes { + kc, err := kube.InCluster() + if err != nil { + slog.Error("-allow-kubernetes needs an in-cluster ServiceAccount", "err", err) + os.Exit(1) + } + d.kube = kc + } + + if *allowHooks { + // Load only lists the directories. Each program is read and compiled + // on its first run, which for kustomize takes seconds without a cache. + bins, err := wasmbin.Load(ctx, strings.Split(*wasmPath, ","), *wasmCacheDir) + if err != nil { + slog.Error("can't load WASI programs", "wasm_path", *wasmPath, "err", err) + os.Exit(1) + } + defer bins.Close(context.Background()) + if len(bins.Names()) == 0 { + slog.Warn("no WASI programs found; hooks cannot run kustomize", "wasm_path", *wasmPath) + } + d.bins = bins + } + if *allowLFS { // The store talks to the bucket, so it gets the hardened client like // every other request path. rawClient survives only for the presigner: @@ -222,6 +258,8 @@ func main() { "bucket", *bucket, "allow_push", *allowPush, "allow_hooks", *allowHooks, + "allow_kubernetes", *allowKubernetes, + "wasm_commands", d.bins.Names(), "allow_lfs", *allowLFS, "external_url", *externalURL, "pack_cache_bytes", *packCacheBytes, diff --git a/cmd/objgitd/shell.go b/cmd/objgitd/shell.go index 91e55d4..60918e1 100644 --- a/cmd/objgitd/shell.go +++ b/cmd/objgitd/shell.go @@ -219,7 +219,7 @@ func (d *daemon) handleShell(s ssh.Session, args []string) { } }() - sh, err := newHookShell(tree, changes, hookEnv(ref.Path(), receivePackHook, u, changes), nil, t, t) + sh, err := newHookShell(tree, changes, hookEnv(ref.Path(), receivePackHook, u, changes), d.bins, d.kube, hookOrigin(ref.Path(), u), nil, t, t) if err != nil { metrics.ObserveGitOp("ssh", "sh", "error", start) log.Error("ssh shell: build shell", "err", err) diff --git a/docs/architecture/hooks.md b/docs/architecture/hooks.md index be5cc75..4e2c4c8 100644 --- a/docs/architecture/hooks.md +++ b/docs/architecture/hooks.md @@ -82,6 +82,46 @@ Open files report their full mounted path so WASI can stat them after open. sets the interpreter's `Dir` to `/src`. Without this setting, interp copies the host working directory of the daemon into `$PWD`. +## Commands outside kefka + +`newHookShell` registers commands that kefka does not have: + +| Command | Package | Notes | +| ------------------------------------------------------ | ------------------ | --------------------------------------------------------------------------------------------------- | +| Each `.wasm` file in `-wasm-path`, such as `kustomize` | `internal/wasmbin` | WASI programs from directories in the image. `d.bins` holds them. It is nil without `-allow-hooks`. | +| `kube:apply`, `tekton:pipelinerun` | `internal/kube` | Real commands when `d.kube` is set by `-allow-kubernetes`. Otherwise stubs that name the flag. | + +`wasmbin.Load` lists the `-wasm-path` directories one time, at startup. It +does not read the files. The first directory with a name wins, as in `PATH`. +`newHookShell` registers the programs after the kefka built-ins and before +`internal/kube`. A program can replace a built-in such as `jq`. A program +cannot replace `kube:apply` or `tekton:pipelinerun`. + +Each program compiles on its first run. Only a successful compile stays in +memory, so the next run reads a missing or damaged file again. The compile of +kustomize takes about 4 seconds. With `-wasm-cache-dir`, wazero keeps the +compiled code on disk, and a later start loads it in about 0.15 seconds. If a +file is a Git LFS pointer, `Exec` returns an error that says so. + +The adapter is not kefka's generic `wasmcommand`, for two reasons. It sets +the guest `PWD` to the shell directory, because Go's wasip1 port takes its +working directory from `PWD`. Its wazero runtime also closes a running +instance when the context ends, so `-hook-timeout` stops a long build. All +programs share one runtime. + +`internal/kube` is a small REST client, not client-go. `InCluster` reads +the ServiceAccount mount and reads the token again for each request, because +bound tokens rotate. Each command run takes one `Session`, which caches +discovery for each `apiVersion` until the run ends. The commands get a `kube.Origin` +from `hookOrigin`, and not from the `OBJGIT_*` variables, because a script +can change those. The audit log (`kube: applied`, `kube: created`) and the +PipelineRun metadata use this origin. `splitAPIVersion` refuses an +`apiVersion` that is not a plain group and version, because both go into +request paths. `kubetest` is a fake +API server for the tests. `TestCluster` runs the same calls against a real +cluster when `OBJGIT_TEST_KUBE_*` is set. See +[../usage/kubernetes-hooks.md](../usage/kubernetes-hooks.md). + ## The interactive shell (`shell.go`) The SSH command `sh [branch]` opens the same sandbox as an interactive diff --git a/docs/plans/tekton-kubernetes-commands.md b/docs/plans/tekton-kubernetes-commands.md new file mode 100644 index 0000000..78bf415 --- /dev/null +++ b/docs/plans/tekton-kubernetes-commands.md @@ -0,0 +1,174 @@ +# Plan: Kubernetes and Tekton commands in the hook shell + +## Context + +Add `kustomize`, `kube:apply`, and `tekton:pipelinerun` to the Kefka shell +that push hooks and the SSH `sh` command use. A new `-allow-kubernetes` flag, +off by default, gates the two commands that talk to the Kubernetes API. + +The [Xe/x Tekton example](https://github.com/Xe/x/tree/master/.tekton) keeps +its Pipeline in a Kustomize bundle and its PipelineRun in a separate +`generateName` template. A hook applies the bundle, then creates a new run for +the pushed commit: + +```sh +kustomize build /src/.tekton | kube:apply && tekton:pipelinerun /src/.tekton/testrun.yaml +``` + +The supplied `var/kustomize.wasm` is a WASI module of 24,380,813 bytes, SHA-256 +`1724a4e906800e5af0c104e816d56213fdf1d2cb6e2cef417c4e25d4baac061b`. It reports +version `(devel)`, so its source revision is unknown. wazero compiles it in +about 4 s on an Apple M-series machine. + +## Decisions + +- **No client-go.** `internal/kube` is a small REST client. It covers in-cluster + config, per-`apiVersion` discovery, Server-Side Apply, and create. client-go + pulls in k8s.io/api and a large dependency tree. The three calls above are + about 300 lines. +- **YAML.** `sigs.k8s.io/yaml` converts each document to JSON the way kubectl + does. A line splitter on `---` separates documents, with the rules of + apimachinery's `YAMLReader`. +- **`kustomize` is always registered.** It renders files and has no cluster + access, so `-allow-kubernetes` does not gate it. +- **Disabled stubs.** Without `-allow-kubernetes`, `kube:apply` and + `tekton:pipelinerun` are registered as stubs. A stub prints how to turn + the command on and exits 1. "Command not found" would hide the reason. +- **`-allow-kubernetes` needs `-allow-hooks`.** The daemon exits at startup + when the first is set and the second is not. It also exits when the + in-cluster config cannot load. +- **Default namespace.** A document without `metadata.namespace` goes to the + pod's ServiceAccount namespace, read from the mounted `namespace` file. +- **Apply validates first.** `kube:apply` parses and validates the whole stream + before its first request. Then it applies in stream order, and stops at the + first API error. +- **PipelineRun `commit` parameter is required.** A missing `commit` parameter + is an error. Tekton can reject a parameter that the Pipeline does not + declare, so the command does not add one. +- **Metadata keys.** Annotations `objgit.tigrisdata.com/repo`, `/ref`, and + `/commit`. Label `objgit.tigrisdata.com/commit-prefix` holds the first 12 + hex characters of the commit. +- **Cancellation.** The kustomize runtime uses `WithCloseOnContextDone(true)`, + so the hook timeout stops a long build. The generic Kefka adapter does not. + +## Tasks + +### Task 1: Embed kustomize.wasm and its Kefka adapter + +Files: `internal/kustomize/kustomize.go`, `internal/kustomize/kustomize.wasm` +(Git LFS), `internal/kustomize/kustomize_test.go`, `.gitattributes`. + +- Move `var/kustomize.wasm` to `internal/kustomize/kustomize.wasm`. Track it with + `git lfs track`. +- `Command` implements `command.Execer`. It compiles once, and mounts + `ec.FS` at `/`. It sets guest `PWD` to `ec.GuestPWD()`, and passes `HOME` + and `TMPDIR` from the shell environment. +- `Exec` returns a clear error when the embedded bytes do not start with the + WASM magic `\0asm`. That is the result of a build from an unhydrated LFS + pointer. +- Tests (write first): the embedded bytes have the magic and the recorded + SHA-256. `kustomize build` of a fixture works from a relative path. + `-o /src/...` fails and `-o /tmp/...` succeeds. A cancelled context returns + an error. An Xe-style Pipeline bundle builds and sets `namespace: ci`. + +Test command: `go test ./internal/kustomize/` + +### Task 2: Kubernetes REST client + +Files: `internal/kube/client.go`, `internal/kube/yaml.go`, +`internal/kube/client_test.go`, `internal/kube/yaml_test.go`. + +- `InCluster() (*Client, error)` reads `KUBERNETES_SERVICE_HOST` and + `KUBERNETES_SERVICE_PORT`, plus `ca.crt` and `namespace`. It reads the token + file again for each request, because bound tokens rotate. +- `New(Config) *Client` is for tests. +- `(*Client).Apply(ctx, obj)` sends a Server-Side Apply PATCH with + `fieldManager=objgitd` and `force=false`. `(*Client).Create(ctx, obj)` sends a + POST. Both resolve kind to resource through discovery + (`/api/v1` or `/apis//`). +- `*APIError` carries the `Status` code, reason, and message. +- `DecodeYAMLStream(io.Reader) ([]Object, error)` splits the stream and + converts each document. It skips empty documents. +- Tests (write first): a fake API server covers discovery, create through + apply, re-apply, a 409 conflict, a 403 denial, an unknown kind, a + cluster-scoped kind, and the default namespace. The YAML tests cover + several documents, comments, empty documents, and malformed input. + +Test command: `go test ./internal/kube/` + +### Task 3: kube:apply and tekton:pipelinerun commands + +Files: `internal/kube/commands.go`, `internal/kube/tekton.go`, +`internal/kube/commands_test.go`. + +- `Register(reg, client)` registers both commands, or disabled stubs when + `client` is nil. +- `kube:apply` reads standard input and takes no arguments. An empty stream, + a document without `apiVersion`, `kind`, or `metadata.name`, an API error, + and a conflict all exit nonzero, with the API message on stderr. It + prints `./ serverside-applied` for each object. +- `tekton:pipelinerun FILE` reads one document from the Kefka filesystem, + relative to the shell directory. It needs group `tekton.dev`, kind + `PipelineRun`, `metadata.generateName`, and no `metadata.name`. It sets the + `commit` parameter (required) and the `branch` parameter (when present) + from `OBJGIT_NEW_SHA` and `OBJGIT_BRANCH`. It adds the metadata, creates + the run, and prints the generated name and the namespace. +- Tests (write first): cover validation, substitution, metadata, the + namespace choice, the printed name, and two calls that make two creates. + +Test command: `go test ./internal/kube/` + +### Task 4: Wire into objgitd + +Files: `cmd/objgitd/main.go`, `cmd/objgitd/git_protocol.go`, +`cmd/objgitd/hooks.go`, `cmd/objgitd/shell.go`, `cmd/objgitd/kube_test.go`. + +- `-allow-kubernetes` flag. `daemon.kube *kube.Client`, nil when disabled. +- `newHookShell` takes the client and registers `kustomize` and the kube + commands. +- Tests (write first): a push hook runs + `kustomize build .tekton | kube:apply && tekton:pipelinerun .tekton/testrun.yaml` + against the fake API server, and the push output shows the run name. With no + client, the stub message reaches the pusher. + +Test command: `go test ./cmd/objgitd/` + +### Task 5: CI and Docker LFS hydration + +Files: `.github/workflows/go.yml`, `.github/workflows/docker.yml`, `Dockerfile`. + +- Check out with `lfs: true`. The Dockerfile fails the build when + `internal/kustomize/kustomize.wasm` does not start with the WASM magic. + +Test command: `docker build .` succeeds, and it fails against a pointer file. + +### Task 6: Example RBAC and cluster checks + +Files: `manifest/tekton-rbac/role.yaml`, `manifest/tekton-rbac/kustomization.yaml`. + +- A Role in `ci` with `create, patch` on `pipelines` and `tasks`, and `create` + on `pipelineruns`. A RoleBinding to ServiceAccount `objgitd` in `objgit`. +- Check in a `kind` cluster with stub Tekton CRDs: `kubectl auth can-i` for + each allowed verb, and a denial for an unrelated resource. Run `kube:apply` + and `tekton:pipelinerun` as the ServiceAccount against the real API server. + +### Task 7: Documentation + +Files: `docs/usage/kubernetes-hooks.md`, `docs/usage/hooks.md`, +`docs/architecture/hooks.md`, `AGENTS.md`, `README.md`. + +- Explain the hook recipe, the flag, the RBAC, and the downsides below. + +## Downsides + +- **Cluster authority.** When hooks and Kubernetes commands are both on, + anybody who can push a hook, or open the SSH shell, can use the pod + ServiceAccount's permissions. The example deployment permits anonymous + pushes when `ALLOW_PUSH=true`. +- **Push latency and duplicates.** Hooks run synchronously after the push is + accepted. A retry, or a second push of the same commit, creates another + PipelineRun. +- **Size.** The module adds about 23 MiB to the binary, and about 4 s of CPU on + first use. LFS adds a hydration step to checkout and CI. +- **`tekton:logs` is deferred.** The commit label makes a later lookup + possible. diff --git a/docs/usage/hooks.md b/docs/usage/hooks.md index 18e5031..0ca19cc 100644 --- a/docs/usage/hooks.md +++ b/docs/usage/hooks.md @@ -13,14 +13,46 @@ Hooks are off by default. Start the server with `-allow-hooks`: ./objgitd -bucket $BUCKET -allow-push -allow-hooks ``` -| Flag | Env | Default | Meaning | -| --------------- | -------------- | ------- | -------------------------------------------------------- | -| `-allow-hooks` | `ALLOW_HOOKS` | `false` | Run `.objgit/hooks/receive-pack` after a successful push | -| `-hook-timeout` | `HOOK_TIMEOUT` | `60s` | Wall-clock limit for a single hook run | +| Flag | Env | Default | Meaning | +| ----------------- | ---------------- | --------------------------------------- | ----------------------------------------------------------- | +| `-allow-hooks` | `ALLOW_HOOKS` | `false` | Run `.objgit/hooks/receive-pack` after a successful push | +| `-hook-timeout` | `HOOK_TIMEOUT` | `60s` | Wall-clock limit for a single hook run | +| `-wasm-path` | `WASM_PATH` | `/app/wasm/bin,/usr/libexec/objgit/bin` | Comma-separated directories of WASI programs for hooks | +| `-wasm-cache-dir` | `WASM_CACHE_DIR` | empty | Directory that keeps compiled WASI programs across restarts | `-allow-hooks` is independent of `-allow-push`, but a hook can only fire on a push, so in practice you want both. +### WASI programs + +Each `.wasm` file in a `-wasm-path` directory is a hook command. The command +name is the file name without `.wasm`, so `kustomize.wasm` is `kustomize`. +The image puts the programs from `bin/` in `/usr/libexec/objgit/bin`. + +`-wasm-path` works like `PATH`: + +- If two directories have a program with the same name, the first directory + wins. +- The daemon skips a directory that does not exist. +- A program replaces a kefka command with the same name, such as `jq`. +- A program cannot replace `kube:apply` or `tekton:pipelinerun`. + +The daemon lists the directories one time, at startup. It reads and compiles a +program on its first run. The compile of kustomize takes about 4 seconds, and +this time counts against `-hook-timeout`. With `-wasm-cache-dir`, the daemon +keeps the compiled programs on disk, and a run after a restart starts in +less than a second. + +To add a program to a deployment: + +1. Compile the program for WASI preview 1, for example with + `GOOS=wasip1 GOARCH=wasm go build`. +2. Put the `.wasm` file in `/app/wasm/bin`, for example from a volume mount. +3. Restart `objgitd`. + +A program sees the hook filesystem at `/`. It has no network. It gets these +environment variables: `PWD`, `HOME`, and `TMPDIR`. + ## When a hook runs - **On push only.** The hook is named after the git service that triggered it, @@ -48,12 +80,18 @@ virtual `bash` interpreter. **This is not a container, VM, or OS sandbox.** It i safe because of what it _cannot_ reach, not because of kernel isolation: - **No system binaries.** Kefka provides Go commands, WASM-backed uutils, and - the WASM programs `jo`, `jq`, `python3` (also `python`), `qjs`, and `rg`. Available + the WASM programs `jo`, `jq`, `python3` (also `python`), `qjs`, and `rg`. + The [WASI programs](#wasi-programs) in `-wasm-path` add to these. Available coreutils include `cat`, `ls`, `printf`, `head`, `tail`, `cut`, `sort`, `uniq`, `wc`, `tr`, `sha256sum`, `base64`, `base32`, `mkdir`, `cp`, `mv`, `rm`, `touch`, `date`, `sleep`, `seq`, and `expr`. There is no `git`, package manager, compiler, or `curl`. -- **No network.** +- **Kustomize and Kubernetes.** `kustomize` renders a bundle from `/src`. + With `-allow-kubernetes`, `kube:apply` and `tekton:pipelinerun` send + objects to the cluster that `objgitd` runs in. See + [kubernetes-hooks.md](kubernetes-hooks.md). +- **No network.** The only exception is the Kubernetes API, for the two + Kubernetes commands, when `-allow-kubernetes` is on. - **No host filesystem.** The only files a hook can see are the two mounts below. @@ -80,17 +118,17 @@ redirections like `echo x > out` — must target `/tmp`. Each run gets variables describing the branch that triggered it: -| Variable | Example | Notes | -| --------------------------- | -------------------------- | --------------------------------------------------------- | -| `OBJGIT_REPO` | `/myproject.git` | Repository path | -| `OBJGIT_SERVICE` | `receive-pack` | Always `receive-pack` | -| `OBJGIT_REF` | `refs/heads/main` | Full ref name | -| `OBJGIT_BRANCH` | `main` | Short branch name | -| `OBJGIT_OLD_SHA` | `0000…0000` | Previous tip; all zeros when the branch was created | -| `OBJGIT_NEW_SHA` | `f43417…` | New tip | -| `OBJGIT_ADDED_FILES_JSON` | `["new.txt"]` | Added paths, as a JSON array | -| `OBJGIT_CHANGED_FILES_JSON` | `["README.md"]` | Changed paths, as a JSON array | -| `OBJGIT_DELETED_FILES_JSON` | `["old.txt"]` | Deleted paths, as a JSON array | +| Variable | Example | Notes | +| --------------------------- | -------------------------- | -------------------------------------------------------- | +| `OBJGIT_REPO` | `/myproject.git` | Repository path | +| `OBJGIT_SERVICE` | `receive-pack` | Always `receive-pack` | +| `OBJGIT_REF` | `refs/heads/main` | Full ref name | +| `OBJGIT_BRANCH` | `main` | Short branch name | +| `OBJGIT_OLD_SHA` | `0000…0000` | Previous tip; all zeros when the branch was created | +| `OBJGIT_NEW_SHA` | `f43417…` | New tip | +| `OBJGIT_ADDED_FILES_JSON` | `["new.txt"]` | Added paths, as a JSON array | +| `OBJGIT_CHANGED_FILES_JSON` | `["README.md"]` | Changed paths, as a JSON array | +| `OBJGIT_DELETED_FILES_JSON` | `["old.txt"]` | Deleted paths, as a JSON array | | `OBJGIT_CHANGES_FILE` | `/tmp/objgit-changes.json` | File containing all three arrays in a single JSON object | The file lists describe the **net change of the branch tip** across the push. @@ -106,7 +144,7 @@ unpadded base64url encoding of its raw bytes. contains the same complete lists. For example: ```json -{"added":["new.txt"],"changed":["README.md"],"deleted":["old.txt"]} +{ "added": ["new.txt"], "changed": ["README.md"], "deleted": ["old.txt"] } ``` For compatibility with scripts written for stock git, the same information is @@ -220,5 +258,6 @@ Hook failure is logged, but it cannot undo the accepted push. scratch space. - No way to reject a push from a hook (it runs after the fact). - No system tooling, network, or arbitrary executables — only kefka's - registered commands. + registered commands. The Kubernetes commands reach only the in-cluster + API server. - Hook output reaches the pusher only when the client negotiated sideband. diff --git a/docs/usage/kubernetes-hooks.md b/docs/usage/kubernetes-hooks.md new file mode 100644 index 0000000..6f20683 --- /dev/null +++ b/docs/usage/kubernetes-hooks.md @@ -0,0 +1,253 @@ +# Kubernetes and Tekton hook commands + +A push hook can render a Kustomize bundle and send it to the Kubernetes +cluster that `objgitd` runs in. Then the hook can start a Tekton +PipelineRun for the pushed commit. Three commands in the hook shell do this +work: + +| Command | What it does | Needs | +| ------------------------- | ------------------------------------------------------------------------------- | -------------------------------------- | +| `kustomize` | Runs kustomize from `-wasm-path`. It reads `/src` and can write only to `/tmp`. | `-allow-hooks` | +| `kube:apply` | Reads YAML on stdin and applies each object with Server-Side Apply. | `-allow-hooks` and `-allow-kubernetes` | +| `tekton:pipelinerun FILE` | Creates a new PipelineRun from a template, set to the pushed commit. | `-allow-hooks` and `-allow-kubernetes` | + +The commands are also in the SSH `sh` shell, because that shell is the hook +sandbox. Read [hooks.md](hooks.md) first. It explains the sandbox, the +environment variables, and when a hook runs. + +## Security + +WARNING: Do not use `-allow-kubernetes` on a server that accepts pushes from +people you do not trust. + +A hook is a script from the pushed commit. When `-allow-kubernetes` is on, +anybody who can push a hook, or open the SSH shell, can use every permission +of the `objgitd` ServiceAccount. The example deployment in `manifest/` +accepts anonymous pushes when `ALLOW_PUSH=true`. Give the ServiceAccount +only the namespace and the verbs that your pipeline needs. + +## Turn the commands on + +1. Apply the example RBAC. It is separate from `manifest/`, because the + `namespace: objgit` field there moves the RoleBinding to `objgit`: + + ```text + kubectl apply -k manifest/tekton-rbac/ + ``` + +2. Set these values in the `objgitd-config` ConfigMap in + `manifest/kustomization.yaml`: + + ```text + ALLOW_HOOKS=true + ALLOW_KUBERNETES=true + ``` + +3. Apply the deployment again with `kubectl apply -k manifest/`. + +| Flag | Env | Default | Meaning | +| ------------------- | ------------------ | ------- | -------------------------------------------------------------- | +| `-allow-kubernetes` | `ALLOW_KUBERNETES` | `false` | Give `kube:apply` and `tekton:pipelinerun` the pod credentials | + +`objgitd` stops at startup in two conditions. The first is +`-allow-kubernetes` without `-allow-hooks`. The second is a pod without an +in-cluster ServiceAccount. The log line names the cause. + +Without `-allow-kubernetes`, the two Kubernetes commands exist, but they +stop with exit status 1 and this message: + +```text +kube:apply: Kubernetes commands are disabled; start objgitd with -allow-kubernetes to enable them +``` + +## Write the hook + +This example uses the layout of the +[Xe/x Tekton example](https://github.com/Xe/x/tree/master/.tekton): + +| File | Contents | +| ---------------------------- | -------------------------------------------------------------- | +| `.tekton/kustomization.yaml` | A Kustomize bundle with `namespace: ci` and the Pipeline file. | +| `.tekton/x.yaml` | The Pipeline. It declares `commit` and `branch` parameters. | +| `.tekton/testrun.yaml` | A PipelineRun template with `metadata.generateName`. | + +Put this script at `.objgit/hooks/receive-pack`: + +```bash +#!/usr/bin/env bash +# .objgit/hooks/receive-pack + +# Start a build only for pushes to main. +if [ "${OBJGIT_BRANCH}" != "main" ]; then + exit 0 +fi + +kustomize build /src/.tekton | kube:apply && tekton:pipelinerun /src/.tekton/testrun.yaml +``` + +The `&&` is necessary. If `kube:apply` fails, the hook does not create a run +against an old Pipeline. + +The push output then shows the result: + +```text +remote: pipeline.tekton.dev/xe-x-build-test serverside-applied +remote: pipelinerun.tekton.dev/x-m-plms2 created in namespace ci +``` + +The hook runs after Git updates the ref. A failed command shows its error to +the pusher and in the server log, but the push stays accepted. + +## Command reference + +### kustomize + +`kustomize` is kustomize compiled to WebAssembly. It takes the usual +arguments, for example `kustomize build .tekton`. Relative paths start at the +shell directory. + +- `/src` is read-only. Write output with `-o` only to a path in `/tmp`. +- The sandbox has no network. Remote bases, such as a Git URL in + `resources`, do not load. +- The first run in each `objgitd` process compiles the module. This takes + about 4 seconds of CPU, and it counts against `-hook-timeout`. With + `-wasm-cache-dir`, only the first run on a new disk compiles it. +- The hook timeout stops a long build. + +The image puts `bin/kustomize.wasm` from this repository in +`/usr/libexec/objgit/bin`. See [WASI programs](hooks.md#wasi-programs). The +module reports its version as `(devel)`, so its source revision is not known. + +### kube:apply + +`kube:apply` reads a stream of YAML documents on stdin and takes no +arguments. It applies the objects in stream order: + +1. It reads the full stream. Empty documents and comment-only documents are + skipped. +2. It makes sure that each object has `apiVersion`, `kind`, and + `metadata.name`. If one object does not, it stops before it sends a + request. +3. It sends each object as a Server-Side Apply request, with field manager + `objgitd`. It does not force conflicts. +4. It stops at the first failed request. + +An object without `metadata.namespace` goes to the namespace of the +`objgitd` ServiceAccount. `kube:apply` never deletes objects. An object that +you remove from the bundle stays in the cluster. + +`kube:apply` stops with exit status 1 for these conditions: + +| Condition | Message contains | +| ----------------------------------------------- | ---------------------------------------------- | +| stdin has no objects | `no objects on standard input` | +| A document is not valid YAML | `document N` | +| An object has no name, kind, or apiVersion | `object N` | +| The cluster does not serve the kind | `no resource of kind` | +| Another field manager owns a field | `Apply failed with` | +| The ServiceAccount does not have the permission | `is forbidden: User "system:serviceaccount:…"` | + +A conflict means that another tool, such as `kubectl apply`, owns the field. +To correct it, remove the field from the bundle, or from the other tool. + +### tekton:pipelinerun + +`tekton:pipelinerun FILE` reads one PipelineRun from `FILE` in the hook +filesystem. A relative path starts at the shell directory. The template must +obey these rules: + +- `apiVersion` is in the `tekton.dev` group, and `kind` is `PipelineRun`. +- `metadata.generateName` is set, and `metadata.name` is not set. The API + server adds a random suffix to the name. +- `spec.params` has a parameter named `commit`. + +The command changes the template before it sends it. The values come from +the update that `objgitd` runs the hook for. They are the same values as the +`OBJGIT_*` variables, but a script cannot change them: + +| Field | New value | +| ---------------------------------------------- | ------------------------------------- | +| The `commit` parameter | The commit, as in `OBJGIT_NEW_SHA` | +| The `branch` parameter, if the template has it | The branch, as in `OBJGIT_BRANCH` | +| Annotation `objgit.tigrisdata.com/repo` | The repository, as in `OBJGIT_REPO` | +| Annotation `objgit.tigrisdata.com/ref` | The ref, as in `OBJGIT_REF` | +| Annotation `objgit.tigrisdata.com/commit` | The commit | +| Label `objgit.tigrisdata.com/commit-prefix` | The first 12 characters of the commit | + +In the SSH `sh` shell on a tag or a commit, there is no branch. The `branch` +parameter then keeps its template value. + +Other parameters and fields do not change. Then the command sends a create +request and prints the generated name and the namespace. A template without +`metadata.namespace` goes to the namespace of the ServiceAccount. + +Each call creates a new run. A second push of the same commit, or a retried +push, therefore creates a second run. To find the runs of one commit, use the +label: + +```text +kubectl -n ci get pipelineruns -l objgit.tigrisdata.com/commit-prefix=658195e83ee5 +``` + +## Permissions + +The example Role in `manifest/tekton-rbac/role.yaml` gives the ServiceAccount +`objgit/objgitd` these permissions in namespace `ci`: + +| Resource | Verbs | Used by | +| ------------------------------------------ | ----------------- | -------------------- | +| `pipelines.tekton.dev`, `tasks.tekton.dev` | `create`, `patch` | `kube:apply` | +| `pipelineruns.tekton.dev` | `create` | `tekton:pipelinerun` | + +Server-Side Apply needs `patch` for each object. For an object that does not +exist yet, it also needs `create`. The Role therefore does not let +`kube:apply` send PipelineRuns, because it has no `patch` on them. + +Other kinds, or other namespaces, need more RBAC. Without it, the API server +refuses the request with 403 Forbidden, and the hook shows that message. + +The Role does not cover these items: + +- The ServiceAccount in `spec.taskRunTemplate.serviceAccountName` of a run. + Tekton runs the tasks as that account, which is a different identity. +- The Tasks, workspace storage, and registry Secrets that the pipeline + uses. The Xe/x pipeline, for example, needs the `git-clone-naive`, `ko`, + and `kaniko` Tasks, the `go-mod-cache` claim, and the `ghcr` Secret. Create + them before the first run. + +## Limits + +- Hooks run synchronously. A slow API server holds the push connection open + until `-hook-timeout`. +- There is no `tekton:logs` command yet. Use the commit label to find a run, + then use `tkn` or `kubectl`. +- The commands use only the in-cluster ServiceAccount. They do not read a + kubeconfig. + +## Test against a real cluster + +`TestCluster` in `internal/kube` sends real requests. It runs only when three +variables are set. For a local [kind](https://kind.sigs.k8s.io/) cluster: + +1. Create the cluster, the `ci` and `objgit` namespaces, and the `objgitd` + ServiceAccount. +2. Install CRDs for `pipelines`, `tasks`, and `pipelineruns` in the + `tekton.dev` group. They must serve `v1`. +3. Apply `manifest/tekton-rbac/`. +4. Set the variables, then run the test: + + ```text + export OBJGIT_TEST_KUBE_HOST=https://127.0.0.1:6443 + export OBJGIT_TEST_KUBE_CA=/path/to/ca.crt + export OBJGIT_TEST_KUBE_TOKEN="$(kubectl -n objgit create token objgitd)" + go test -run TestCluster ./internal/kube/ + ``` + +To make sure that the Role gives only what it must, use `kubectl auth can-i`: + +```text +kubectl auth can-i patch pipelines.tekton.dev -n ci --as=system:serviceaccount:objgit:objgitd +kubectl auth can-i create configmaps -n ci --as=system:serviceaccount:objgit:objgitd +``` + +The first command prints `yes`, and the second prints `no`. diff --git a/docs/usage/kubernetes.md b/docs/usage/kubernetes.md index 010713e..2bdb737 100644 --- a/docs/usage/kubernetes.md +++ b/docs/usage/kubernetes.md @@ -105,6 +105,10 @@ git clone ssh://git@objgit.objgit.svc.cluster.local/org/repo.git Scrape metrics from `objgit-metrics.objgit.svc.cluster.local:9090/metrics`. The same listener serves pprof under `/debug/pprof/`. +Hooks can apply Kustomize bundles and start Tekton PipelineRuns in the same +cluster. This needs `ALLOW_KUBERNETES=true` and the RBAC in +`manifest/tekton-rbac/`. See [kubernetes-hooks.md](kubernetes-hooks.md). + ## Health check The metrics listener answers `GET /healthz` with `200 ok`. The smart-HTTP mux @@ -126,6 +130,10 @@ The startup, liveness, and readiness probes in the manifest all send the SSH host key live in the Tigris bucket. On startup, the daemon sweeps cache directories that an earlier run left behind, so a crash does not leak disk space across restarts. +- The **WASI compilation cache** (`WASM_CACHE_DIR`) is at + `/var/cache/objgit/wasm` on the same claim. The compiled kustomize is about + 100 MB. With this cache, the first hook after a restart does not compile + kustomize again. - **Upload scratch data** is separate from the pack cache. The tigris storer stages pack writes to the OS temp directory through `os.CreateTemp`. The root filesystem of the container is read-only, so the OS temp directory diff --git a/go.mod b/go.mod index 127b0a2..a64a7ed 100644 --- a/go.mod +++ b/go.mod @@ -20,6 +20,7 @@ require ( golang.org/x/sync v0.20.0 golang.org/x/term v0.43.0 mvdan.cc/sh/v3 v3.13.1 + sigs.k8s.io/yaml v1.6.0 ) require ( @@ -66,7 +67,7 @@ require ( github.com/prometheus/common v0.66.1 // indirect github.com/prometheus/procfs v0.16.1 // indirect github.com/sergi/go-diff v1.4.0 // indirect - github.com/tetratelabs/wazero v1.11.0 // indirect + github.com/tetratelabs/wazero v1.11.0 go.yaml.in/yaml/v2 v2.4.2 // indirect golang.org/x/exp/typeparams v0.0.0-20231108232855-2478ac86f678 // indirect golang.org/x/mod v0.36.0 // indirect diff --git a/go.sum b/go.sum index ad7e44c..b796b47 100644 --- a/go.sum +++ b/go.sum @@ -136,6 +136,8 @@ go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto= go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE= go.yaml.in/yaml/v2 v2.4.2 h1:DzmwEr2rDGHl7lsFgAHxmNz/1NlQ7xLIrlN2h5d1eGI= go.yaml.in/yaml/v2 v2.4.2/go.mod h1:081UH+NErpNdqlCXm3TtEran0rJZGxAYx9hb/ELlsPU= +go.yaml.in/yaml/v3 v3.0.3 h1:bXOww4E/J3f66rav3pX3m8w6jDE4knZjGOw8b5Y6iNE= +go.yaml.in/yaml/v3 v3.0.3/go.mod h1:tBHosrYAkRZjRAOREWbDnBXUf08JOwYq++0QNwQiWzI= golang.org/x/crypto v0.51.0 h1:IBPXwPfKxY7cWQZ38ZCIRPI50YLeevDLlLnyC5wRGTI= golang.org/x/crypto v0.51.0/go.mod h1:8AdwkbraGNABw2kOX6YFPs3WM22XqI4EXEd8g+x7Oc8= golang.org/x/exp/typeparams v0.0.0-20231108232855-2478ac86f678 h1:1P7xPZEwZMoBoz0Yze5Nx2/4pxj6nw9ZqHWXqP0iRgQ= @@ -172,3 +174,5 @@ honnef.co/go/tools v0.7.0 h1:w6WUp1VbkqPEgLz4rkBzH/CSU6HkoqNLp6GstyTx3lU= honnef.co/go/tools v0.7.0/go.mod h1:pm29oPxeP3P82ISxZDgIYeOaf9ta6Pi0EWvCFoLG2vc= mvdan.cc/sh/v3 v3.13.1 h1:DP3TfgZhDkT7lerUdnp6PTGKyxxzz6T+cOlY/xEvfWk= mvdan.cc/sh/v3 v3.13.1/go.mod h1:lXJ8SexMvEVcHCoDvAGLZgFJ9Wsm2sulmoNEXGhYZD0= +sigs.k8s.io/yaml v1.6.0 h1:G8fkbMSAFqgEFgh4b1wmtzDnioxFCUgTZhlbj5P9QYs= +sigs.k8s.io/yaml v1.6.0/go.mod h1:796bPqUfzR/0jLAl6XjHl3Ck7MiyVv8dbTdyT3/pMf4= diff --git a/internal/kube/client.go b/internal/kube/client.go new file mode 100644 index 0000000..1d88b21 --- /dev/null +++ b/internal/kube/client.go @@ -0,0 +1,421 @@ +// Package kube is the small Kubernetes API client behind the kube:apply and +// tekton:pipelinerun hook commands. It covers in-cluster configuration, +// discovery of one apiVersion at a time, Server-Side Apply, and create. It +// is not client-go: it has no typed objects, no watch, and no cache beyond +// one command run. +package kube + +import ( + "bytes" + "context" + "crypto/tls" + "crypto/x509" + "encoding/json" + "errors" + "fmt" + "io" + "net" + "net/http" + "net/url" + "os" + "path/filepath" + "regexp" + "strings" + "sync" + "time" +) + +// FieldManager is the Server-Side Apply field manager, and the create field +// manager, of every request objgitd sends. +const FieldManager = "objgitd" + +// serviceAccountDir is where Kubernetes mounts a pod's ServiceAccount +// credentials. +const serviceAccountDir = "/var/run/secrets/kubernetes.io/serviceaccount" + +// ErrNotInCluster means the process does not run in a Kubernetes pod. +var ErrNotInCluster = errors.New("kube: KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT are not set; objgitd is not running in a Kubernetes pod") + +// Object is one Kubernetes object in its JSON form. +type Object map[string]any + +// APIVersion returns the object's apiVersion, or "". +func (o Object) APIVersion() string { s, _ := o["apiVersion"].(string); return s } + +// Kind returns the object's kind, or "". +func (o Object) Kind() string { s, _ := o["kind"].(string); return s } + +// Name returns metadata.name, or "". +func (o Object) Name() string { return o.meta("name") } + +// GenerateName returns metadata.generateName, or "". +func (o Object) GenerateName() string { return o.meta("generateName") } + +// Namespace returns metadata.namespace, or "". +func (o Object) Namespace() string { return o.meta("namespace") } + +func (o Object) meta(key string) string { + m, _ := o["metadata"].(map[string]any) + s, _ := m[key].(string) + return s +} + +// Resource is a discovered API resource. +type Resource struct { + Group, Version string + Name string // the plural resource name, such as "pipelineruns" + Kind string + Namespaced bool +} + +// String returns the kubectl form of the resource: kind in lower case, then +// the group when it is not the core group, such as "pipeline.tekton.dev". +func (r Resource) String() string { + s := strings.ToLower(r.Kind) + if r.Group != "" { + s += "." + r.Group + } + return s +} + +// Result is the object the API server returned, and its resource. +type Result struct { + Resource Resource + Object Object +} + +// APIError is a failed API request. Message is the server's Status message, +// which names the resource and, for a denial, the missing permission. +type APIError struct { + Code int // the HTTP status code + Reason string // the Status reason, such as "Forbidden" or "Conflict" + Message string +} + +func (e *APIError) Error() string { + if e.Message == "" { + return fmt.Sprintf("kubernetes API: %d %s", e.Code, http.StatusText(e.Code)) + } + return e.Message +} + +// Config configures a Client. +type Config struct { + Host string // the API server base URL, such as https://10.0.0.1:443 + HTTPClient *http.Client // nil uses http.DefaultClient + // Token returns the bearer token for one request. Nil sends no token. + Token func() (string, error) + // Namespace is the namespace of an object without metadata.namespace. + Namespace string +} + +// Client talks to one Kubernetes API server. It is safe for concurrent use. +type Client struct { + cfg Config +} + +// New returns a Client for cfg. +func New(cfg Config) *Client { + if cfg.HTTPClient == nil { + cfg.HTTPClient = http.DefaultClient + } + cfg.Host = strings.TrimSuffix(cfg.Host, "/") + return &Client{cfg: cfg} +} + +// InCluster returns a Client that uses the pod's ServiceAccount: the API +// server address from the environment, the mounted CA, and the mounted +// token. The token file is read for each request, because bound tokens +// rotate. The default namespace is the ServiceAccount's namespace. +func InCluster() (*Client, error) { + return inCluster(serviceAccountDir, os.Getenv) +} + +func inCluster(dir string, getenv func(string) string) (*Client, error) { + host, port := getenv("KUBERNETES_SERVICE_HOST"), getenv("KUBERNETES_SERVICE_PORT") + if host == "" || port == "" { + return nil, ErrNotInCluster + } + caPEM, err := os.ReadFile(filepath.Join(dir, "ca.crt")) + if err != nil { + return nil, fmt.Errorf("kube: read ServiceAccount CA: %w", err) + } + pool := x509.NewCertPool() + if !pool.AppendCertsFromPEM(caPEM) { + return nil, fmt.Errorf("kube: %s holds no PEM certificate", filepath.Join(dir, "ca.crt")) + } + ns, err := os.ReadFile(filepath.Join(dir, "namespace")) + if err != nil { + return nil, fmt.Errorf("kube: read ServiceAccount namespace: %w", err) + } + tokenFile := filepath.Join(dir, "token") + if _, err := os.Stat(tokenFile); err != nil { + return nil, fmt.Errorf("kube: ServiceAccount token: %w", err) + } + + transport := http.DefaultTransport.(*http.Transport).Clone() + transport.TLSClientConfig = &tls.Config{RootCAs: pool, MinVersion: tls.VersionTLS12} + return New(Config{ + Host: "https://" + net.JoinHostPort(host, port), + HTTPClient: &http.Client{Transport: transport, Timeout: 30 * time.Second}, + Token: func() (string, error) { + data, err := os.ReadFile(tokenFile) + if err != nil { + return "", fmt.Errorf("kube: read ServiceAccount token: %w", err) + } + return strings.TrimSpace(string(data)), nil + }, + Namespace: strings.TrimSpace(string(ns)), + }), nil +} + +// Namespace returns the namespace of an object without metadata.namespace. +func (c *Client) Namespace() string { return c.cfg.Namespace } + +// Session returns a view of c that caches discovery until it is dropped. A +// command takes one per run, so a stream of objects discovers each +// apiVersion once, and a CRD installed later is still found by the next run. +func (c *Client) Session() *Session { + return &Session{c: c, lists: map[string][]Resource{}} +} + +// Apply is c.Session().Apply. +func (c *Client) Apply(ctx context.Context, obj Object) (Result, error) { + return c.Session().Apply(ctx, obj) +} + +// Create is c.Session().Create. +func (c *Client) Create(ctx context.Context, obj Object) (Result, error) { + return c.Session().Create(ctx, obj) +} + +// Session is a Client with a discovery cache. It is safe for concurrent use. +type Session struct { + c *Client + + mu sync.Mutex + lists map[string][]Resource // by apiVersion +} + +// Apply sends obj as a Server-Side Apply patch with field manager objgitd. +// Conflicts are not forced: a field that another manager owns fails the +// request with a 409 APIError. The server creates a missing object, which +// needs the create verb; an existing object needs patch. +func (s *Session) Apply(ctx context.Context, obj Object) (Result, error) { + if err := check(obj, false); err != nil { + return Result{}, err + } + res, err := s.resource(ctx, obj.APIVersion(), obj.Kind()) + if err != nil { + return Result{}, err + } + u := s.c.cfg.Host + collectionPath(res, s.namespace(res, obj)) + "/" + url.PathEscape(obj.Name()) + + "?" + url.Values{"fieldManager": {FieldManager}, "force": {"false"}}.Encode() + out, err := s.c.send(ctx, http.MethodPatch, u, "application/apply-patch+yaml", obj) + return Result{Resource: res, Object: out}, err +} + +// Create sends obj as a create request with field manager objgitd. obj may +// set metadata.generateName instead of metadata.name; the server then picks +// the name, and Result.Object carries it. +func (s *Session) Create(ctx context.Context, obj Object) (Result, error) { + if err := check(obj, true); err != nil { + return Result{}, err + } + res, err := s.resource(ctx, obj.APIVersion(), obj.Kind()) + if err != nil { + return Result{}, err + } + u := s.c.cfg.Host + collectionPath(res, s.namespace(res, obj)) + + "?" + url.Values{"fieldManager": {FieldManager}}.Encode() + out, err := s.c.send(ctx, http.MethodPost, u, "application/json", obj) + return Result{Resource: res, Object: out}, err +} + +// check reports the first identifying field obj lacks. +func check(obj Object, generateNameOK bool) error { + switch { + case obj.APIVersion() == "": + return errors.New("object has no apiVersion") + case obj.Kind() == "": + return fmt.Errorf("%s object has no kind", obj.APIVersion()) + case obj.Name() == "" && !generateNameOK: + return fmt.Errorf("%s object has no metadata.name", obj.Kind()) + case obj.Name() == "" && obj.GenerateName() == "": + return fmt.Errorf("%s object has no metadata.name or metadata.generateName", obj.Kind()) + } + return nil +} + +func (s *Session) namespace(res Resource, obj Object) string { + if !res.Namespaced { + return "" + } + if ns := obj.Namespace(); ns != "" { + return ns + } + return s.c.cfg.Namespace +} + +// collectionPath returns the URL path of res in namespace ns ("" for a +// cluster-scoped resource). +func collectionPath(res Resource, ns string) string { + p := "/api/" + res.Version + if res.Group != "" { + p = "/apis/" + res.Group + "/" + res.Version + } + if ns != "" { + p += "/namespaces/" + url.PathEscape(ns) + } + return p + "/" + res.Name +} + +// resource finds kind in apiVersion through discovery. +func (s *Session) resource(ctx context.Context, apiVersion, kind string) (Resource, error) { + s.mu.Lock() + list, ok := s.lists[apiVersion] + s.mu.Unlock() + if !ok { + var err error + if list, err = s.c.discover(ctx, apiVersion); err != nil { + return Resource{}, err + } + s.mu.Lock() + s.lists[apiVersion] = list + s.mu.Unlock() + } + for _, r := range list { + if r.Kind == kind { + return r, nil + } + } + return Resource{}, fmt.Errorf("no resource of kind %q in %s; is its CustomResourceDefinition installed?", kind, apiVersion) +} + +// Kubernetes names API groups as DNS-1123 subdomains, and versions as short +// lower-case alphanumerics such as v1 or v1beta1. +var ( + groupPattern = regexp.MustCompile(`^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$`) + versionPattern = regexp.MustCompile(`^[a-z0-9]+$`) +) + +// splitAPIVersion splits apiVersion into its group ("" for the core group) +// and version. It refuses anything else, because both parts go into request +// paths: an escaped slash or a "?" there would reach a different API path. +func splitAPIVersion(apiVersion string) (group, version string, err error) { + group, version, grouped := strings.Cut(apiVersion, "/") + if !grouped { + group, version = "", apiVersion + } + if (grouped && (len(group) > 253 || !groupPattern.MatchString(group))) || !versionPattern.MatchString(version) { + return "", "", fmt.Errorf("invalid apiVersion %q; want group/version, such as tekton.dev/v1, or v1", apiVersion) + } + return group, version, nil +} + +// discover lists the top-level resources of apiVersion. Subresources, such +// as pipelineruns/status, are left out. +func (c *Client) discover(ctx context.Context, apiVersion string) ([]Resource, error) { + group, version, err := splitAPIVersion(apiVersion) + if err != nil { + return nil, err + } + p := "/api/" + version + if group != "" { + p = "/apis/" + group + "/" + version + } + + var list struct { + Resources []struct { + Name string `json:"name"` + Kind string `json:"kind"` + Namespaced bool `json:"namespaced"` + } `json:"resources"` + } + data, err := c.do(ctx, http.MethodGet, c.cfg.Host+p, "", nil) + if apiErr, ok := errors.AsType[*APIError](err); ok && apiErr.Code == http.StatusNotFound { + return nil, fmt.Errorf("%s is not served by the Kubernetes API server", apiVersion) + } + if err != nil { + return nil, fmt.Errorf("discover %s: %w", apiVersion, err) + } + if err := json.Unmarshal(data, &list); err != nil { + return nil, fmt.Errorf("discover %s: %w", apiVersion, err) + } + var out []Resource + for _, r := range list.Resources { + if strings.Contains(r.Name, "/") { + continue + } + out = append(out, Resource{Group: group, Version: version, Name: r.Name, Kind: r.Kind, Namespaced: r.Namespaced}) + } + return out, nil +} + +// send encodes obj as JSON, sends it, and decodes the returned object. JSON +// is valid YAML, so it also serves as an apply-patch+yaml body. +func (c *Client) send(ctx context.Context, method, u, contentType string, obj Object) (Object, error) { + body, err := json.Marshal(obj) + if err != nil { + return nil, fmt.Errorf("encode %s: %w", obj.Kind(), err) + } + data, err := c.do(ctx, method, u, contentType, body) + if err != nil { + return nil, err + } + var out Object + dec := json.NewDecoder(bytes.NewReader(data)) + dec.UseNumber() + if err := dec.Decode(&out); err != nil { + return nil, fmt.Errorf("decode API response: %w", err) + } + return out, nil +} + +// maxResponse caps an API response body. Objects are at most about 1.5 MiB +// in etcd, and a discovery list is far smaller. +const maxResponse = 8 << 20 + +func (c *Client) do(ctx context.Context, method, u, contentType string, body []byte) ([]byte, error) { + var r io.Reader + if body != nil { + r = bytes.NewReader(body) + } + req, err := http.NewRequestWithContext(ctx, method, u, r) + if err != nil { + return nil, err + } + req.Header.Set("Accept", "application/json") + req.Header.Set("User-Agent", "objgitd") + if contentType != "" { + req.Header.Set("Content-Type", contentType) + } + if c.cfg.Token != nil { + token, err := c.cfg.Token() + if err != nil { + return nil, err + } + req.Header.Set("Authorization", "Bearer "+token) + } + resp, err := c.cfg.HTTPClient.Do(req) + if err != nil { + return nil, err + } + defer resp.Body.Close() + data, err := io.ReadAll(io.LimitReader(resp.Body, maxResponse)) + if err != nil { + return nil, err + } + if resp.StatusCode >= 200 && resp.StatusCode < 300 { + return data, nil + } + apiErr := &APIError{Code: resp.StatusCode} + var status struct { + Reason string `json:"reason"` + Message string `json:"message"` + } + if json.Unmarshal(data, &status) == nil { + apiErr.Reason, apiErr.Message = status.Reason, status.Message + } + return nil, apiErr +} diff --git a/internal/kube/client_test.go b/internal/kube/client_test.go new file mode 100644 index 0000000..44a3d86 --- /dev/null +++ b/internal/kube/client_test.go @@ -0,0 +1,298 @@ +package kube + +import ( + "context" + "encoding/pem" + "errors" + "net" + "net/http" + "net/url" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/tigrisdata/objgit/internal/kube/kubetest" +) + +// testClient returns a client for srv whose default namespace is "objgit". +func testClient(srv *kubetest.Server) *Client { + return New(Config{ + Host: srv.URL, + HTTPClient: srv.Client(), + Token: func() (string, error) { return srv.Token(), nil }, + Namespace: "objgit", + }) +} + +func obj(apiVersion, kind, namespace, name string) Object { + meta := map[string]any{"name": name} + if namespace != "" { + meta["namespace"] = namespace + } + return Object{"apiVersion": apiVersion, "kind": kind, "metadata": meta, "spec": map[string]any{"x": "y"}} +} + +func TestApply(t *testing.T) { + tests := []struct { + name string + setup func(*kubetest.Server) + obj Object + wantPath string + wantCode int // APIError status code; 0 means success + wantErr string // error substring when the error is not an APIError + }{ + { + name: "namespaced object without a namespace uses the default", + obj: obj("v1", "ConfigMap", "", "greeting"), + wantPath: "/api/v1/namespaces/objgit/configmaps/greeting", + }, + { + name: "metadata.namespace wins over the default", + obj: obj("tekton.dev/v1beta1", "Pipeline", "ci", "build"), + wantPath: "/apis/tekton.dev/v1beta1/namespaces/ci/pipelines/build", + }, + { + name: "cluster-scoped object has no namespace in its path", + obj: obj("v1", "Namespace", "", "ci"), + wantPath: "/api/v1/namespaces/ci", + }, + { + name: "re-apply of an object objgitd owns", + setup: func(s *kubetest.Server) { + s.Put("/apis/tekton.dev/v1/namespaces/ci/pipelines/build", FieldManager, map[string]any{}) + }, + obj: obj("tekton.dev/v1", "Pipeline", "ci", "build"), + wantPath: "/apis/tekton.dev/v1/namespaces/ci/pipelines/build", + }, + { + name: "field conflict with another manager", + setup: func(s *kubetest.Server) { + s.Put("/apis/tekton.dev/v1/namespaces/ci/pipelines/build", "kubectl", map[string]any{}) + }, + obj: obj("tekton.dev/v1", "Pipeline", "ci", "build"), + wantPath: "/apis/tekton.dev/v1/namespaces/ci/pipelines/build", + wantCode: http.StatusConflict, + }, + { + name: "RBAC denial", + setup: func(s *kubetest.Server) { s.Deny("create", "pipelines") }, + obj: obj("tekton.dev/v1", "Pipeline", "ci", "build"), + wantPath: "/apis/tekton.dev/v1/namespaces/ci/pipelines/build", + wantCode: http.StatusForbidden, + }, + { + // A real API server checks patch for every apply, and create as + // well when the object is new. + name: "RBAC denial of patch on a new object", + setup: func(s *kubetest.Server) { s.Deny("patch", "pipelines") }, + obj: obj("tekton.dev/v1", "Pipeline", "ci", "build"), + wantPath: "/apis/tekton.dev/v1/namespaces/ci/pipelines/build", + wantCode: http.StatusForbidden, + }, + { + name: "unknown kind", + obj: obj("v1", "Widget", "", "w"), + wantErr: `no resource of kind "Widget" in v1`, + }, + { + name: "apiVersion the server does not serve", + obj: obj("example.com/v1", "Widget", "", "w"), + wantErr: "example.com/v1 is not served", + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + srv := kubetest.New(t) + if tt.setup != nil { + tt.setup(srv) + } + res, err := testClient(srv).Apply(context.Background(), tt.obj) + switch { + case tt.wantErr != "": + if err == nil || !strings.Contains(err.Error(), tt.wantErr) { + t.Fatalf("err = %v, want it to contain %q", err, tt.wantErr) + } + if n := len(srv.Requests()); n != 0 { + t.Errorf("%d mutating requests sent, want 0", n) + } + return + case tt.wantCode != 0: + apiErr, ok := errors.AsType[*APIError](err) + if !ok || apiErr.Code != tt.wantCode { + t.Fatalf("err = %v, want an APIError with code %d", err, tt.wantCode) + } + if apiErr.Message == "" || !strings.Contains(err.Error(), apiErr.Message) { + t.Errorf("err = %q, want it to carry the API message %q", err, apiErr.Message) + } + case err != nil: + t.Fatalf("Apply: %v", err) + default: + if res.Object.Name() != tt.obj.Name() { + t.Errorf("result name = %q, want %q", res.Object.Name(), tt.obj.Name()) + } + if _, ok := srv.Object(tt.wantPath); !ok { + t.Errorf("no object stored at %s", tt.wantPath) + } + } + + reqs := srv.Requests() + if len(reqs) != 1 { + t.Fatalf("%d mutating requests, want 1", len(reqs)) + } + r := reqs[0] + if r.Method != http.MethodPatch || r.Path != tt.wantPath { + t.Errorf("request = %s %s, want PATCH %s", r.Method, r.Path, tt.wantPath) + } + if r.ContentType != "application/apply-patch+yaml" { + t.Errorf("Content-Type = %q, want application/apply-patch+yaml", r.ContentType) + } + if r.Query["fieldManager"] != "objgitd" || r.Query["force"] != "false" { + t.Errorf("query = %v, want fieldManager=objgitd and force=false", r.Query) + } + }) + } +} + +func TestApplyRejectsMalformedAPIVersion(t *testing.T) { + tests := []struct { + name string + apiVersion string + }{ + {"escaped slashes reach another path", "v1%2Fnamespaces%2Fci%2Fsecrets%2Fx"}, + {"escaped slashes in a group", "tekton.dev/v1%2Fnamespaces%2Fci"}, + {"a query string", "tekton.dev/v1?dryRun=All&x="}, + {"a fragment", "v1#x"}, + {"too many slashes", "tekton.dev/v1/pipelines"}, + {"an empty group", "/v1"}, + {"an empty version", "tekton.dev/"}, + {"an upper-case group", "Tekton.dev/v1"}, + {"a dot segment", "../v1"}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + srv := kubetest.New(t) + _, err := testClient(srv).Apply(context.Background(), obj(tt.apiVersion, "ConfigMap", "", "a")) + if err == nil || !strings.Contains(err.Error(), "invalid apiVersion") { + t.Fatalf("err = %v, want an invalid apiVersion error", err) + } + if paths := srv.Paths(); len(paths) != 0 { + t.Errorf("requests sent for a malformed apiVersion: %q", paths) + } + }) + } +} + +func TestApplyRejectsIncompleteObjects(t *testing.T) { + tests := []struct { + name string + obj Object + want string + }{ + {"no apiVersion", Object{"kind": "ConfigMap", "metadata": map[string]any{"name": "a"}}, "apiVersion"}, + {"no kind", Object{"apiVersion": "v1", "metadata": map[string]any{"name": "a"}}, "kind"}, + {"no name", Object{"apiVersion": "v1", "kind": "ConfigMap"}, "metadata.name"}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + srv := kubetest.New(t) + _, err := testClient(srv).Apply(context.Background(), tt.obj) + if err == nil || !strings.Contains(err.Error(), tt.want) { + t.Fatalf("err = %v, want it to name %s", err, tt.want) + } + }) + } +} + +func TestCreate(t *testing.T) { + srv := kubetest.New(t) + run := Object{ + "apiVersion": "tekton.dev/v1", + "kind": "PipelineRun", + "metadata": map[string]any{"generateName": "x-m-", "namespace": "ci"}, + } + c := testClient(srv) + var names []string + for range 2 { + res, err := c.Create(context.Background(), run) + if err != nil { + t.Fatalf("Create: %v", err) + } + if !strings.HasPrefix(res.Object.Name(), "x-m-") || res.Object.Namespace() != "ci" { + t.Errorf("created %s/%s, want ci/x-m-*", res.Object.Namespace(), res.Object.Name()) + } + names = append(names, res.Object.Name()) + } + if names[0] == names[1] { + t.Errorf("two creates returned the same name %q", names[0]) + } + for _, r := range srv.Requests() { + if r.Method != http.MethodPost || r.Path != "/apis/tekton.dev/v1/namespaces/ci/pipelineruns" { + t.Errorf("request = %s %s, want POST to the pipelineruns collection", r.Method, r.Path) + } + if r.ContentType != "application/json" || r.Query["fieldManager"] != "objgitd" { + t.Errorf("Content-Type %q, query %v", r.ContentType, r.Query) + } + } + + srv.Deny("create", "pipelineruns") + _, err := c.Create(context.Background(), run) + if apiErr, ok := errors.AsType[*APIError](err); !ok || apiErr.Code != http.StatusForbidden { + t.Fatalf("err = %v, want a 403 APIError", err) + } + if !strings.Contains(err.Error(), "forbidden") { + t.Errorf("err = %q, want the forbidden message", err) + } +} + +func TestCreateNeedsANameOrGenerateName(t *testing.T) { + srv := kubetest.New(t) + _, err := testClient(srv).Create(context.Background(), Object{"apiVersion": "tekton.dev/v1", "kind": "PipelineRun"}) + if err == nil || !strings.Contains(err.Error(), "generateName") { + t.Fatalf("err = %v, want it to name metadata.generateName", err) + } +} + +func TestInCluster(t *testing.T) { + srv := kubetest.New(t) + dir := t.TempDir() + write := func(name, data string) { + t.Helper() + if err := os.WriteFile(filepath.Join(dir, name), []byte(data), 0o600); err != nil { + t.Fatal(err) + } + } + write("ca.crt", string(pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: srv.Certificate().Raw}))) + write("token", srv.Token()+"\n") + write("namespace", "objgit\n") + + u, err := url.Parse(srv.URL) + if err != nil { + t.Fatal(err) + } + host, port, _ := net.SplitHostPort(u.Host) + env := map[string]string{"KUBERNETES_SERVICE_HOST": host, "KUBERNETES_SERVICE_PORT": port} + + c, err := inCluster(dir, func(k string) string { return env[k] }) + if err != nil { + t.Fatalf("inCluster: %v", err) + } + if c.Namespace() != "objgit" { + t.Errorf("Namespace() = %q, want objgit", c.Namespace()) + } + if _, err := c.Apply(context.Background(), obj("v1", "ConfigMap", "", "a")); err != nil { + t.Fatalf("Apply with in-cluster config: %v", err) + } + + // Bound ServiceAccount tokens rotate, so each request reads the file. + srv.SetToken("rotated") + write("token", "rotated\n") + if _, err := c.Apply(context.Background(), obj("v1", "ConfigMap", "", "b")); err != nil { + t.Fatalf("Apply after token rotation: %v", err) + } + + if _, err := inCluster(dir, func(string) string { return "" }); !errors.Is(err, ErrNotInCluster) { + t.Errorf("inCluster without service env = %v, want ErrNotInCluster", err) + } +} diff --git a/internal/kube/cluster_test.go b/internal/kube/cluster_test.go new file mode 100644 index 0000000..b45d6e2 --- /dev/null +++ b/internal/kube/cluster_test.go @@ -0,0 +1,135 @@ +package kube + +import ( + "context" + "crypto/tls" + "crypto/x509" + "errors" + "net/http" + "net/url" + "os" + "strconv" + "strings" + "testing" + "time" +) + +// testCluster returns a client for a real cluster, or skips. Set +// OBJGIT_TEST_KUBE_HOST (the API server URL), OBJGIT_TEST_KUBE_CA (a CA file), +// and OBJGIT_TEST_KUBE_TOKEN (a token for ServiceAccount objgit/objgitd with +// manifest/tekton-rbac applied, from kubectl create token). The cluster needs +// the Tekton CRDs and the ci namespace. See docs/usage/kubernetes-hooks.md. +func testCluster(t *testing.T) *Client { + t.Helper() + host, caFile, token := os.Getenv("OBJGIT_TEST_KUBE_HOST"), os.Getenv("OBJGIT_TEST_KUBE_CA"), os.Getenv("OBJGIT_TEST_KUBE_TOKEN") + if host == "" || caFile == "" || token == "" { + t.Skip("skipping: OBJGIT_TEST_KUBE_HOST, OBJGIT_TEST_KUBE_CA, and OBJGIT_TEST_KUBE_TOKEN are not set") + } + caPEM, err := os.ReadFile(caFile) + if err != nil { + t.Fatal(err) + } + pool := x509.NewCertPool() + if !pool.AppendCertsFromPEM(caPEM) { + t.Fatalf("%s holds no certificate", caFile) + } + return New(Config{ + Host: host, + HTTPClient: &http.Client{Transport: &http.Transport{TLSClientConfig: &tls.Config{RootCAs: pool}}}, + Token: func() (string, error) { return token, nil }, + Namespace: "objgit", + }) +} + +// TestCluster runs the hook commands' API calls against a real API server as +// the objgitd ServiceAccount, with the example RBAC. +func TestCluster(t *testing.T) { + c := testCluster(t) + ctx := context.Background() + // A new suffix for each run, because the objects stay in the cluster. + suffix := strconv.FormatInt(time.Now().UnixNano(), 36) + + pipeline := Object{ + "apiVersion": "tekton.dev/v1", + "kind": "Pipeline", + "metadata": map[string]any{"name": "objgit-" + suffix, "namespace": "ci"}, + "spec": map[string]any{"description": "first"}, + } + if _, err := c.Apply(ctx, pipeline); err != nil { + t.Fatalf("apply creates the Pipeline: %v", err) + } + pipeline["spec"] = map[string]any{"description": "second"} + res, err := c.Apply(ctx, pipeline) + if err != nil { + t.Fatalf("re-apply patches the Pipeline: %v", err) + } + if got := res.Object["spec"].(map[string]any)["description"]; got != "second" { + t.Errorf("re-applied description = %v, want second", got) + } + + // Another field manager takes the field, so objgitd's next apply conflicts. + task := Object{ + "apiVersion": "tekton.dev/v1", + "kind": "Task", + "metadata": map[string]any{"name": "objgit-" + suffix, "namespace": "ci"}, + "spec": map[string]any{"description": "owned by objgitd"}, + } + if _, err := c.Apply(ctx, task); err != nil { + t.Fatalf("apply the Task: %v", err) + } + other := Object{ + "apiVersion": "tekton.dev/v1", "kind": "Task", + "metadata": map[string]any{"name": "objgit-" + suffix, "namespace": "ci"}, + "spec": map[string]any{"description": "taken by another manager"}, + } + s := c.Session() + tres, err := s.resource(ctx, "tekton.dev/v1", "Task") + if err != nil { + t.Fatal(err) + } + u := c.cfg.Host + collectionPath(tres, "ci") + "/objgit-" + suffix + "?" + + url.Values{"fieldManager": {"someone-else"}, "force": {"true"}}.Encode() + if _, err := c.send(ctx, http.MethodPatch, u, "application/apply-patch+yaml", other); err != nil { + t.Fatalf("forced apply by another manager: %v", err) + } + _, err = c.Apply(ctx, task) + if apiErr, ok := errors.AsType[*APIError](err); !ok || apiErr.Code != http.StatusConflict { + t.Errorf("apply over another manager's field = %v, want 409 Conflict", err) + } else { + t.Logf("conflict message: %s", apiErr.Message) + } + + run := Object{ + "apiVersion": "tekton.dev/v1", + "kind": "PipelineRun", + "metadata": map[string]any{"generateName": "objgit-" + suffix + "-", "namespace": "ci"}, + "spec": map[string]any{"pipelineRef": map[string]any{"name": "objgit-" + suffix}}, + } + var names []string + for range 2 { + res, err := c.Create(ctx, run) + if err != nil { + t.Fatalf("create PipelineRun: %v", err) + } + names = append(names, res.Object.Name()) + } + if names[0] == names[1] || !strings.HasPrefix(names[0], "objgit-"+suffix+"-") { + t.Errorf("run names = %q, want two different generated names", names) + } + + denied := []struct { + name string + obj Object + }{ + {"a ConfigMap in ci", Object{"apiVersion": "v1", "kind": "ConfigMap", "metadata": map[string]any{"name": "x", "namespace": "ci"}}}, + {"a Pipeline in default", Object{"apiVersion": "tekton.dev/v1", "kind": "Pipeline", "metadata": map[string]any{"name": "x", "namespace": "default"}}}, + } + for _, tt := range denied { + _, err := c.Apply(ctx, tt.obj) + if apiErr, ok := errors.AsType[*APIError](err); !ok || apiErr.Code != http.StatusForbidden { + t.Errorf("apply %s = %v, want 403 Forbidden", tt.name, err) + } else { + t.Logf("%s: %s", tt.name, apiErr.Message) + } + } +} diff --git a/internal/kube/commands.go b/internal/kube/commands.go new file mode 100644 index 0000000..f74827d --- /dev/null +++ b/internal/kube/commands.go @@ -0,0 +1,124 @@ +package kube + +import ( + "context" + "fmt" + "io" + "log/slog" + "strings" + + "github.com/Xe/kefka/command" + "github.com/Xe/kefka/command/registry" + "mvdan.cc/sh/v3/interp" +) + +// Origin is the update that a shell runs for, as the daemon knows it. The +// commands take it from the daemon and not from the OBJGIT_* variables, +// because a script can change those. The audit log and the PipelineRun +// metadata are the only record of which repository changed the cluster. +type Origin struct { + Repo string // the repository path + Ref string // the full ref name + Branch string // the short branch name; "" when Ref is not a branch + Commit string // the commit hash +} + +// Register adds kube:apply and tekton:pipelinerun to reg, for the update +// origin. A nil client registers stubs that explain how to turn the commands +// on, so a hook that calls them fails with a reason instead of "command not +// found". +func Register(reg *registry.Impl, c *Client, origin Origin) { + if c == nil { + reg.Register("kube:apply", disabled("kube:apply")) + reg.Register("tekton:pipelinerun", disabled("tekton:pipelinerun")) + return + } + reg.Register("kube:apply", ApplyCommand{Client: c, Origin: origin}) + reg.Register("tekton:pipelinerun", PipelineRunCommand{Client: c, Origin: origin}) +} + +type disabled string + +func (d disabled) Exec(_ context.Context, ec *command.ExecContext, _ []string) error { + fmt.Fprintf(ec.Stderr, "%s: Kubernetes commands are disabled; start objgitd with -allow-kubernetes to enable them\n", string(d)) + return interp.ExitStatus(1) +} + +// ApplyCommand is kube:apply. It reads a multi-document YAML stream from +// standard input and applies each object with Server-Side Apply, in stream +// order. The whole stream is parsed and checked before the first request. The +// first failed request stops the command. It never prunes. +type ApplyCommand struct { + Client *Client + Origin Origin +} + +// Exec runs kube:apply. +func (a ApplyCommand) Exec(ctx context.Context, ec *command.ExecContext, args []string) error { + const name = "kube:apply" + if len(args) != 0 { + fmt.Fprintln(ec.Stderr, "usage: kube:apply < manifests.yaml") + return interp.ExitStatus(2) + } + in := ec.Stdin + if in == nil { + in = strings.NewReader("") + } + objs, err := DecodeYAMLStream(in) + if err != nil { + return fail(ec, name, "standard input: %v", err) + } + if len(objs) == 0 { + return fail(ec, name, "no objects on standard input") + } + for i, obj := range objs { + if err := check(obj, false); err != nil { + return fail(ec, name, "object %d: %v", i+1, err) + } + } + + s := a.Client.Session() + for _, obj := range objs { + res, err := s.Apply(ctx, obj) + if err != nil { + return fail(ec, name, "%s: %v", describe(res.Resource, obj), err) + } + audit(a.Origin, "kube: applied", res) + fmt.Fprintf(ec.Stdout, "%s serverside-applied\n", describe(res.Resource, res.Object)) + } + return nil +} + +// describe names obj the way kubectl does, such as +// "pipeline.tekton.dev/build". Before discovery succeeds, res is zero, so the +// kind stands in. +func describe(res Resource, obj Object) string { + if res.Kind == "" { + return strings.ToLower(obj.Kind()) + "/" + obj.Name() + } + return res.String() + "/" + obj.Name() +} + +// fail writes "name: message" to stderr and returns exit status 1. +func fail(ec *command.ExecContext, name, format string, a ...any) error { + fmt.Fprintf(ec.Stderr, name+": "+format+"\n", a...) + return interp.ExitStatus(1) +} + +// audit logs one change to the cluster, with the update that made it. +func audit(origin Origin, msg string, res Result) { + slog.Info(msg, + "repo", origin.Repo, + "ref", origin.Ref, + "sha", origin.Commit, + "resource", res.Resource.String(), + "namespace", res.Object.Namespace(), + "name", res.Object.Name(), + ) +} + +// readAll is io.ReadAll that closes r. +func readAll(r io.ReadCloser) ([]byte, error) { + defer r.Close() + return io.ReadAll(r) +} diff --git a/internal/kube/commands_test.go b/internal/kube/commands_test.go new file mode 100644 index 0000000..40f0b71 --- /dev/null +++ b/internal/kube/commands_test.go @@ -0,0 +1,504 @@ +package kube + +import ( + "bytes" + "context" + "errors" + "log/slog" + "strings" + "testing" + + "github.com/Xe/kefka/command" + "github.com/Xe/kefka/command/registry" + "github.com/go-git/go-billy/v6/memfs" + "github.com/go-git/go-billy/v6/util" + "github.com/tigrisdata/objgit/internal/kube/kubetest" + "mvdan.cc/sh/v3/expand" + "mvdan.cc/sh/v3/interp" +) + +const testSHA = "0123456789abcdef0123456789abcdef01234567" + +// testOrigin is the update that the daemon gives the commands. +var testOrigin = Origin{Repo: "xe/x", Ref: "refs/heads/main", Branch: "main", Commit: testSHA} + +// hookEnv is the part of the hook environment the commands read. +var hookEnv = []string{ + "OBJGIT_REPO=xe/x", + "OBJGIT_REF=refs/heads/main", + "OBJGIT_BRANCH=main", + "OBJGIT_NEW_SHA=" + testSHA, +} + +// run executes cmd and returns its stdout, stderr, and exit status. +func run(t *testing.T, cmd command.Execer, ec *command.ExecContext, args ...string) (string, string, int) { + t.Helper() + var stdout, stderr bytes.Buffer + ec.Stdout, ec.Stderr = &stdout, &stderr + if ec.Environ == nil { + ec.Environ = expand.ListEnviron(hookEnv...) + } + if ec.FS == nil { + ec.FS = memfs.New() + } + err := cmd.Exec(context.Background(), ec, args) + code := 0 + if err != nil { + var exit interp.ExitStatus + if !errors.As(err, &exit) { + t.Fatalf("Exec returned %v, want nil or an interp.ExitStatus", err) + } + code = int(exit) + } + return stdout.String(), stderr.String(), code +} + +const twoConfigMaps = `apiVersion: v1 +kind: ConfigMap +metadata: + name: a +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: b + namespace: ci +` + +func TestApplyCommand(t *testing.T) { + tests := []struct { + name string + setup func(*kubetest.Server) + stdin string + args []string + wantCode int + wantStdout []string + wantStderr string + wantPaths []string // PATCH paths, in order + }{ + { + name: "applies every document in order", + stdin: twoConfigMaps, + wantStdout: []string{"configmap/a serverside-applied", "configmap/b serverside-applied"}, + wantPaths: []string{"/api/v1/namespaces/objgit/configmaps/a", "/api/v1/namespaces/ci/configmaps/b"}, + }, + { + name: "Tekton kinds use their group", + stdin: "apiVersion: tekton.dev/v1beta1\nkind: Pipeline\nmetadata:\n name: build\n namespace: ci\n", + wantStdout: []string{"pipeline.tekton.dev/build serverside-applied"}, + wantPaths: []string{"/apis/tekton.dev/v1beta1/namespaces/ci/pipelines/build"}, + }, + { + name: "empty input fails", + stdin: "# nothing here\n", + wantCode: 1, + wantStderr: "no objects", + }, + { + name: "malformed input fails before any request", + stdin: twoConfigMaps + "---\nkind: [\n", + wantCode: 1, + wantStderr: "document 3", + }, + { + name: "an invalid later document fails before any request", + stdin: twoConfigMaps + "---\napiVersion: v1\nkind: ConfigMap\n", + wantCode: 1, + wantStderr: "object 3: ConfigMap object has no metadata.name", + }, + { + name: "field conflict fails", + setup: func(s *kubetest.Server) { + s.Put("/api/v1/namespaces/objgit/configmaps/a", "kubectl", map[string]any{}) + }, + stdin: twoConfigMaps, + wantCode: 1, + wantStderr: "configmap/a: Apply failed with 1 conflict", + wantPaths: []string{"/api/v1/namespaces/objgit/configmaps/a"}, + }, + { + name: "RBAC denial fails and stops the stream", + setup: func(s *kubetest.Server) { s.Deny("create", "configmaps") }, + stdin: twoConfigMaps, + wantCode: 1, + wantStderr: `cannot create resource "configmaps"`, + wantPaths: []string{"/api/v1/namespaces/objgit/configmaps/a"}, + }, + { + name: "unknown kind fails", + stdin: "apiVersion: v1\nkind: Widget\nmetadata:\n name: w\n", + wantCode: 1, + wantStderr: `no resource of kind "Widget"`, + }, + { + name: "arguments are a usage error", + args: []string{"-f", "x.yaml"}, + wantCode: 2, + wantStderr: "usage: kube:apply", + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + srv := kubetest.New(t) + if tt.setup != nil { + tt.setup(srv) + } + stdout, stderr, code := run(t, ApplyCommand{Client: testClient(srv), Origin: testOrigin}, + &command.ExecContext{Stdin: strings.NewReader(tt.stdin)}, tt.args...) + if code != tt.wantCode { + t.Fatalf("exit = %d, want %d; stderr: %s", code, tt.wantCode, stderr) + } + for _, want := range tt.wantStdout { + if !strings.Contains(stdout, want) { + t.Errorf("stdout lacks %q:\n%s", want, stdout) + } + } + if !strings.Contains(stderr, tt.wantStderr) { + t.Errorf("stderr = %q, want it to contain %q", stderr, tt.wantStderr) + } + var paths []string + for _, r := range srv.Requests() { + paths = append(paths, r.Path) + } + if strings.Join(paths, " ") != strings.Join(tt.wantPaths, " ") { + t.Errorf("requests = %q, want %q", paths, tt.wantPaths) + } + }) + } +} + +func TestApplyCommandReapply(t *testing.T) { + srv := kubetest.New(t) + cmd := ApplyCommand{Client: testClient(srv), Origin: testOrigin} + for i := range 2 { + if _, stderr, code := run(t, cmd, &command.ExecContext{Stdin: strings.NewReader(twoConfigMaps)}); code != 0 { + t.Fatalf("apply %d: exit %d; stderr: %s", i+1, code, stderr) + } + } + if n := len(srv.Requests()); n != 4 { + t.Errorf("%d requests, want 4", n) + } +} + +const testRun = `apiVersion: tekton.dev/v1 +kind: PipelineRun +metadata: + generateName: x-m- + namespace: ci + labels: + app: x +spec: + params: + - name: commit + value: master + - name: branch + value: master + - name: actor + value: did:plc:e5nncb3dr5thdkjir5cfaqfe + pipelineRef: + name: xe-x-build-test +` + +func TestPipelineRunCommand(t *testing.T) { + srv := kubetest.New(t) + fsys := memfs.New() + if err := util.WriteFile(fsys, "src/.tekton/testrun.yaml", []byte(testRun), 0o644); err != nil { + t.Fatal(err) + } + cmd := PipelineRunCommand{Client: testClient(srv), Origin: testOrigin} + + stdout, stderr, code := run(t, cmd, &command.ExecContext{Dir: "src", FS: fsys}, ".tekton/testrun.yaml") + if code != 0 { + t.Fatalf("exit %d; stderr: %s", code, stderr) + } + if want := "pipelinerun.tekton.dev/x-m-00001 created in namespace ci\n"; stdout != want { + t.Errorf("stdout = %q, want %q", stdout, want) + } + + reqs := srv.Requests() + if len(reqs) != 1 || reqs[0].Method != "POST" || reqs[0].Path != "/apis/tekton.dev/v1/namespaces/ci/pipelineruns" { + t.Fatalf("requests = %+v, want one POST to ci pipelineruns", reqs) + } + body := Object(reqs[0].Body) + params := map[string]any{} + for _, p := range body["spec"].(map[string]any)["params"].([]any) { + p := p.(map[string]any) + params[p["name"].(string)] = p["value"] + } + for name, want := range map[string]any{"commit": testSHA, "branch": "main", "actor": "did:plc:e5nncb3dr5thdkjir5cfaqfe"} { + if params[name] != want { + t.Errorf("param %s = %v, want %v", name, params[name], want) + } + } + meta := body["metadata"].(map[string]any) + annotations, _ := meta["annotations"].(map[string]any) + for key, want := range map[string]string{ + "objgit.tigrisdata.com/repo": "xe/x", + "objgit.tigrisdata.com/ref": "refs/heads/main", + "objgit.tigrisdata.com/commit": testSHA, + } { + if annotations[key] != want { + t.Errorf("annotation %s = %v, want %s", key, annotations[key], want) + } + } + labels, _ := meta["labels"].(map[string]any) + if labels["objgit.tigrisdata.com/commit-prefix"] != testSHA[:12] || labels["app"] != "x" { + t.Errorf("labels = %v, want the commit prefix label next to app=x", labels) + } + + // Each call is a new run, even for the same commit. + stdout, _, code = run(t, cmd, &command.ExecContext{Dir: "src", FS: fsys}, "/src/.tekton/testrun.yaml") + if code != 0 || !strings.Contains(stdout, "x-m-00002") { + t.Errorf("second call: exit %d, stdout %q, want run x-m-00002", code, stdout) + } + if n := len(srv.Requests()); n != 2 { + t.Errorf("%d create requests after two calls, want 2", n) + } +} + +func TestPipelineRunCommandDefaultNamespace(t *testing.T) { + srv := kubetest.New(t) + fsys := memfs.New() + manifest := strings.Replace(testRun, " namespace: ci\n", "", 1) + if err := util.WriteFile(fsys, "run.yaml", []byte(manifest), 0o644); err != nil { + t.Fatal(err) + } + stdout, stderr, code := run(t, PipelineRunCommand{Client: testClient(srv), Origin: testOrigin}, &command.ExecContext{FS: fsys}, "run.yaml") + if code != 0 { + t.Fatalf("exit %d; stderr: %s", code, stderr) + } + if !strings.HasSuffix(stdout, "created in namespace objgit\n") { + t.Errorf("stdout = %q, want the default namespace", stdout) + } + if reqs := srv.Requests(); len(reqs) != 1 || reqs[0].Path != "/apis/tekton.dev/v1/namespaces/objgit/pipelineruns" { + t.Errorf("requests = %+v, want one POST to objgit pipelineruns", reqs) + } +} + +func TestPipelineRunCommandWithoutBranchParam(t *testing.T) { + srv := kubetest.New(t) + fsys := memfs.New() + manifest := strings.Replace(testRun, " - name: branch\n value: master\n", "", 1) + if err := util.WriteFile(fsys, "run.yaml", []byte(manifest), 0o644); err != nil { + t.Fatal(err) + } + if _, stderr, code := run(t, PipelineRunCommand{Client: testClient(srv), Origin: testOrigin}, &command.ExecContext{FS: fsys}, "run.yaml"); code != 0 { + t.Fatalf("exit %d; stderr: %s", code, stderr) + } + for _, p := range srv.Requests()[0].Body["spec"].(map[string]any)["params"].([]any) { + if p.(map[string]any)["name"] == "branch" { + t.Errorf("branch parameter was added: %v", p) + } + } +} + +func TestPipelineRunCommandErrors(t *testing.T) { + tests := []struct { + name string + setup func(*kubetest.Server) + manifest string // written to run.yaml; "" writes nothing + origin *Origin // nil means testOrigin + args []string // nil means run.yaml + noArgs bool + wantCode int + wantStderr string + }{ + {name: "no file argument", manifest: testRun, noArgs: true, wantCode: 2, wantStderr: "usage: tekton:pipelinerun FILE"}, + {name: "two file arguments", manifest: testRun, args: []string{"run.yaml", "run.yaml"}, wantCode: 2, wantStderr: "usage"}, + {name: "missing file", args: []string{"run.yaml"}, wantCode: 1, wantStderr: "run.yaml"}, + { + name: "fixed name", + manifest: strings.Replace(testRun, "generateName: x-m-", "name: x-m-fixed", 1), + wantCode: 1, + wantStderr: "metadata.generateName", + }, + { + name: "name and generateName", + manifest: strings.Replace(testRun, "generateName: x-m-", "generateName: x-m-\n name: fixed", 1), + wantCode: 1, + wantStderr: "must not set metadata.name", + }, + { + name: "not a PipelineRun", + manifest: strings.Replace(testRun, "kind: PipelineRun", "kind: Pipeline", 1), + wantCode: 1, + wantStderr: "not a tekton.dev PipelineRun", + }, + { + name: "not Tekton", + manifest: strings.Replace(testRun, "tekton.dev/v1", "example.com/v1", 1), + wantCode: 1, + wantStderr: "not a tekton.dev PipelineRun", + }, + { + name: "no commit parameter", + manifest: strings.Replace(testRun, " - name: commit\n value: master\n", "", 1), + wantCode: 1, + wantStderr: "no commit parameter", + }, + { + name: "two documents", + manifest: testRun + "---\n" + testRun, + wantCode: 1, + wantStderr: "holds 2 objects", + }, + { + name: "no commit in the origin", + manifest: testRun, + origin: &Origin{Repo: "xe/x", Ref: "refs/heads/main", Branch: "main"}, + wantCode: 1, + wantStderr: "no commit", + }, + { + name: "an all-zero commit in the origin", + manifest: testRun, + origin: &Origin{Repo: "xe/x", Commit: strings.Repeat("0", 40)}, + wantCode: 1, + wantStderr: "no commit", + }, + { + name: "RBAC denial", + setup: func(s *kubetest.Server) { s.Deny("create", "pipelineruns") }, + manifest: testRun, + wantCode: 1, + wantStderr: "forbidden", + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + srv := kubetest.New(t) + if tt.setup != nil { + tt.setup(srv) + } + fsys := memfs.New() + if tt.manifest != "" { + if err := util.WriteFile(fsys, "run.yaml", []byte(tt.manifest), 0o644); err != nil { + t.Fatal(err) + } + } + args := tt.args + if args == nil && !tt.noArgs { + args = []string{"run.yaml"} + } + origin := testOrigin + if tt.origin != nil { + origin = *tt.origin + } + _, stderr, code := run(t, PipelineRunCommand{Client: testClient(srv), Origin: origin}, &command.ExecContext{FS: fsys}, args...) + if code != tt.wantCode { + t.Fatalf("exit = %d, want %d; stderr: %s", code, tt.wantCode, stderr) + } + if !strings.Contains(stderr, tt.wantStderr) { + t.Errorf("stderr = %q, want it to contain %q", stderr, tt.wantStderr) + } + if tt.setup == nil && len(srv.Requests()) != 0 { + t.Errorf("a request was sent for an invalid run") + } + }) + } +} + +func TestRegister(t *testing.T) { + srv := kubetest.New(t) + tests := []struct { + name string + client *Client + wantCode int + wantStderr string + }{ + {name: "disabled", client: nil, wantCode: 1, wantStderr: "start objgitd with -allow-kubernetes"}, + {name: "enabled", client: testClient(srv), wantCode: 1, wantStderr: "no objects"}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + reg := registry.New() + Register(reg, tt.client, testOrigin) + for _, name := range []string{"kube:apply", "tekton:pipelinerun"} { + cmd, ok := reg.Get(name) + if !ok { + t.Fatalf("%s is not registered", name) + } + if name != "kube:apply" && tt.client != nil { + continue + } + _, stderr, code := run(t, cmd, &command.ExecContext{Stdin: strings.NewReader("")}) + if code != tt.wantCode || !strings.Contains(stderr, tt.wantStderr) { + t.Errorf("%s: exit %d, stderr %q; want exit %d and %q", name, code, stderr, tt.wantCode, tt.wantStderr) + } + } + }) + } +} + +// TestCommandsIgnoreSpoofedEnvironment runs both commands with OBJGIT_* +// variables that a script changed. The audit log, the run metadata, and the +// parameters must come from the daemon's origin instead. +func TestCommandsIgnoreSpoofedEnvironment(t *testing.T) { + var logs bytes.Buffer + prev := slog.Default() + slog.SetDefault(slog.New(slog.NewTextHandler(&logs, nil))) + defer slog.SetDefault(prev) + + spoofed := expand.ListEnviron( + "OBJGIT_REPO=victim/repo", + "OBJGIT_REF=refs/heads/prod", + "OBJGIT_BRANCH=prod", + "OBJGIT_NEW_SHA=ffffffffffffffffffffffffffffffffffffffff", + ) + srv := kubetest.New(t) + fsys := memfs.New() + if err := util.WriteFile(fsys, "run.yaml", []byte(testRun), 0o644); err != nil { + t.Fatal(err) + } + + if _, stderr, code := run(t, ApplyCommand{Client: testClient(srv), Origin: testOrigin}, + &command.ExecContext{Stdin: strings.NewReader(twoConfigMaps), Environ: spoofed}); code != 0 { + t.Fatalf("kube:apply: exit %d; stderr: %s", code, stderr) + } + if _, stderr, code := run(t, PipelineRunCommand{Client: testClient(srv), Origin: testOrigin}, + &command.ExecContext{FS: fsys, Environ: spoofed}, "run.yaml"); code != 0 { + t.Fatalf("tekton:pipelinerun: exit %d; stderr: %s", code, stderr) + } + + if strings.Contains(logs.String(), "victim") || strings.Contains(logs.String(), "prod") { + t.Errorf("audit log uses the spoofed environment:\n%s", logs.String()) + } + if strings.Count(logs.String(), "repo=xe/x") != 3 { + t.Errorf("audit log lacks repo=xe/x for each change:\n%s", logs.String()) + } + + reqs := srv.Requests() + body := Object(reqs[len(reqs)-1].Body) + annotations := body["metadata"].(map[string]any)["annotations"].(map[string]any) + if annotations[AnnotationRepo] != "xe/x" || annotations[AnnotationRef] != "refs/heads/main" || annotations[AnnotationCommit] != testSHA { + t.Errorf("annotations = %v, want the origin", annotations) + } + for _, p := range body["spec"].(map[string]any)["params"].([]any) { + p := p.(map[string]any) + if (p["name"] == "commit" && p["value"] != testSHA) || (p["name"] == "branch" && p["value"] != "main") { + t.Errorf("param %v, want the origin value", p) + } + } +} + +// TestPipelineRunCommandWithoutBranch is the SSH sh shell on a tag or a +// commit: the origin has no branch, so the template's branch stays. +func TestPipelineRunCommandWithoutBranch(t *testing.T) { + srv := kubetest.New(t) + fsys := memfs.New() + if err := util.WriteFile(fsys, "run.yaml", []byte(testRun), 0o644); err != nil { + t.Fatal(err) + } + origin := Origin{Repo: "xe/x", Ref: "refs/tags/v1.0", Commit: testSHA} + if _, stderr, code := run(t, PipelineRunCommand{Client: testClient(srv), Origin: origin}, &command.ExecContext{FS: fsys}, "run.yaml"); code != 0 { + t.Fatalf("exit %d; stderr: %s", code, stderr) + } + for _, p := range srv.Requests()[0].Body["spec"].(map[string]any)["params"].([]any) { + p := p.(map[string]any) + if p["name"] == "branch" && p["value"] != "master" { + t.Errorf("branch param = %v, want the template value master", p["value"]) + } + } +} diff --git a/internal/kube/kubetest/kubetest.go b/internal/kube/kubetest/kubetest.go new file mode 100644 index 0000000..cbd8594 --- /dev/null +++ b/internal/kube/kubetest/kubetest.go @@ -0,0 +1,372 @@ +// Package kubetest is a fake Kubernetes API server for tests. It serves +// discovery for a fixed set of resources, stores objects in memory, and +// answers Server-Side Apply, create, and get. It is not a conformant API +// server: apply replaces the whole object, and a conflict is any apply over +// an object that another field manager owns. Its RBAC checks match a real +// server's verbs, which internal/kube's TestCluster confirms. +package kubetest + +import ( + "encoding/json" + "fmt" + "io" + "maps" + "net/http" + "net/http/httptest" + "strings" + "sync" + "testing" +) + +// Resource is one discoverable resource of the fake server. +type Resource struct { + Group, Version, Kind, Name string + Namespaced bool +} + +// Resources is the default discovery data: core ConfigMaps and Namespaces, +// and the Tekton kinds that a hook creates. +var Resources = []Resource{ + {"", "v1", "ConfigMap", "configmaps", true}, + {"", "v1", "Namespace", "namespaces", false}, + {"tekton.dev", "v1", "Pipeline", "pipelines", true}, + {"tekton.dev", "v1", "Task", "tasks", true}, + {"tekton.dev", "v1", "PipelineRun", "pipelineruns", true}, + {"tekton.dev", "v1beta1", "Pipeline", "pipelines", true}, + {"tekton.dev", "v1beta1", "PipelineRun", "pipelineruns", true}, +} + +// Request is one request the server received. +type Request struct { + Method, Path, ContentType, Authorization string + Query map[string]string + Body map[string]any +} + +// Server is a running fake API server. +type Server struct { + *httptest.Server + + mu sync.Mutex + token string // the bearer token every request must carry + objects map[string]stored // by object path + denied map[string]bool // "verb resource", such as "patch pipelines" + requests []Request + paths []string // the raw path of every request, discovery included + nextName int +} + +type stored struct { + manager string + obj map[string]any +} + +// New starts a TLS fake API server that stops when t ends. +func New(t testing.TB) *Server { + s := &Server{ + token: "test-token", + objects: map[string]stored{}, + denied: map[string]bool{}, + } + s.Server = httptest.NewTLSServer(http.HandlerFunc(s.serve)) + t.Cleanup(s.Close) + return s +} + +// Token returns the bearer token the server accepts. +func (s *Server) Token() string { + s.mu.Lock() + defer s.mu.Unlock() + return s.token +} + +// SetToken changes the bearer token the server accepts, as a rotation does. +func (s *Server) SetToken(token string) { + s.mu.Lock() + defer s.mu.Unlock() + s.token = token +} + +// Deny makes the server refuse verb ("create", "patch", "get") on resource, +// such as "pipelineruns", with 403 Forbidden. +func (s *Server) Deny(verb, resource string) { + s.mu.Lock() + defer s.mu.Unlock() + s.denied[verb+" "+resource] = true +} + +// Put stores obj at path as owned by manager, so a later apply by another +// manager conflicts. +func (s *Server) Put(path, manager string, obj map[string]any) { + s.mu.Lock() + defer s.mu.Unlock() + s.objects[path] = stored{manager: manager, obj: obj} +} + +// Object returns the object stored at path, such as +// "/apis/tekton.dev/v1/namespaces/ci/pipelines/build". +func (s *Server) Object(path string) (map[string]any, bool) { + s.mu.Lock() + defer s.mu.Unlock() + o, ok := s.objects[path] + return o.obj, ok +} + +// Requests returns every mutating request (not discovery or get) so far. +func (s *Server) Requests() []Request { + s.mu.Lock() + defer s.mu.Unlock() + return append([]Request(nil), s.requests...) +} + +// Paths returns the escaped path of every request so far, discovery and get +// included. +func (s *Server) Paths() []string { + s.mu.Lock() + defer s.mu.Unlock() + return append([]string(nil), s.paths...) +} + +func (s *Server) serve(w http.ResponseWriter, r *http.Request) { + s.mu.Lock() + s.paths = append(s.paths, r.URL.EscapedPath()) + s.mu.Unlock() + if r.Header.Get("Authorization") != "Bearer "+s.Token() { + status(w, http.StatusUnauthorized, "Unauthorized", "Unauthorized") + return + } + if list, ok := s.discovery(r.URL.Path); ok { + if r.Method != http.MethodGet { + status(w, http.StatusMethodNotAllowed, "MethodNotAllowed", "discovery is read-only") + return + } + writeJSON(w, http.StatusOK, list) + return + } + res, ns, name, ok := s.route(r.URL.Path) + if !ok { + status(w, http.StatusNotFound, "NotFound", "the server could not find the requested resource") + return + } + + var body map[string]any + if r.Method == http.MethodPost || r.Method == http.MethodPatch { + data, _ := io.ReadAll(r.Body) + if err := json.Unmarshal(data, &body); err != nil { + status(w, http.StatusBadRequest, "BadRequest", "body is not JSON: "+err.Error()) + return + } + q := map[string]string{} + for k := range r.URL.Query() { + q[k] = r.URL.Query().Get(k) + } + s.mu.Lock() + s.requests = append(s.requests, Request{ + Method: r.Method, Path: r.URL.Path, ContentType: r.Header.Get("Content-Type"), + Authorization: r.Header.Get("Authorization"), Query: q, Body: body, + }) + s.mu.Unlock() + } + + switch { + case r.Method == http.MethodGet && name != "": + s.get(w, res, ns, name, r.URL.Path) + case r.Method == http.MethodPost && name == "": + s.create(w, r, res, ns, body) + case r.Method == http.MethodPatch && name != "": + s.apply(w, r, res, ns, name, body) + default: + status(w, http.StatusMethodNotAllowed, "MethodNotAllowed", r.Method+" is not supported here") + } +} + +func (s *Server) forbidden(w http.ResponseWriter, verb string, res Resource, ns, name string) bool { + s.mu.Lock() + denied := s.denied[verb+" "+res.Name] + s.mu.Unlock() + if !denied { + return false + } + qualified := res.Name + if res.Group != "" { + qualified += "." + res.Group + } + status(w, http.StatusForbidden, "Forbidden", fmt.Sprintf( + "%s %q is forbidden: User \"system:serviceaccount:objgit:objgitd\" cannot %s resource %q in API group %q in the namespace %q", + qualified, name, verb, res.Name, res.Group, ns)) + return true +} + +func (s *Server) get(w http.ResponseWriter, res Resource, ns, name, path string) { + if s.forbidden(w, "get", res, ns, name) { + return + } + obj, ok := s.Object(path) + if !ok { + status(w, http.StatusNotFound, "NotFound", fmt.Sprintf("%s %q not found", res.Name, name)) + return + } + writeJSON(w, http.StatusOK, obj) +} + +func (s *Server) create(w http.ResponseWriter, r *http.Request, res Resource, ns string, body map[string]any) { + if r.Header.Get("Content-Type") != "application/json" { + status(w, http.StatusUnsupportedMediaType, "UnsupportedMediaType", "create needs application/json") + return + } + meta, _ := body["metadata"].(map[string]any) + name, _ := meta["name"].(string) + if name == "" { + gen, _ := meta["generateName"].(string) + if gen == "" { + status(w, http.StatusUnprocessableEntity, "Invalid", "metadata.name or metadata.generateName is required") + return + } + s.mu.Lock() + s.nextName++ + name = fmt.Sprintf("%s%05d", gen, s.nextName) + s.mu.Unlock() + } + if s.forbidden(w, "create", res, ns, name) { + return + } + path := r.URL.Path + "/" + name + if _, exists := s.Object(path); exists { + status(w, http.StatusConflict, "AlreadyExists", fmt.Sprintf("%s %q already exists", res.Name, name)) + return + } + obj := withMeta(body, ns, name, res) + s.Put(path, r.URL.Query().Get("fieldManager"), obj) + writeJSON(w, http.StatusCreated, obj) +} + +func (s *Server) apply(w http.ResponseWriter, r *http.Request, res Resource, ns, name string, body map[string]any) { + if r.Header.Get("Content-Type") != "application/apply-patch+yaml" { + status(w, http.StatusUnsupportedMediaType, "UnsupportedMediaType", "this fake only serves Server-Side Apply patches") + return + } + manager := r.URL.Query().Get("fieldManager") + if manager == "" { + status(w, http.StatusBadRequest, "BadRequest", "fieldManager is required for apply requests") + return + } + // Like a real API server: every apply needs patch, and an apply that + // creates the object needs create as well. + existing, exists := s.lookup(r.URL.Path) + if s.forbidden(w, "patch", res, ns, name) || (!exists && s.forbidden(w, "create", res, ns, name)) { + return + } + if exists && existing.manager != manager && r.URL.Query().Get("force") != "true" { + status(w, http.StatusConflict, "Conflict", fmt.Sprintf( + "Apply failed with 1 conflict: conflict with %q: .spec", existing.manager)) + return + } + code := http.StatusOK + if !exists { + code = http.StatusCreated + } + obj := withMeta(body, ns, name, res) + s.Put(r.URL.Path, manager, obj) + writeJSON(w, code, obj) +} + +func (s *Server) lookup(path string) (stored, bool) { + s.mu.Lock() + defer s.mu.Unlock() + o, ok := s.objects[path] + return o, ok +} + +// withMeta returns a copy of body with the server-assigned name and namespace. +func withMeta(body map[string]any, ns, name string, res Resource) map[string]any { + obj := maps.Clone(body) + meta, _ := obj["metadata"].(map[string]any) + meta = maps.Clone(meta) + if meta == nil { + meta = map[string]any{} + } + meta["name"] = name + if res.Namespaced { + meta["namespace"] = ns + } + obj["metadata"] = meta + return obj +} + +// discovery answers /api/v1 and /apis//. +func (s *Server) discovery(path string) (map[string]any, bool) { + var group, version string + switch parts := strings.Split(strings.Trim(path, "/"), "/"); { + case len(parts) == 2 && parts[0] == "api": + version = parts[1] + case len(parts) == 3 && parts[0] == "apis": + group, version = parts[1], parts[2] + default: + return nil, false + } + var resources []map[string]any + for _, r := range Resources { + if r.Group == group && r.Version == version { + // A subresource comes first, so a lookup that does not skip it fails. + resources = append(resources, map[string]any{ + "name": r.Name + "/status", "kind": r.Kind, "namespaced": r.Namespaced, + "verbs": []string{"get", "patch"}, + }) + resources = append(resources, map[string]any{ + "name": r.Name, "kind": r.Kind, "namespaced": r.Namespaced, + "verbs": []string{"create", "get", "patch"}, + }) + } + } + if resources == nil { + return nil, false + } + gv := version + if group != "" { + gv = group + "/" + version + } + return map[string]any{"kind": "APIResourceList", "groupVersion": gv, "resources": resources}, true +} + +// route parses an object or collection path into its resource, namespace, +// and name. +func (s *Server) route(path string) (res Resource, ns, name string, ok bool) { + parts := strings.Split(strings.Trim(path, "/"), "/") + var group, version string + switch { + case len(parts) >= 3 && parts[0] == "api": + version, parts = parts[1], parts[2:] + case len(parts) >= 4 && parts[0] == "apis": + group, version, parts = parts[1], parts[2], parts[3:] + default: + return Resource{}, "", "", false + } + if len(parts) >= 3 && parts[0] == "namespaces" { + ns, parts = parts[1], parts[2:] + } + if len(parts) < 1 || len(parts) > 2 { + return Resource{}, "", "", false + } + for _, r := range Resources { + if r.Group == group && r.Version == version && r.Name == parts[0] && r.Namespaced == (ns != "") { + if len(parts) == 2 { + name = parts[1] + } + return r, ns, name, true + } + } + return Resource{}, "", "", false +} + +func status(w http.ResponseWriter, code int, reason, message string) { + writeJSON(w, code, map[string]any{ + "kind": "Status", "apiVersion": "v1", "status": "Failure", + "message": message, "reason": reason, "code": code, + }) +} + +func writeJSON(w http.ResponseWriter, code int, v any) { + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(code) + _ = json.NewEncoder(w).Encode(v) +} diff --git a/internal/kube/tekton.go b/internal/kube/tekton.go new file mode 100644 index 0000000..b1c4a19 --- /dev/null +++ b/internal/kube/tekton.go @@ -0,0 +1,150 @@ +package kube + +import ( + "bytes" + "context" + "fmt" + "path" + "strings" + + "github.com/Xe/kefka/command" + "mvdan.cc/sh/v3/interp" +) + +// Metadata that tekton:pipelinerun adds to each run, so a later command can +// find the runs of a commit. +const ( + AnnotationRepo = "objgit.tigrisdata.com/repo" + AnnotationRef = "objgit.tigrisdata.com/ref" + AnnotationCommit = "objgit.tigrisdata.com/commit" + LabelCommitPrefix = "objgit.tigrisdata.com/commit-prefix" +) + +// commitPrefixLen is how much of the commit hash the label keeps. A label +// value is at most 63 characters, which a SHA-256 hash does not fit. +const commitPrefixLen = 12 + +// PipelineRunCommand is tekton:pipelinerun FILE. It reads one PipelineRun +// template from the hook filesystem, points it at the commit of Origin, and +// creates it. The template must use metadata.generateName, so each call +// creates a new run. +type PipelineRunCommand struct { + Client *Client + Origin Origin +} + +// Exec runs tekton:pipelinerun. +func (p PipelineRunCommand) Exec(ctx context.Context, ec *command.ExecContext, args []string) error { + const name = "tekton:pipelinerun" + if len(args) != 1 { + fmt.Fprintln(ec.Stderr, "usage: tekton:pipelinerun FILE") + return interp.ExitStatus(2) + } + file := args[0] + if strings.Trim(p.Origin.Commit, "0") == "" { + return fail(ec, name, "this shell has no commit to build") + } + + f, err := ec.FS.Open(resolve(ec.Dir, file)) + if err != nil { + return fail(ec, name, "%s: %v", file, err) + } + data, err := readAll(f) + if err != nil { + return fail(ec, name, "%s: %v", file, err) + } + objs, err := DecodeYAMLStream(bytes.NewReader(data)) + if err != nil { + return fail(ec, name, "%s: %v", file, err) + } + if len(objs) != 1 { + return fail(ec, name, "%s holds %d objects, want one PipelineRun", file, len(objs)) + } + run := objs[0] + if err := preparePipelineRun(run, p.Origin); err != nil { + return fail(ec, name, "%s: %v", file, err) + } + + res, err := p.Client.Create(ctx, run) + if err != nil { + return fail(ec, name, "create %s: %v", strings.TrimSuffix(run.GenerateName(), "-"), err) + } + audit(p.Origin, "kube: created", res) + fmt.Fprintf(ec.Stdout, "%s/%s created in namespace %s\n", res.Resource, res.Object.Name(), res.Object.Namespace()) + return nil +} + +// resolve maps a shell path to an fsys-relative path: absolute paths start +// at the filesystem root, and relative paths start at dir. Unlike the Kefka +// registry, it does not clamp a path that climbs above the root. The hook +// filesystem (internal/mountfs) cleans every path against its root. +func resolve(dir, p string) string { + if path.IsAbs(p) { + p = strings.TrimPrefix(path.Clean(p), "/") + } else { + p = path.Join(dir, p) + } + if p == "" { + return "." + } + return p +} + +// preparePipelineRun checks that run is a generateName PipelineRun, and sets +// its commit and branch parameters and its objgit metadata from origin. The +// branch parameter keeps its template value when origin is not a branch. +func preparePipelineRun(run Object, origin Origin) error { + commit := origin.Commit + group, _, _ := strings.Cut(run.APIVersion(), "/") + if group != "tekton.dev" || run.Kind() != "PipelineRun" { + return fmt.Errorf("%s %s is not a tekton.dev PipelineRun", run.APIVersion(), run.Kind()) + } + if run.Name() != "" { + return fmt.Errorf("a PipelineRun template must not set metadata.name (%q); use metadata.generateName so each push creates a new run", run.Name()) + } + if run.GenerateName() == "" { + return fmt.Errorf("a PipelineRun template needs metadata.generateName") + } + + spec, _ := run["spec"].(map[string]any) + params, _ := spec["params"].([]any) + var haveCommit bool + for _, p := range params { + p, ok := p.(map[string]any) + if !ok { + continue + } + switch p["name"] { + case "commit": + p["value"] = commit + haveCommit = true + case "branch": + if origin.Branch != "" { + p["value"] = origin.Branch + } + } + } + if !haveCommit { + return fmt.Errorf("the PipelineRun has no commit parameter; add one to spec.params") + } + + meta := run["metadata"].(map[string]any) // GenerateName found it + annotations := subMap(meta, "annotations") + for key, v := range map[string]string{AnnotationRepo: origin.Repo, AnnotationRef: origin.Ref, AnnotationCommit: commit} { + if v != "" { + annotations[key] = v + } + } + subMap(meta, "labels")[LabelCommitPrefix] = commit[:min(len(commit), commitPrefixLen)] + return nil +} + +// subMap returns m[key] as a map, and creates it when it is missing. +func subMap(m map[string]any, key string) map[string]any { + sub, ok := m[key].(map[string]any) + if !ok { + sub = map[string]any{} + m[key] = sub + } + return sub +} diff --git a/internal/kube/yaml.go b/internal/kube/yaml.go new file mode 100644 index 0000000..dcafa51 --- /dev/null +++ b/internal/kube/yaml.go @@ -0,0 +1,79 @@ +package kube + +import ( + "bufio" + "bytes" + "encoding/json" + "fmt" + "io" + + "sigs.k8s.io/yaml" +) + +// DecodeYAMLStream reads a multi-document YAML stream, such as kustomize +// output, and returns one Object per document. Documents that hold only +// comments or whitespace are skipped. An empty stream returns no objects and +// no error; the caller decides whether that is a failure. +// +// Documents split on a line that starts with "---", as in apimachinery's +// YAMLReader. Each document converts to JSON the way kubectl converts it, and +// integers keep their full precision. +func DecodeYAMLStream(r io.Reader) ([]Object, error) { + docs, err := splitYAML(r) + if err != nil { + return nil, err + } + var objs []Object + for i, doc := range docs { + data, err := yaml.YAMLToJSONStrict(doc) + if err != nil { + return nil, fmt.Errorf("document %d: %w", i+1, err) + } + data = bytes.TrimSpace(data) + if bytes.Equal(data, []byte("null")) { + continue + } + if len(data) == 0 || data[0] != '{' { + return nil, fmt.Errorf("document %d is not a mapping", i+1) + } + dec := json.NewDecoder(bytes.NewReader(data)) + dec.UseNumber() + var obj Object + if err := dec.Decode(&obj); err != nil { + return nil, fmt.Errorf("document %d: %w", i+1, err) + } + objs = append(objs, obj) + } + return objs, nil +} + +// splitYAML splits a stream into documents. A separator line is "---" +// followed by nothing, whitespace, or a comment. Other content after "---", +// such as "--- {a: 1}", is an error: the YAML converter would read only the +// first document of such a piece and drop the rest without an error. +func splitYAML(r io.Reader) ([][]byte, error) { + var docs [][]byte + var cur bytes.Buffer + br := bufio.NewReader(r) + for { + line, err := br.ReadBytes('\n') + if len(line) > 0 { + if rest, ok := bytes.CutPrefix(line, []byte("---")); ok { + if rest = bytes.TrimSpace(rest); len(rest) != 0 && rest[0] != '#' { + return nil, fmt.Errorf("document %d: invalid document separator %q; put the content on the next line", len(docs)+2, bytes.TrimRight(line, "\r\n")) + } + docs = append(docs, bytes.Clone(cur.Bytes())) + cur.Reset() + line = nil + } + cur.Write(line) + } + if err == io.EOF { + break + } + if err != nil { + return nil, err + } + } + return append(docs, cur.Bytes()), nil +} diff --git a/internal/kube/yaml_test.go b/internal/kube/yaml_test.go new file mode 100644 index 0000000..555dc5c --- /dev/null +++ b/internal/kube/yaml_test.go @@ -0,0 +1,123 @@ +package kube + +import ( + "encoding/json" + "strings" + "testing" +) + +func TestDecodeYAMLStream(t *testing.T) { + tests := []struct { + name string + input string + wantNames []string + wantErr string // substring; "" means success + }{ + { + name: "one document", + input: "apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: a\n", + wantNames: []string{"a"}, + }, + { + name: "kustomize output", + input: "apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: a\n---\napiVersion: v1\nkind: ConfigMap\nmetadata:\n name: b\n", + wantNames: []string{"a", "b"}, + }, + { + name: "leading and trailing separators", + input: "---\nmetadata:\n name: a\n---\n", + wantNames: []string{"a"}, + }, + { + name: "separator with a comment", + input: "metadata:\n name: a\n--- # next\nmetadata:\n name: b\n", + wantNames: []string{"a", "b"}, + }, + { + name: "comment-only and empty documents are skipped", + input: "# header\n---\n\n---\nmetadata:\n name: a\n---\n# trailer\n", + wantNames: []string{"a"}, + }, + { + name: "three dashes inside a block scalar do not split", + input: "metadata:\n name: a\ndata:\n script: |\n echo one\n ---x\n", + wantNames: []string{"a"}, + }, + { + name: "CRLF line endings", + input: "metadata:\r\n name: a\r\n---\r\nmetadata:\r\n name: b\r\n", + wantNames: []string{"a", "b"}, + }, + { + name: "empty stream", + input: "", + wantNames: nil, + }, + { + name: "malformed YAML names the document", + input: "metadata:\n name: a\n---\nmetadata: [unclosed\n", + wantErr: "document 2", + }, + { + name: "a list is not an object", + input: "- a\n- b\n", + wantErr: "not a mapping", + }, + { + name: "a scalar is not an object", + input: "hello\n", + wantErr: "not a mapping", + }, + { + name: "a separator with content is an error, not a lost document", + input: "metadata:\n name: a\n--- {metadata: {name: b}}\n", + wantErr: "document separator", + }, + { + name: "a separator with a block scalar is an error", + input: "metadata:\n name: a\n--- |\n text\n", + wantErr: "document separator", + }, + { + name: "duplicate keys are rejected", + input: "metadata:\n name: a\n name: b\n", + wantErr: "document 1", + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + objs, err := DecodeYAMLStream(strings.NewReader(tt.input)) + if tt.wantErr != "" { + if err == nil || !strings.Contains(err.Error(), tt.wantErr) { + t.Fatalf("err = %v, want it to contain %q", err, tt.wantErr) + } + return + } + if err != nil { + t.Fatalf("DecodeYAMLStream: %v", err) + } + var names []string + for _, o := range objs { + names = append(names, o.Name()) + } + if strings.Join(names, ",") != strings.Join(tt.wantNames, ",") { + t.Errorf("names = %q, want %q", names, tt.wantNames) + } + }) + } +} + +func TestDecodeYAMLStreamKeepsLargeIntegers(t *testing.T) { + objs, err := DecodeYAMLStream(strings.NewReader("spec:\n big: 9007199254740993\n ratio: 0.5\n")) + if err != nil { + t.Fatalf("DecodeYAMLStream: %v", err) + } + data, err := json.Marshal(objs[0]) + if err != nil { + t.Fatalf("marshal: %v", err) + } + if want := `{"spec":{"big":9007199254740993,"ratio":0.5}}`; string(data) != want { + t.Errorf("JSON = %s, want %s", data, want) + } +} diff --git a/internal/wasmbin/testdata/base/configmap.yaml b/internal/wasmbin/testdata/base/configmap.yaml new file mode 100644 index 0000000..dc9689a --- /dev/null +++ b/internal/wasmbin/testdata/base/configmap.yaml @@ -0,0 +1,6 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: greeting +data: + message: hello diff --git a/internal/wasmbin/testdata/base/kustomization.yaml b/internal/wasmbin/testdata/base/kustomization.yaml new file mode 100644 index 0000000..5b0c161 --- /dev/null +++ b/internal/wasmbin/testdata/base/kustomization.yaml @@ -0,0 +1,2 @@ +resources: + - configmap.yaml diff --git a/internal/wasmbin/testdata/overlay/kustomization.yaml b/internal/wasmbin/testdata/overlay/kustomization.yaml new file mode 100644 index 0000000..c7aab45 --- /dev/null +++ b/internal/wasmbin/testdata/overlay/kustomization.yaml @@ -0,0 +1,3 @@ +namePrefix: overlay- +resources: + - ../base diff --git a/internal/wasmbin/testdata/tekton/kustomization.yaml b/internal/wasmbin/testdata/tekton/kustomization.yaml new file mode 100644 index 0000000..1e0e2b4 --- /dev/null +++ b/internal/wasmbin/testdata/tekton/kustomization.yaml @@ -0,0 +1,5 @@ +# A trimmed copy of https://github.com/Xe/x/tree/master/.tekton. +namespace: ci + +resources: + - x.yaml diff --git a/internal/wasmbin/testdata/tekton/testrun.yaml b/internal/wasmbin/testdata/tekton/testrun.yaml new file mode 100644 index 0000000..ebe05aa --- /dev/null +++ b/internal/wasmbin/testdata/tekton/testrun.yaml @@ -0,0 +1,14 @@ +apiVersion: tekton.dev/v1 +kind: PipelineRun +metadata: + generateName: x-m- + namespace: ci + +spec: + params: + - name: commit + value: master + - name: branch + value: master + pipelineRef: + name: xe-x-build-test diff --git a/internal/wasmbin/testdata/tekton/x.yaml b/internal/wasmbin/testdata/tekton/x.yaml new file mode 100644 index 0000000..51a2a89 --- /dev/null +++ b/internal/wasmbin/testdata/tekton/x.yaml @@ -0,0 +1,24 @@ +apiVersion: tekton.dev/v1beta1 +kind: Pipeline +metadata: + name: xe-x-build-test + namespace: ci + +spec: + params: + - name: "branch" + type: string + - name: "commit" + type: string + workspaces: + - name: repo + tasks: + - name: clone-repo + taskRef: + name: git-clone-naive + workspaces: + - name: output + workspace: repo + params: + - name: revision + value: $(params.commit) diff --git a/internal/wasmbin/wasmbin.go b/internal/wasmbin/wasmbin.go new file mode 100644 index 0000000..5a0b1d8 --- /dev/null +++ b/internal/wasmbin/wasmbin.go @@ -0,0 +1,226 @@ +// Package wasmbin runs WASI programs from directories on the host as Kefka +// shell commands, so a hook can use a program such as kustomize that the +// daemon binary does not embed. +// +// Load lists the directories once. Each *.wasm file in them becomes a command +// named after the file without the extension. When two directories have the +// same name, the first one wins, as in PATH. A program is read and compiled on +// its first run, not by Load, and every shell shares the compiled form. With a +// cache directory, the compiled form also survives a restart. +// +// The image ships the repository's bin directory, whose .wasm files are Git +// LFS objects. A checkout without LFS hydration has pointer files there, and +// Exec then fails with an error that says so. +package wasmbin + +import ( + "bytes" + "context" + "errors" + "fmt" + "io/fs" + "maps" + "os" + "path/filepath" + "slices" + "strings" + "sync" + + "github.com/Xe/kefka/command" + "github.com/Xe/kefka/command/registry" + "github.com/Xe/kefka/wasm/billyfs" + "github.com/tetratelabs/wazero" + "github.com/tetratelabs/wazero/experimental/sysfs" + "github.com/tetratelabs/wazero/imports/wasi_snapshot_preview1" + wsys "github.com/tetratelabs/wazero/sys" + "mvdan.cc/sh/v3/interp" +) + +// wasmMagic starts every WebAssembly binary module. +var wasmMagic = []byte("\x00asm") + +// Set is the WASI programs found in a list of directories. A nil *Set has no +// programs. Its methods are safe for concurrent use. +type Set struct { + runtime wazero.Runtime + cache wazero.CompilationCache // nil without a cache directory + modules map[string]*module +} + +// Load lists the *.wasm files in dirs and returns them as a Set. It does not +// read the files. Empty entries and directories that do not exist are +// skipped, as in PATH. When cacheDir is not empty, compiled programs are +// also kept there, and Load creates it if necessary. +func Load(ctx context.Context, dirs []string, cacheDir string) (*Set, error) { + modules := map[string]*module{} + for _, dir := range dirs { + dir = strings.TrimSpace(dir) + if dir == "" { + continue + } + entries, err := os.ReadDir(dir) + if errors.Is(err, fs.ErrNotExist) { + continue + } + if err != nil { + return nil, fmt.Errorf("wasmbin: %w", err) + } + for _, e := range entries { + name, ok := strings.CutSuffix(e.Name(), ".wasm") + if !ok || name == "" || modules[name] != nil { + continue + } + p := filepath.Join(dir, e.Name()) + // Stat, not the entry type, so that a symlink to a program counts. + info, err := os.Stat(p) + if err != nil { + return nil, fmt.Errorf("wasmbin: %w", err) + } + if !info.Mode().IsRegular() { + continue + } + modules[name] = &module{name: name, path: p} + } + } + + // Close a running instance when its context ends, so -hook-timeout can + // stop a long build. + config := wazero.NewRuntimeConfig().WithCloseOnContextDone(true) + var cache wazero.CompilationCache + if cacheDir != "" { + var err error + if cache, err = wazero.NewCompilationCacheWithDir(cacheDir); err != nil { + return nil, fmt.Errorf("wasmbin: compilation cache: %w", err) + } + config = config.WithCompilationCache(cache) + } + runtime := wazero.NewRuntimeWithConfig(ctx, config) + if _, err := wasi_snapshot_preview1.Instantiate(ctx, runtime); err != nil { + _ = runtime.Close(ctx) + if cache != nil { + _ = cache.Close(ctx) + } + return nil, fmt.Errorf("wasmbin: %w", err) + } + for _, m := range modules { + m.runtime = runtime + } + return &Set{runtime: runtime, cache: cache, modules: modules}, nil +} + +// Names returns the command names in sorted order. +func (s *Set) Names() []string { + if s == nil { + return nil + } + return slices.Sorted(maps.Keys(s.modules)) +} + +// Register adds every program in s to reg. A program replaces a command of +// the same name that reg already has. +func (s *Set) Register(reg *registry.Impl) { + if s == nil { + return + } + for name, m := range s.modules { + reg.Register(name, m) + } +} + +// Close releases the compiled programs. Commands from s fail after it. +func (s *Set) Close(ctx context.Context) error { + if s == nil { + return nil + } + err := s.runtime.Close(ctx) + if s.cache != nil { + err = errors.Join(err, s.cache.Close(ctx)) + } + return err +} + +// module is one program in a Set. It is not kefka's generic wasmcommand, which +// neither sets the guest PWD nor stops an instance when its context ends. +type module struct { + name string + path string + runtime wazero.Runtime + + // mu makes concurrent first runs wait for one compile. Only a success is + // kept, so a failed read is tried again on the next run. + mu sync.Mutex + compiled wazero.CompiledModule +} + +func (m *module) compile() (wazero.CompiledModule, error) { + m.mu.Lock() + defer m.mu.Unlock() + if m.compiled != nil { + return m.compiled, nil + } + bin, err := os.ReadFile(m.path) + if err != nil { + return nil, err + } + if !bytes.HasPrefix(bin, wasmMagic) { + return nil, fmt.Errorf("%s is not a WebAssembly module; it is probably a Git LFS pointer, so run git lfs pull and rebuild", m.path) + } + // Not the caller's context: a compile that a hook timeout stopped would + // only have to start again on the next run. + compiled, err := m.runtime.CompileModule(context.Background(), bin) + if err != nil { + return nil, err + } + m.compiled = compiled + return compiled, nil +} + +// Exec runs the program with args against ec.FS mounted at /. +func (m *module) Exec(ctx context.Context, ec *command.ExecContext, args []string) error { + compiled, err := m.compile() + if err != nil { + return fmt.Errorf("%s: %w", m.name, err) + } + if err := ctx.Err(); err != nil { + return fmt.Errorf("%s: %w", m.name, err) + } + + fsConfig := wazero.NewFSConfig().(sysfs.FSConfig). + WithSysFSMount(billyfs.New(ec.FS), "/") + + // Go's wasip1 port takes its working directory from PWD, so a relative + // path such as "." resolves against the shell's directory. + config := wazero.NewModuleConfig(). + WithStdin(ec.Stdin). + WithStdout(ec.Stdout). + WithStderr(ec.Stderr). + WithArgs(append([]string{m.name}, args...)...). + WithName(""). + WithEnv("PWD", ec.GuestPWD()). + WithFSConfig(fsConfig). + WithSysNanosleep(). + WithSysNanotime(). + WithSysWalltime() + if ec.Environ != nil { + for _, name := range []string{"HOME", "TMPDIR"} { + if v := ec.Environ.Get(name); v.IsSet() { + config = config.WithEnv(name, v.String()) + } + } + } + + mod, err := m.runtime.InstantiateModule(ctx, compiled, config) + if err != nil { + if ctxErr := ctx.Err(); ctxErr != nil { + return fmt.Errorf("%s: %w", m.name, ctxErr) + } + if exitErr, ok := errors.AsType[*wsys.ExitError](err); ok { + if code := exitErr.ExitCode(); code != 0 { + return interp.ExitStatus(uint8(code)) + } + return nil + } + return err + } + return mod.Close(ctx) +} diff --git a/internal/wasmbin/wasmbin_test.go b/internal/wasmbin/wasmbin_test.go new file mode 100644 index 0000000..fcfc89b --- /dev/null +++ b/internal/wasmbin/wasmbin_test.go @@ -0,0 +1,476 @@ +package wasmbin + +import ( + "bytes" + "context" + "errors" + "io" + "io/fs" + "maps" + "os" + "path" + "path/filepath" + "slices" + "sort" + "strings" + "sync" + "testing" + + "github.com/Xe/kefka/command" + "github.com/Xe/kefka/command/registry" + "github.com/go-git/go-billy/v6" + "github.com/go-git/go-billy/v6/memfs" + "github.com/go-git/go-billy/v6/util" + "github.com/go-git/go-git/v6/plumbing" + "github.com/go-git/go-git/v6/plumbing/filemode" + "github.com/go-git/go-git/v6/plumbing/object" + "github.com/go-git/go-git/v6/storage" + "github.com/go-git/go-git/v6/storage/memory" + "github.com/tigrisdata/objgit/internal/mountfs" + "github.com/tigrisdata/objgit/internal/treefs" + "mvdan.cc/sh/v3/expand" +) + +// repoBin is the bin directory at the root of the repository, which the +// image copies to /usr/libexec/objgit/bin. +const repoBin = "../../bin" + +// trueWASM is the smallest WASI command: one exported _start that returns. +var trueWASM = []byte{ + 0x00, 0x61, 0x73, 0x6d, 0x01, 0x00, 0x00, 0x00, // magic, version 1 + 0x01, 0x04, 0x01, 0x60, 0x00, 0x00, // type section: () -> () + 0x03, 0x02, 0x01, 0x00, // function section: one function of type 0 + 0x07, 0x0a, 0x01, 0x06, '_', 's', 't', 'a', 'r', 't', 0x00, 0x00, // export _start + 0x0a, 0x04, 0x01, 0x02, 0x00, 0x0b, // code section: an empty body +} + +// lfsPointer is what a checkout without Git LFS hydration has in place of a +// .wasm file. +var lfsPointer = []byte("version https://git-lfs.github.com/spec/v1\noid sha256:1724a4e9\nsize 24380813\n") + +// binDir writes files into a new directory and returns its path. A name that +// ends in a slash makes an empty directory. +func binDir(t *testing.T, files map[string][]byte) string { + t.Helper() + dir := t.TempDir() + for name, data := range files { + p := filepath.Join(dir, filepath.FromSlash(name)) + if strings.HasSuffix(name, "/") { + if err := os.MkdirAll(p, 0o755); err != nil { + t.Fatal(err) + } + continue + } + if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(p, data, 0o644); err != nil { + t.Fatal(err) + } + } + return dir +} + +func load(t *testing.T, cacheDir string, dirs ...string) *Set { + t.Helper() + s, err := Load(context.Background(), dirs, cacheDir) + if err != nil { + t.Fatalf("Load(%q): %v", dirs, err) + } + t.Cleanup(func() { _ = s.Close(context.Background()) }) + return s +} + +func quietExec() *command.ExecContext { + return &command.ExecContext{Stdout: io.Discard, Stderr: io.Discard, FS: memfs.New()} +} + +func TestLoad(t *testing.T) { + tests := []struct { + name string + dirs []map[string][]byte // a nil map is a directory that does not exist + want map[string]int // command name -> index of the directory it comes from + }{ + { + name: "one command for each .wasm file", + dirs: []map[string][]byte{{"kustomize.wasm": trueWASM, "jq.wasm": trueWASM}}, + want: map[string]int{"jq": 0, "kustomize": 0}, + }, + { + name: "other files and directories are skipped", + dirs: []map[string][]byte{{ + "README.md": []byte("# bin\n"), + "kustomize": trueWASM, + ".wasm": trueWASM, + "dir.wasm/": nil, + "nested/jq.wasm": trueWASM, + "yq.wasm": trueWASM, + "notes.wasm.orig": trueWASM, + }}, + want: map[string]int{"yq": 0}, + }, + { + name: "an empty directory has no commands", + dirs: []map[string][]byte{{}}, + want: map[string]int{}, + }, + { + name: "Load does not read the files", + dirs: []map[string][]byte{{"kustomize.wasm": lfsPointer}}, + want: map[string]int{"kustomize": 0}, + }, + { + name: "an earlier directory wins, as in PATH", + dirs: []map[string][]byte{ + {"kustomize.wasm": trueWASM}, + {"kustomize.wasm": trueWASM, "jq.wasm": trueWASM}, + }, + want: map[string]int{"kustomize": 0, "jq": 1}, + }, + { + name: "a directory that does not exist is skipped", + dirs: []map[string][]byte{nil, {"jq.wasm": trueWASM}}, + want: map[string]int{"jq": 1}, + }, + { + name: "no directory exists", + dirs: []map[string][]byte{nil, nil}, + want: map[string]int{}, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + var dirs []string + for _, files := range tt.dirs { + if files == nil { + dirs = append(dirs, filepath.Join(t.TempDir(), "nope")) + continue + } + dirs = append(dirs, binDir(t, files)) + } + s := load(t, "", dirs...) + + want := slices.Sorted(maps.Keys(tt.want)) + if got := s.Names(); !slices.Equal(got, want) { + t.Fatalf("Names() = %q, want %q", got, want) + } + for name, i := range tt.want { + if got, want := s.modules[name].path, filepath.Join(dirs[i], name+".wasm"); got != want { + t.Errorf("%s comes from %s, want %s", name, got, want) + } + } + }) + } +} + +func TestLoadSkipsEmptyEntries(t *testing.T) { + s := load(t, "", "", binDir(t, map[string][]byte{"jq.wasm": trueWASM}), " ") + if got := s.Names(); !slices.Equal(got, []string{"jq"}) { + t.Fatalf("Names() = %q, want [jq]", got) + } +} + +func TestLoadNotADirectory(t *testing.T) { + file := filepath.Join(binDir(t, map[string][]byte{"jq.wasm": trueWASM}), "jq.wasm") + if _, err := Load(context.Background(), []string{file}, ""); err == nil { + t.Fatal("Load of a file returned nil") + } +} + +// stub is a built-in command that a bin directory can replace. +type stub struct{} + +func (stub) Exec(context.Context, *command.ExecContext, []string) error { + return errors.New("stub ran") +} + +func TestRegister(t *testing.T) { + reg := registry.New() + reg.Register("jq", stub{}) + reg.Register("rg", stub{}) + load(t, "", binDir(t, map[string][]byte{"jq.wasm": trueWASM, "true.wasm": trueWASM})).Register(reg) + + for _, name := range []string{"jq", "true"} { + cmd, ok := reg.Get(name) + if !ok { + t.Fatalf("%s is not registered", name) + } + if err := cmd.Exec(context.Background(), quietExec(), nil); err != nil { + t.Errorf("%s: %v", name, err) + } + } + if cmd, _ := reg.Get("rg"); cmd != (stub{}) { + t.Errorf("rg = %T, want the built-in it did not replace", cmd) + } +} + +func TestNilSet(t *testing.T) { + var s *Set + reg := registry.New() + s.Register(reg) + if names := reg.Names(); len(names) != 0 { + t.Errorf("a nil Set registered %q", names) + } + if names := s.Names(); names != nil { + t.Errorf("Names() = %q, want nil", names) + } + if err := s.Close(context.Background()); err != nil { + t.Errorf("Close: %v", err) + } +} + +func TestExecReadsOnFirstUse(t *testing.T) { + dir := binDir(t, map[string][]byte{"true.wasm": lfsPointer}) + s := load(t, "", dir) + if err := os.WriteFile(filepath.Join(dir, "true.wasm"), trueWASM, 0o644); err != nil { + t.Fatal(err) + } + if err := s.modules["true"].Exec(context.Background(), quietExec(), nil); err != nil { + t.Fatalf("Exec after the file changed: %v", err) + } +} + +func TestExecRejectsLFSPointer(t *testing.T) { + s := load(t, "", binDir(t, map[string][]byte{"kustomize.wasm": lfsPointer})) + err := s.modules["kustomize"].Exec(context.Background(), quietExec(), []string{"version"}) + if err == nil || !strings.Contains(err.Error(), "Git LFS pointer") { + t.Fatalf("err = %v, want an error that names the Git LFS pointer", err) + } +} + +func TestExecMissingFile(t *testing.T) { + dir := binDir(t, map[string][]byte{"true.wasm": trueWASM}) + s := load(t, "", dir) + if err := os.Remove(filepath.Join(dir, "true.wasm")); err != nil { + t.Fatal(err) + } + err := s.modules["true"].Exec(context.Background(), quietExec(), nil) + if !errors.Is(err, fs.ErrNotExist) { + t.Fatalf("err = %v, want fs.ErrNotExist", err) + } +} + +func TestExecCancelled(t *testing.T) { + s := load(t, "", binDir(t, map[string][]byte{"true.wasm": trueWASM})) + ctx, cancel := context.WithCancel(context.Background()) + cancel() + if err := s.modules["true"].Exec(ctx, quietExec(), nil); !errors.Is(err, context.Canceled) { + t.Fatalf("err = %v, want context.Canceled", err) + } +} + +func TestCompilationCache(t *testing.T) { + cacheDir := filepath.Join(t.TempDir(), "wasm") + s := load(t, cacheDir, binDir(t, map[string][]byte{"true.wasm": trueWASM})) + if err := s.modules["true"].Exec(context.Background(), quietExec(), nil); err != nil { + t.Fatalf("Exec: %v", err) + } + entries, err := os.ReadDir(cacheDir) + if err != nil { + t.Fatalf("read cache directory: %v", err) + } + if len(entries) == 0 { + t.Error("the compilation cache is empty after a run") + } +} + +// TestRepoBin checks the programs that the image ships. A checkout without +// Git LFS hydration has pointer files here. +func TestRepoBin(t *testing.T) { + s := load(t, "", repoBin) + if !slices.Contains(s.Names(), "kustomize") { + t.Fatalf("Names() = %q, want kustomize", s.Names()) + } + for _, name := range s.Names() { + data, err := os.ReadFile(filepath.Join(repoBin, name+".wasm")) + if err != nil { + t.Fatal(err) + } + if !bytes.HasPrefix(data, wasmMagic) { + t.Errorf("%s.wasm starts with %q, not the WASM magic; run git lfs pull", name, data[:min(len(data), 16)]) + } + } +} + +// repoSet loads repoBin once, so the kustomize tests compile it once. +var repoSet = sync.OnceValues(func() (*Set, error) { + return Load(context.Background(), []string{repoBin}, "") +}) + +func TestKustomize(t *testing.T) { + s, err := repoSet() + if err != nil { + t.Fatal(err) + } + kustomize := s.modules["kustomize"] + + tests := []struct { + name string + dir string // shell working directory, fsys-relative + args []string + wantErr bool + wantStdout []string // substrings of stdout + wantTmp string // file under /tmp that must exist afterwards + }{ + { + name: "relative path from the shell directory", + dir: "src/overlay", + args: []string{"build", "."}, + wantStdout: []string{"kind: ConfigMap", "name: overlay-greeting", "message: hello"}, + }, + { + name: "absolute path", + dir: "src", + args: []string{"build", "/src/base"}, + wantStdout: []string{"name: greeting"}, + }, + { + name: "Xe-style Tekton bundle", + dir: "src", + args: []string{"build", ".tekton"}, + wantStdout: []string{"kind: Pipeline", "name: xe-x-build-test", "namespace: ci", "$(params.commit)"}, + }, + { + name: "output into read-only /src fails", + dir: "src", + args: []string{"build", "base", "-o", "/src/out.yaml"}, + wantErr: true, + }, + { + name: "output into writable /tmp works", + dir: "src", + args: []string{"build", "base", "-o", "/tmp/out.yaml"}, + wantTmp: "tmp/out.yaml", + }, + { + name: "missing directory fails", + dir: "src", + args: []string{"build", "nope"}, + wantErr: true, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + fsys := sandbox(t) + var stdout, stderr bytes.Buffer + err := kustomize.Exec(context.Background(), &command.ExecContext{ + Stdin: strings.NewReader(""), + Stdout: &stdout, + Stderr: &stderr, + Dir: tt.dir, + Environ: expand.ListEnviron("HOME=/tmp", "TMPDIR=/tmp"), + FS: fsys, + }, tt.args) + if (err != nil) != tt.wantErr { + t.Fatalf("err = %v, wantErr %v; stderr:\n%s", err, tt.wantErr, stderr.String()) + } + if tt.wantErr { + t.Logf("stderr: %s", stderr.String()) + } + for _, want := range tt.wantStdout { + if !strings.Contains(stdout.String(), want) { + t.Errorf("stdout lacks %q:\n%s", want, stdout.String()) + } + } + if tt.wantTmp != "" { + data, err := util.ReadFile(fsys, tt.wantTmp) + if err != nil { + t.Fatalf("read %s: %v", tt.wantTmp, err) + } + if !strings.Contains(string(data), "kind: ConfigMap") { + t.Errorf("%s = %q, want the built ConfigMap", tt.wantTmp, data) + } + } + }) + } +} + +// sandbox builds the hook filesystem: testdata as a read-only git tree at +// /src, with tekton/ renamed to .tekton as a repository keeps it, and an +// empty writable /tmp. +func sandbox(t *testing.T) billy.Filesystem { + t.Helper() + files := map[string]string{} + err := filepath.WalkDir("testdata", func(p string, d fs.DirEntry, err error) error { + if err != nil || d.IsDir() { + return err + } + data, err := os.ReadFile(p) + if err != nil { + return err + } + rel := filepath.ToSlash(strings.TrimPrefix(p, "testdata"+string(filepath.Separator))) + rel = strings.Replace(rel, "tekton/", ".tekton/", 1) + files[rel] = string(data) + return nil + }) + if err != nil { + t.Fatalf("read testdata: %v", err) + } + store := memory.NewStorage() + tree, err := object.GetTree(store, putDir(t, store, files, "")) + if err != nil { + t.Fatalf("get tree: %v", err) + } + return mountfs.New(map[string]billy.Filesystem{ + "src": treefs.New(tree), + "tmp": memfs.New(), + }) +} + +// putDir writes the files under dir (a slash path, "" for the root) as git +// objects and returns the tree hash. +func putDir(t *testing.T, store storage.Storer, files map[string]string, dir string) plumbing.Hash { + t.Helper() + prefix := dir + if prefix != "" { + prefix += "/" + } + subdirs := map[string]bool{} + var entries []object.TreeEntry + for p, data := range files { + rest, ok := strings.CutPrefix(p, prefix) + if !ok { + continue + } + if name, _, nested := strings.Cut(rest, "/"); nested { + if !subdirs[name] { + subdirs[name] = true + entries = append(entries, object.TreeEntry{Name: name, Mode: filemode.Dir, Hash: putDir(t, store, files, path.Join(dir, name))}) + } + continue + } + entries = append(entries, object.TreeEntry{Name: rest, Mode: filemode.Regular, Hash: putBlob(t, store, data)}) + } + sort.Sort(object.TreeEntrySorter(entries)) + o := store.NewEncodedObject() + if err := (&object.Tree{Entries: entries}).Encode(o); err != nil { + t.Fatalf("encode tree: %v", err) + } + h, err := store.SetEncodedObject(o) + if err != nil { + t.Fatalf("set tree: %v", err) + } + return h +} + +func putBlob(t *testing.T, store storage.Storer, data string) plumbing.Hash { + t.Helper() + o := store.NewEncodedObject() + o.SetType(plumbing.BlobObject) + w, err := o.Writer() + if err != nil { + t.Fatalf("blob writer: %v", err) + } + if _, err := io.WriteString(w, data); err != nil { + t.Fatalf("blob write: %v", err) + } + _ = w.Close() + h, err := store.SetEncodedObject(o) + if err != nil { + t.Fatalf("set blob: %v", err) + } + return h +} diff --git a/manifest/kustomization.yaml b/manifest/kustomization.yaml index e094dd3..04cce05 100644 --- a/manifest/kustomization.yaml +++ b/manifest/kustomization.yaml @@ -69,6 +69,14 @@ configMapGenerator: # Run .objgit/hooks/receive-pack from the repository after a push. See # docs/usage/hooks.md. - ALLOW_HOOKS=false + # Give hooks the kube:apply and tekton:pipelinerun commands, with the + # objgitd ServiceAccount. Needs ALLOW_HOOKS=true and the RBAC in + # manifest/tekton-rbac/. Anybody who can push can then use that + # ServiceAccount. See docs/usage/kubernetes-hooks.md. + - ALLOW_KUBERNETES=false + # Keep compiled WASI programs, such as kustomize, on the PVC. Without + # it, the first hook after each restart spends seconds on a compile. + - WASM_CACHE_DIR=/var/cache/objgit/wasm - SLOG_LEVEL=INFO secretGenerator: diff --git a/manifest/tekton-rbac/kustomization.yaml b/manifest/tekton-rbac/kustomization.yaml new file mode 100644 index 0000000..f72b622 --- /dev/null +++ b/manifest/tekton-rbac/kustomization.yaml @@ -0,0 +1,19 @@ +# Example RBAC for the kube:apply and tekton:pipelinerun hook commands +# (objgitd -allow-kubernetes). It lets the objgitd ServiceAccount in the +# objgit namespace manage Tekton Pipelines and Tasks, and create +# PipelineRuns, in the ci namespace. Nothing else. +# +# Apply it separately from manifest/: +# +# kubectl apply -k manifest/tekton-rbac/ +# +# This directory has no namespace: field on purpose. The Role and the +# RoleBinding go in ci, and the subject stays in objgit. The namespace: +# objgit field of manifest/kustomization.yaml moves all of them to objgit. +# +# See docs/usage/kubernetes-hooks.md. +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +resources: + - role.yaml diff --git a/manifest/tekton-rbac/role.yaml b/manifest/tekton-rbac/role.yaml new file mode 100644 index 0000000..de59550 --- /dev/null +++ b/manifest/tekton-rbac/role.yaml @@ -0,0 +1,39 @@ +# kube:apply uses Server-Side Apply. Kubernetes checks "create" when the +# object does not exist yet, and "patch" when it does. tekton:pipelinerun +# only creates new runs, so pipelineruns get "create" and nothing more. +# +# Other kinds or namespaces in a kube:apply stream need their own Role. The +# API server refuses them with 403 Forbidden, and the hook prints that +# message. +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: objgitd-tekton + namespace: ci + labels: + app.kubernetes.io/name: objgitd + app.kubernetes.io/part-of: objgit +rules: + - apiGroups: ["tekton.dev"] + resources: ["pipelines", "tasks"] + verbs: ["create", "patch"] + - apiGroups: ["tekton.dev"] + resources: ["pipelineruns"] + verbs: ["create"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: objgitd-tekton + namespace: ci + labels: + app.kubernetes.io/name: objgitd + app.kubernetes.io/part-of: objgit +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: objgitd-tekton +subjects: + - kind: ServiceAccount + name: objgitd + namespace: objgit