VSArena

The official open benchmark for embodied (VLA) policies.

Server-authoritative stacking evaluation · signed results · public ELO the client cannot write.

Site · Leaderboard · Work cell · Protocol · Eval integrity · SDK ·

What it is

VSArena is a public evaluation product for spatial / VLA policies: one stacking task, one wire protocol, one leaderboard that only the official harness may update.

  Policy (Python SDK)  ──state: RGB 128×128 + instruction──►  Harness (Rapier 60 Hz)
                       ◄──action: joints / ee_delta──────────
                                                              │
                         SHA-256 digest + Ed25519 DSSE        ▼
                                                    Official ingest → public ELO
                                                    (browser / Studio cannot write)

Contract	Value
Product	`1.0.0`
Task	`block_stacking.v1`
Observation schema	`obs.v1` — RGB + language, no cube poses on the public track
Action schema	`action.v1` — validated before physics
Physics	Rapier `0.20.0` · 60 Hz
Replay	`vsarena-replay-v1` (sparse privileged poses, not RGB)
Public ELO track	VLA only · binary outcome (full tower = 1, else 0)

Studio (/simulation) is a public work cell for teleop, demos, and debug. It does not write public ELO. Official scores come from the hosted harness.

Why reliability is the product

Private sims and PDF tables are hard to audit. VSArena publishes the judge path: shared weekly seed, control arm, signed manifest, VLA-only ingest, and a board the browser cannot touch.

Eval integrity (shipped)

Guarantee	Implementation
Provenance on every official result	`product 1.0.0`, Rapier `0.20.0`, 60 Hz, git SHA, Node, mode, latency budget, policy Hz, `sampler_seed`, scene id / seed / hash / arm — also on harness `GET /health`
Pinned sampler seed	`hash("eval:" + ISO-week)` — same seed for every agent that week, not derived from the agent name
Benign control arm	Production: public-canonical episode (no held-out, no jitter) before the scored episode, same policy / socket; row reports control vs scored fail rates
Signed run manifest	SHA-256 digest of the canonical manifest (ingest rejects mismatch) + Ed25519 over DSSE PAE; verify with published key `GET /api/eval/keys` (and harness `/health`). Ingest secret = channel auth, not the receipt
Agent ownership	Official `hello.agent` must exist on `/account` and match the profile behind `api_key`; Postgres rejects stranger names
VLA-only public ELO	`mode=state` may run for debug; harness skips ingest; `/api/matches` rejects non-VLA provenance
Binary ELO	Full stack on the pad only; partial towers do not move rating
Held-out scored set (prod)	Production defaults to `held_out` and requires `VSARENA_HELD_OUT_JSON` (or explicit staging allow-in-repo flag)
Action contract	Invalid actions (`NaN`, unknown joints, oversized `ee_delta`) never touch Rapier; budget → `protocol.invalid_action` (no ELO)
Latency budget	VLA: 2 s / 5 Hz (8 late → `policy.timeout`). State debug: 150 ms / 20 Hz
Protocol / disconnect	`protocol.`* and mid-match disconnect do not ingest ELO — only policy outcomes on completed official runs
Failure taxonomy	Dotted codes: `policy.`* / `protocol.` / `harness.`
Run inspection	`/runs/[id]` + `GET /api/matches/[id]` — score, termination, counters, provenance, pose replay
Match queue	Single world; FIFO ≤ 8 with `harness.queued` updates

Full matrix and honest limits: docs/eval-integrity.md.

What we do not claim

Bit-identical Rapier trajectories across OS/CPU after many steps
That in-repo held-out layouts are secret (they are open; production should use private VSARENA_HELD_OUT_JSON)
That ColorSeek / Baseline-IK are neural VLAs, or that Studio demos count toward public ELO
RGB video replay (V1 replay is sparse poses only)

V1 surface

In V1	Out of V1
Official harness + ingest → public board	Multi-task suite
Work cell (Rapier) for debug / demos	Head-to-head 1v1 product
Python SDK dry-run + live	PyPI package
Manifest, Ed25519 receipt, sparse replay

Run the work cell locally

git clone https://github.com/ONISCOR/VSArena.git
cd VSArena
cp .env.example .env.local   # Supabase + secrets
npm install
npm run dev

Open http://localhost:3000/simulation. Teleop / Baseline-IK / ColorSeek / demo record are local tools — no public ELO.

npm test
npm run harness   # http://127.0.0.1:8787/health · ws://127.0.0.1:8787

Official hosted harness: wss://vsarena-harness.onrender.com
Health: https://vsarena-harness.onrender.com/health
Spectator (control arm / weekly highlights): simulation?view=live · wss://…/spectate
Deploy kit: deploy/harness/README.md.

Environment

Variable	Purpose
`NEXT_PUBLIC_SUPABASE_URL`	Supabase URL
`NEXT_PUBLIC_SUPABASE_ANON_KEY`	Anon key (browser)
`SUPABASE_SERVICE_ROLE_KEY`	Server · profiles / ingest
`HARNESS_INGEST_SECRET`	≥16 chars · header `x-vsarena-ingest` (channel)
`VSARENA_RESULTS_ED25519_PRIVATE` / `_PUBLIC`	Signs / publishes official receipts
`VSARENA_APP_URL`	Where the harness POSTs results
`VSARENA_HARNESS_URL`	SDK live socket (`wss://…` or local `ws://127.0.0.1:8787`)
`NEXT_PUBLIC_SITE_URL`	Canonical origin
`VSARENA_SCENE_SET`	`public` or `held_out`
`VSARENA_HELD_OUT_JSON`	Private three-cube layouts on the harness host

Apply supabase/schema.sql. GitHub OAuth callbacks: http://localhost:3000/auth/callback, https://www.vsarena.app/auth/callback.

Submit a policy

pip install -e sdk/python
python -m vsarena          # dry-run sanity check

from vsarena import Agent, run_match

class MyAgent(Agent):
    def act(self, state: dict) -> dict:
        # VLA: state["instruction"] + state["images"]["scene"]
        # scene.blocks is empty on the public track by design
        joints = state["scene"]["joint_states"]
        return {"joint_targets": dict(joints), "gripper_state": "open"}

print(run_match(MyAgent(), dry_run=True, mode="vla"))

Live (writes ELO when ingest + signing are configured):

pip install -e "sdk/python[live]"
export VSARENA_API_KEY=…   # from /account
export VSARENA_HARNESS_URL=wss://vsarena-harness.onrender.com

run_match(
    MyAgent(),
    dry_run=False,
    mode="vla",
    api_key="…",
    agent_name="MyAgent",
)

Contract: docs/harness.md · SDK: docs/sdk.md.

Scoring

flowchart LR
  P[Policy] -->|action| H[Harness]
  H -->|RGB + instruction| P
  H -->|privileged poses| S[Scoring]
  S -->|digest + Ed25519| I[Ingest]
  I --> B[Public ELO]
  Client -.->|blocked| B

Spatial accuracy — distance / orientation to stack slots (telemetry)
Task completion — full tower cyan → orange → magenta on the pad
ELO — binary vs house 1200 · only via signed harness ingest

Repository layout

app/            Site, work cell, API, run detail
components/     UI + R3F (no physics authority)
simulation/     Rapier world, FK/IK, grasp
lib/harness/    Protocol codec
lib/eval/       Provenance, taxonomy, scenes, replay, product versions
lib/scoring/    Pure scoring + ELO
lib/vision/     128×128 VLA raster
lib/agents/     Reference policies (Baseline-IK, ColorSeek)
sdk/python/     Agent SDK
server/         Official WebSocket harness
deploy/harness/ Hosting kit
docs/           Protocol, integrity
supabase/       Schema + RLS
public/brand/   Product mark (cube icon)

Roadmap

Work cell + Rapier 60 Hz
VLA observation track + reference policies
Harness protocol + Python SDK
Public leaderboard (VLA ingest only)
Eval integrity: weekly seed, control arm, SHA-256 + Ed25519, taxonomy, provenance, replay
Product 1.0.0 / task block_stacking.v1
PyPI vsarena
Additional tasks under the same protocol
Hosted capacity / private scene packs

Contributing

npm test
npx tsc --noEmit

CONTRIBUTING.md · CODE_OF_CONDUCT.md · SECURITY.md · SUPPORT.md · GOVERNANCE.md · CITATION.cff

License

MIT — LICENSE. ONISCOR · Aran Kair.

One task. One protocol. One board the client cannot fake.

Navigation Menu

Repository navigation

About

Repository files navigation

VSArena

What it is

Why reliability is the product

Eval integrity (shipped)

What we do not claim

V1 surface

Run the work cell locally

Environment

Submit a policy

Scoring

Repository layout

Roadmap

Contributing

License

About

Releases

Contributors

Languages

Name		Name	Last commit message	Last commit date
.github		.github
app		app
components		components
deploy/harness		deploy/harness
docs		docs
lib		lib
public		public
sdk/python		sdk/python
server		server
simulation		simulation
supabase		supabase
.dockerignore		.dockerignore
.env.example		.env.example
.eslintrc.json		.eslintrc.json
.gitignore		.gitignore
CITATION.cff		CITATION.cff
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
Dockerfile		Dockerfile
GOVERNANCE.md		GOVERNANCE.md
LICENSE		LICENSE
README.md		README.md
SECURITY.md		SECURITY.md
SUPPORT.md		SUPPORT.md
components.json		components.json
middleware.ts		middleware.ts
next.config.mjs		next.config.mjs
package-lock.json		package-lock.json
package.json		package.json
postcss.config.mjs		postcss.config.mjs
render.yaml		render.yaml
tailwind.config.ts		tailwind.config.ts
tsconfig.json		tsconfig.json
vitest.config.ts		vitest.config.ts

Latest commit

History

Folders and files

Repository files navigation

VSArena

What it is

Why reliability is the product

Eval integrity (shipped)

What we do not claim

V1 surface

Run the work cell locally

Environment

Submit a policy

Scoring

Repository layout

Roadmap

Contributing

License

About

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

Watchers

Forks

Releases

Contributors

Languages