Cortex is a governance layer — hooks, policies, validation, long-horizon planning — that wraps any coding agent (Codex, Claude Code, OpenCode). This benchmark measures, head to head, how much safer and more capable a base model becomes once Cortex is in front of it. Click any number to see the full, reproducible detail.
Each harness is attacked with obfuscation techniques across direct, repo-file, tool-output and memory surfaces (text / image / audio). We measure Attack Success Rate (ASR) — how often the harness is talked into an unsafe action (lower is safer) — with Wilson 95% CIs, and track over-refusal so that refusing benign tasks does not count as 'safe'.
Harnesses solve real coding tasks; we run real static analysis (Bandit / Semgrep, CWE-mapped), dependency checks, and the hidden acceptance tests (pytest), plus an LLM judge for architecture / readability. Reports requirement coverage, functional pass rate, vulnerabilities and maintainability.
Each harness builds a real ChatGPT-style assistant app backed by a local small model (offline, no external API); it is then built, served and Playwright-measured. Scored on a capability composite — feature completeness (a live chat round-trip through the local model, plus persistence/streaming), build & serve, plan/trajectory adherence, visual styling (CLIP) and honesty — where raw harnesses fade across the build horizon and the Cortex governance loop holds completeness.
One open-ended brief to build a whole storefront (backend + React/Vite frontend + PWA + cart + checkout + Stripe test) in a hardened Docker sandbox; scored on a build-gated composite of functional e2e, VERTEX trajectory similarity, security (Semgrep), visual fidelity and architecture.