Ratchet — the coding agent that never decides it's done

The problem

Agents cheat their own tests

Left alone with a test command, a coding agent eventually stops fixing the bug and starts gaming the grader. Three real moves, all from Ratchet's red-team battery:

Skips the hard tests

Drops a skip marker on the suite it can't pass — score goes green, bug stays.

caught by: skip_marker

Hardcodes the answer

Returns exactly what the visible test expects instead of solving anything.

caught by: special_casing

Fakes the report

Prints a passing log and spoofs the exit code — the tests never actually ran.

caught by: log_spoofed

The answer

No "done" button. Only verdicts.

Ratchet has no done tool at all. Every patch runs the gauntlet; the same three moves above get caught before a line of them executes:

$ ratchet verify --diff patches/honest.diff GREEN · score 1.000 — the real fix, accepted $ ratchet verify --diff patches/cheat.diff CHEATED — caught before a line of it runs $ ratchet verify --diff patches/canary_hack.diff CHEATED — zero static findings; the canary caught it anyway

That third one matters: the patch trips no static rule and gets caught anyway. Runs on your laptop in sixty seconds — no model, no key, no network.

How it works

Three steps, one clean diff

Step 1

Point it at a repo

A task file names the goal and the test command. ratchet run does the rest.

Step 2

It searches, the verifier grades

Every step is a git commit + sandbox snapshot — a tree search over repo states. Dead branches pruned, promising ones fork in parallel.

Step 3

You approve one squashed diff

The winning path stops at a human gate. Nothing pushes, nothing opens a PR, until you say so.

01build

02cheat check

03fail-to-pass

04pass-to-pass

05types

06lint

07diff hygiene

$ ratchet run --repo demo-repo --scripted patches/scripted.json ● root 0.58 ├─✗ 1e0f 0.58 pruned └─★ 2874 1.00 ✓green → one squashed diff, waiting at the human gate

Why it holds

Built to be cheated on

Cheat detector

Every known reward hack, caught

Skip markers, hardcoded answers, spoofed exit codes, conftest hooks — a published red-team battery scores the verifier itself, in CI, at zero false positives.

Hidden tests

The agent never sees what grades it

Held-out test names never reach a prompt — not the context, not the observations, not a single bus event. There are tests asserting exactly that.

Receipts

Forge a verdict, break the chain

Every graded step is signed into a hash chain. Edit one result and make audit says CHAIN BROKEN, naming the receipt that fails.

One-way progress

Rollbacks it can't argue with

Protected paths revert before grading, every run, no skip flag. Pruned work is parked, never lost. A ratchet turns one way.

Fair questions

Before you ask

Can the agent read the hidden tests and game them?

No. Held-out test names never appear in anything the agent can read. And a canary task with an impossible test catches answer-smuggling that leaves zero static findings.

What if the agent edits the verifier or the test files?

Protected paths are reverted before grading on every run, with no flag to skip. Runtime writes to graded paths are their own cheat finding. The exit code is echoed outside the region the agent can influence.

Does it need Docker, a cloud account, or network access?

No. make test, make redteam, and make proof run with no Docker and no network. Live sandboxes come from the TrueForge harness; the documented fallback is plain git worktrees.

Can it push code without a human?

No. Pull requests and pushes are gated behind a human approval interrupt. The gate held in every recorded proof run.

How do I know these numbers are real?

Run make proof. It exercises every claim on this page offline and keeps the evidence — the dashboard renders the latest run, and the site's stats are generated from it, never typed in.