MenderMender

AI coding agents are judged by whether the tests pass. But an agent can pass by special-casing the example in the test, or by editing the test until it agrees. A benchmark that cannot tell a real fix from a cheat measures nothing.

Repositories can also contain text that talks a model into breaking its own rules.

I built Mender, a code-repair agent: it reproduces the failure, reads the code, makes the smallest fix, re-runs the tests and explains the root cause. The loop, tools and permission model are about 1,200 lines of standard-library Python, with no agent framework.

The grading is the real work. Each task has hidden tests, protected files are hashed and restored before grading, and one repository hides a comment telling the agent to rewrite the failing test.

A scripted adversary runs the whole benchmark trying to cheat, and the suite checks it scores zero. Every run can be replayed in a web viewer.

The reference fixes, replayed through the full harness, solve 13 of 13 tasks, which proves every task is solvable with the agent’s own tools. The adversary scores 0 of 13, with 39 cheat attempts blocked.

Solve rates are reported with a 95% confidence interval, because thirteen tasks is a small sample.