01 SEALED
Your grader, run as-is
Every candidate runs in a fresh container from the task’s image, with the network off and all capabilities dropped, using the task’s own eval script and grading.
78 of 191 tasks paid full reward. For a wrong patch.
From our audit of a 200-task sample of SWE-bench Verified, the benchmark much of the field reports against. In each case the task’s own tests passed a patch that fails what the issue asks for. Count every wrong patch, not only the ones the issue rules out, and it is 143 of 191.
password-reset task · ID withheld pending maintainer noticesealed · network off
Finding confirmed, then closed. The new test passes the maintainers’ fix and fails every wrong patch the audit found.
The method
We attack your grader. Then we prove what we find.
Each task runs through its own grader, unchanged, in a sealed container. We hand it wrong solutions and record which ones it pays for. Nothing counts as a finding until an independent check shows the patch is wrong.
02 ATTACK
Wrong solutions, two ways
Pieces of the reference fix with parts removed, needing no model, and plausible bugs and partial fixes written by a model. Each source is reported on its own.
03 PROOF
An independent check
A paid-out patch must differ from the reference fix in code. Then a separate check must pass on the reference fix, fail on the unfixed code, and fail on the patch.
04 SCOPE
Required, or just a gap
Each finding is labeled: the issue as written requires the missing behavior (strict), or it is a real test gap outside the issue (broad). Both are reported, separately.
05 SKEPTIC
Re-run before it reaches you
A sample of findings is re-run in fresh containers, check scripts included. Every number in the report traces to a run log, or the report is held.
06 REPAIR
Tests that close the hole
New tests that pass the reference fix and fail every wrong patch we found, verified in the container, then attacked again with fresh wrong patches.
What you get
01
A verdict per task
KEEP, FIX or DROP, with the wrong patch that passed, the hash of its run log, and the check that shows it is wrong.
02
Repaired tests
A test patch for every FIX task, run against the reference fix and every known wrong patch before you see it.
03
A walkthrough
We go through the findings with your team, task by task, and show how to reproduce each one.
The evidence
Measured on a benchmark you can check.
We ran the auditor on public tasks first, so every claim here can be reproduced from the run artifacts.
STRICT SHARE
40.8%78 of 191 tested tasks in a 200-task sample of SWE-bench Verified paid out for a patch that fails the issue.
OUTSIDE REVIEW
19 of 20Sampled findings judged genuine by all three independent AI reviewers (two Claude, one ChatGPT) reading the evidence cold.
REPAIR TRIAL
21 of 21Known wrong patches blocked by verified new tests across six repaired tasks.
For comparison, a June 2026 audit (Rajan) reported at least 28.5% on its 50-task sample of the same benchmark. On those same tasks our strict count is 17 of 48; theirs was 14 of 49.
How we work
Evidence first. Disclosure first.
A reward signal is only as good as the tests behind it. A model trained on a grader that pays for wrong code learns the wrong lesson, and the benchmark scores built on that grader stop meaning what they say.
- Public tasks only, unless you engage us on your own environments.
- Maintainers of public datasets get 30 days’ notice before anything is published.
- No finding is sent before an independent check confirms it.
- Every run is sealed: a fresh container, network off, capabilities dropped.
A plumb line checks against a true vertical. We check every grader against the reference fix, not the tests’ own say-so.
Does your grader pay for wrong code?
Send us a sample of your environment’s tasks. We’ll show you what passes that shouldn’t, and how to close it.
Request an audit or write to shane@plumblinegrader.com