Prove Your Check Can Fail

I've built for the web since 1994. Today I run many client projects at the same time, with AI agents doing much of the work under my direction, inside a system I built called HALO. Much of what keeps that system honest is checking: the tests, reviews and quality gates that decide whether a piece of work is done.

A quality gate is any automatic step that has to pass before work moves forward. This post is for anyone who lets a gate, a test or an AI reviewer tell them the work is fine. I'll make the case through my own checks, the ones that said yes when the right answer was no. Everything here is how I work as of October 2026.

Why does a passing check tell you nothing until you've seen it fail?

A pass from a check you've never seen fail has two possible meanings: the work is right, or the check is blind. From the outside the two look the same. A blind check passes every run, whatever state the work is in. Only a run where the check fails on purpose tells you which kind you have.

My own review decides when a job is done, and a check only gets a say in that once it has shown it can fail. I've written about why writing code got cheap, and checking it did not. A check that has never failed doesn't remove that cost. It moves it to whoever finds the defect later.

What did my own checks miss?

My review had passed four defects in work I'd already called finished, and I caught them on 2026-09-23. All four got through the same way: I had checked the code and not the result. Checking the code means reading what was written and judging that it looks right. Checking the result means running the work and looking at what it actually does.

One was a status panel. I had shown it against invented test data, the made-up records you use before real ones exist. With that data on screen, a finished job that would display '0 of 10 done' never appeared.

Another was a database module, the part of the software that talks to the database. The module read correctly. The running server leaked a connection on every poll. A leaked connection is one the server opens to the database and never closes. The server stopped answering twice.

A third was a macOS call that only exists on Linux. A call is an instruction a program hands to the operating system. This one read as correct and silently did nothing.

An earlier case, dated 2026-08-10, had already shown me the pattern. My cycle-record check had been reporting zero records and returning success every time, in all six workspaces. One of those was a proposal run that went through ten stages and left eleven summaries and no plans.

How do I prove a check can fail?

The test I use has three steps: plant the defect the check is for, confirm the check names it, restore the original. I run it every time I add a check, before I trust a single pass from it.

On 2026-09-28 I made this a written rule for every new assertion. An assertion is a single line inside a test that states something that must be true about the work. Under the rule, a brand-new check that passes on its first run is unproven until it has been through those three steps.

Step two asks for more than a failure: the check has to name the defect I planted. A check that fails for some other reason hasn't shown it can see this one. If it does not name the planted defect, it is not a check yet.

The rule exists because of a layout check on a client build. A phone title broke in the wrong place on every iPhone, and the check called it correct. Three things inside it were wrong. It counted lines of text by reading the rectangles that outline an element on the page, and that count is always 1. It read a style value with the wrong call, which hands back the text as it was declared instead of a number. Its selector, the instruction that tells a check which part of the page to measure, grabbed a wrapper around the title that measured right by coincidence at one screen width.

You can ask for this without reading any code. Ask whoever owns a check to show you a run where it failed on the problem it guards. If they can't, treat its passes as unconfirmed.

Why should a different AI review the finished work?

A different AI reviews my finished work so the review comes from an engine independent of the one that wrote it. I keep the writer and the checker as separate models on purpose. On the day those four defects turned up, I built a tool that hands every uncommitted change, meaning edits not yet saved into the project's record, to the other engine, the one that did not write it.

The instruction matters as much as the choice of reviewer. I ask the reviewer to reproduce defects rather than read the code and approve it. To reproduce a defect, the reviewer has to make the failure actually happen and show it. That tests the result, the thing my review had not checked on those four defects.

Sometimes the other engine can't run. When that happens, the first line of the report says it was a same-engine review, and the tool never calls that independent. Whoever reads the report knows from the top which kind of review they got.

How do you test a claim that a removed control broke nothing?

Run the control yourself on two cases: one where it must still fire and one where it must not. A control is any rule that blocks or flags work. On the same day as those four defects, I also wrote a rule for taking someone's word that a control was removed and nothing broke.

The reason is that a change that makes everything pass looks identical to a change that switched the control off. Both give you a clean run. Only a case built to trip the control tells them apart.

This came up when a compliance acknowledgement gate was retired across my workspaces. Taking the change on report would have risked 19 blocking findings quietly becoming advice. A blocking finding stops work until someone deals with it. Advice can be read and set aside.

The case that settled it was a workspace with no rule pack at all, a rule pack being the set of compliance rules a workspace runs under. Only that workspace proved the new rule was scoped to the right cases and was not blanket.

Common questions