Every Test Passed. So I Started Reverting the Fixes.

I audited my own 3,600-star project and found 138 problems. The interesting ones were not the bugs. They were my own fixes that only looked fixed.

Jul 28, 2026

TL;DR

After writing a test for a fix, I started deleting the fix and running the test again. If it still passes, the test is decoration. Five of mine were, in one sitting.
The same habit caught a benchmark scoring 83% because every question was written inside its own answer, and a feature I had gated so carefully it would have run for nobody.
The genuinely damaging bug was the one no benchmark caught: 29% of my notes had silently fallen out of the search index, and the score never moved.

Sketchnote title card lettered PASSED WITHOUT THE FIX. Two test result cards side by side, both marked PASS, the right one with a square torn out of its middle and the torn patch lying beside it. A rust arrow between them reads deleted the fix.

I maintain obsidian-second-brain, an open-source tool with about 3,600 stars. It is the same project 1,374 strangers cloned before I had written a single test for it. I recently ran a full audit. Security, correctness, dead code, retrieval quality, all of it. 138 findings. I have closed 135.

The bugs themselves are ordinary. The interesting part is a habit I picked up partway through, which is that after writing a test for a fix, I revert the fix and run the test again.

It caught more than I expected. None of what follows shipped. All of it would have, with a green suite and a clear conscience, if I had trusted the green.

Reverting the fix

The rule is one line: if a test passes with its fix removed, the test is decoration.

I expected the occasional slip. I got five in the first session, in three distinct shapes.

The test asserted the wrong property. I had made a file write atomic. Write to a temp file, then os.replace. My test checked that the resulting file kept its original permission bits. It passed with the fix reverted, because a plain write_text preserves mode bits too. Real test, real assertion, nothing to do with atomicity.

The fixture never reached the code path. I fixed a freshness check meant to fire on notes with no “as of” date. My fixture note had an “as of” date in it. The check never ran. Green.

The code did not do what it reads like. path.lstrip("./") does not strip a leading ./. It strips any leading run of . and / characters in any order. My test used a path where the two happened to agree.

None of these survive a revert. None of them would have been caught by review either, because all three diffs look correct.

Three-panel sketchnote of the three fake-test shapes. An arrow missing the target and hitting a box beside it, labelled wrong property. A path forking around a fenced-off branch, labelled never entered. A note card under a magnifier on one word, labelled reads differently.

The benchmark that read its own answer key

This is the one I would most want someone to tell me about.

My project searches a folder of markdown notes, and I have a rule attached to it: no change to search ships without before-and-after numbers. Useful rule. It had also only ever run against my own private notes, which means every number I published was a claim nobody could reproduce.

So I built a synthetic corpus. 300 generated notes, fixed seed, a hash so two people can confirm they measured the same thing, and three query sets: exact keyword, English paraphrase, and the same descriptions in Spanish and Russian against English notes.

The paraphrase set is the hard one. The query deliberately avoids the words in the answer.

First run, pure keyword search scored 83% on it.

That is a suspicious number for a set built to defeat keyword search. The generator wrote each note’s summary as <title> is <description>, and the description was the exact text of that note’s paraphrase query. Every query was sitting inside its own answer.

Deleting one line dropped the score from 83% to 25%.

Both are real measurements of a real system. One is a measurement of the answer key.

Sketchnote of a note card with the test question copied inside its own body, an arrow labelled same words, and two score cards falling from 83 percent to 25 percent after one line is deleted.

The feature I built so carefully it would have run for nobody

Smaller, and it has stuck with me.

I added a prompt that asks once, ever, whether you might star the repo. It fires only after something good and visible happened. A vault gets created, or a health check comes back with zero issues. Once per machine. Suppressed under CI. Suppressed by an environment variable. Never on a machine-readable output path.

I also gated it on stdout.isatty(), so it would not fire into a pipe.

In this project the scripts are almost always run by an AI agent on the user’s behalf. Stdout is a pipe essentially every time. The person still reads every line, in their terminal, in the agent’s transcript.

So the careful version would have fired for exactly nobody, while looking thorough in review and passing its tests. The tty check felt like rigor. It was a way of doing nothing safely.

There is now a test asserting the prompt still fires with stdout captured. Strange-looking test until you know why it exists.

The bug that was not the bug I named

One finding stayed open longest. On my multilingual query set, recall at 5 and recall at 10 were both exactly 0.625. Identical. Widening the window recovered nothing, so six of sixteen target notes were not ranked low. They were nowhere.

I called it the highest-value problem in the system and treated it as a cross-language failure. Three attempts to fix it by reweighting the ranking all failed, each costing something elsewhere.

Then I looked at what the six actually were. Not six failures. Three topics, each asked once in Russian and once in Spanish.

If cross-language alignment were the problem, Russian and Spanish would fail on different topics. So I translated the three into English and asked again.

Ranks 176, 22 and 38. No better. One materially worse than its Spanish version.

Never a language problem. Then I measured the thing I should have measured first: how much closer each target needed to be to reach the top ten. Between 0.009 and 0.071 in cosine distance. The entire spread between the first and tenth result is 0.02 to 0.04.

Four of the six need a correction larger than the whole visible range. Any weight big enough to rescue one scrambles the top ten of every other query in the vault.

That is not a bug I failed to fix. It is a bug that cannot be fixed the way I was trying, and one number would have said so on day one. I shipped the tool that prints that number, so the next person spends their afternoon somewhere better.

Sketchnote ruler from 0 to 0.07. A short bracket marks the visible band between rank one and rank ten, 0.02 to 0.04. A much longer bracket marks the correction needed, up to 0.071, with four target dots sitting outside the visible band.

What was actually broken while I stared at rankings

The damaging bug was next door, and no benchmark caught it.

Semantic search reads a vector index. The index is built on demand and never invalidates itself. My README told people to build it once.

My vault had 1,828 notes. The index had 1,303. 29% had no vector at all, and nothing anywhere said so.

That sounds like a staleness nit and is worse than one. An unindexed note is still findable by literal word match, so on English queries the keyword half quietly covers the gap and you never notice. On a query in another language the keyword half contributes nothing, so those notes are not ranked low. They are unreachable. It is a quieter version of the problem I wrote about in your notes are a graveyard: the note is there, and nothing can reach it.

My benchmark scored exactly the same before and after I rebuilt the index, because every target in it happened to already be indexed.

A flat number is not evidence that nothing is wrong. It is evidence that your measurement did not look there.

Sketchnote of a shelf holding 1,828 note cards, a smaller box holding 1,303 indexed, and a rust stack of 525 unreachable cards, with a dead-flat line labelled score unchanged running past all three.

What I would keep

None of this is about being careless. Every one was caught, by a cheap habit, before it reached anybody. The point is that a passing test is a much weaker signal than it feels like, and we have built almost no habits around checking the checkers.

Revert the fix, run the test. Cheap, fast, five fake tests in one sitting. If a test passes without its fix, delete it or rewrite it. A green assertion that proves nothing is worse than no test, because it occupies the space where a real one would go.

Be suspicious of a good first number. 83% on a set built to be hard was the loudest signal I got all month and I nearly banked it. A result that arrives easier than expected is a hypothesis about your instrument, not your system.

Ask what your measurement cannot see. A benchmark that does not move is not the same as a system that is fine. Mine was blind to a defect costing real users whole notes.

The audit is open at three items. Findings, dates and fixes are all in the repo, and the retrieval benchmark generates in one command if you want to point it at something of your own.

Frequently asked questions

Is reverting the fix the same as mutation testing?
Same idea, done by hand at the one point that matters most. Mutation testing perturbs the code automatically and scores how much your suite notices. This is one deliberate mutation, the removal of the exact fix, run at the moment you are most likely to be fooled.

Is this not just test-driven development?
TDD gives you the same signal for free, if you actually watch the test fail first. These five all came from a fix written before its test, which is what most bug fixing looks like in practice. The revert is how you recover the red you skipped.

Does this not double the time to write a test?
It adds seconds. Revert, run, restore. Against that, all five of these would have shipped as permanent green assertions protecting nothing.

Why not just review the test more carefully?
All three shapes above look correct in review. That is what makes them dangerous. lstrip("./") reads exactly like the thing it is not.

Can I run the retrieval benchmark?
Yes. One command generates the corpus, the seed is fixed, and it prints a hash so you can confirm you measured the same thing I did.

Key takeaways

If a test passes with its fix reverted, it is decoration. Delete it or rewrite it.
The check costs seconds and it is the only one that tells you whether a test is load-bearing.
A result that arrives easier than expected is a hypothesis about your instrument.
A benchmark that does not move is not proof the system is fine. It may just not be looking.
Publish measurements other people can reproduce, or they are claims.

About the author

Eugeniu Ghelbur builds AI tooling and writes The AI Operator. He maintains obsidian-second-brain, an open-source AI second brain starred by over 3,600 developers, and writes up what holds and what breaks when these systems run against real work.

Discussion about this post

Ready for more?