On 14 September I shipped v0.16.0 of a tool 4,449 people run, and named it The Silent Failure. Thirteen of its fixes are checks that stopped matching reality.
The worst one let a note containing an API key pass a secret scan for ten days on every Windows install, because the scan never ran and said nothing.
A check that cannot fail loudly is decoration. A check that fires when nothing is wrong is worse, because you train yourself to ignore it.
Most bug reports arrive as a crash. You get a stack trace, a line number, and a bad afternoon.
The reports in this release were different. Almost none of them crashed. The tool ran, returned success, and did the wrong thing quietly. Some of it had been doing that for ten days across every Windows install.
I maintain obsidian-second-brain, an open-source tool that runs an Obsidian vault as a knowledge base an agent can read. It has 4,449 stars, so it runs on a lot of machines that are nothing like mine. That is the only reason I found any of this.
The one that actually mattered
My tool has a write-time check that runs every time an agent writes a note. Among other things, it scans for secrets, so an API key pasted into a note never reaches a commit.
On Windows, that scan did not run. It reported success anyway.
A note holding an sk- key wrote clean on every Windows install for ten days, between the previous release and this one. Nobody was told. The check was there, the check was listed, the check passed.
One wrong word broke three checks
All three Windows failures trace to a single line: my hooks called python3.
On macOS and Linux that is Python. On Windows it is usually a Microsoft Store alias, a placeholder that exists so the system can offer to install Python for you. It exists, it prints nothing, and it exits with an error code.
Here is why that was so quiet. My code tested whether python3 existed before running the checks. The alias exists. So the test passed, the checks were skipped, and nothing complained.
That is the whole bug. A test for presence instead of a test for function.
The general shape
Any check that asks “is the tool there” instead of “did the tool do the work” has this hole. Presence is cheap to test and proves nothing.
I wrote about this before in what a harness is: instructions are requests, gates are code. This release was the bill for a gate I had written as a request.
The lie that was worse than the silence
One report was worse than a silent skip.
Claude Code caps the text a startup hook can inject at 10,000 characters. Anything longer is cut, and the session gets a short preview instead.
My hook loaded the vault’s operating manual at startup. On a real vault that manual runs well past the cap. So the session received a fragment, under a header that told it the manual was already loaded.
Read that again, because the failure is not the truncation. The truncation is a documented platform limit and fine. The failure is that my own header asserted something false, so the agent had no way to know it was working from a fragment.
A tool that goes quiet costs you a bug. A tool that reports success it did not earn costs you your judgment, because you then spend a week debugging the wrong layer.
The other direction: checks that screamed for nothing
Half of this release is the opposite failure, and I think it is underrated.
A user with a Chinese vault found that my write-time check blocked every single write. The check flags unusual characters that language models slip in, like a curly quote where a straight one belongs. Chinese uses those same characters as its ordinary punctuation. So the check called correct Chinese a violation, on every write, with no way to turn it off.
Another user with a Cyrillic vault ran my health command and was told most of the vault was missing from its search index. The index held every note. Two scripts disagreed about how to spell a non-English filename, and the report believed the wrong one.
A third case recommended a repair that could never work. The health check warned about notes a user had deliberately excluded, then told them to rebuild the index to fix it. Rebuilding excluded them again.
These are not milder than silence. A check that fires when nothing is wrong gets ignored within a week, and after that it is the same as having no check.
The failure you cannot see at all
One report was the hardest to accept, because nothing was broken.
Two processes edited the same note. Both writes succeeded. The later one won, and the earlier edit was gone. No error, no conflict, no trace.
Each individual write was already atomic, so the write was never the race. The gap was the read, change, write sequence that every caller performs. A scheduled agent, a second session, or a sync client landing an edit inside that gap just vanished.
This is the failure mode that arrives with parallel agents, and it is going to bite a lot of people this year. If you run several agents at once against the same files, you are running this race.
Why I did not use a lock
The obvious fix is a lock file. I rejected it for three reasons worth stating, because they generalize.
A lock drops files into a vault that is backed by git and replicated by a sync tool, so the lock itself becomes synced litter.
Locking needs a second implementation on Windows, and I had just spent a release learning what I do not know about Windows.
It would not have covered the reported case anyway. An edit landing on disk from a sync client holds no lock.
What I shipped instead is a check on the way out. Before writing, confirm the file still holds the bytes you read. If it does not, refuse and say so. The tiny window that remains is documented rather than hidden.
What I changed structurally
Four rules came out of this release, and they apply to any agent tooling, not just mine.
Test the work, not the tool. Never ask whether a binary exists. Ask whether it produced the output you needed.
Assert your checks ran. A skipped check must be an error, not a silence. If a scan cannot run, the write should fail.
Never label output you did not verify. Do not write “loaded” unless you counted the bytes.
Let users switch off a rule that misfires. My punctuation check had no escape hatch, which is why it broke an entire language.
What I still cannot verify
I do not own a Windows machine.
Both Windows fixes are tested against a stub that behaves the way the report described: it exists, prints nothing, exits with code 9009. That is a reproduction of the symptom, not a test on real hardware. One branch of the fix cannot be exercised on macOS or Linux at all.
So the honest status is that two fixes are unverified in the environment they target. That limitation is written into the release notes. The issue stays open for anyone on Windows to tell me it is still broken.
I would rather publish that sentence than a version number that implies more than I tested. Three of these bugs came from one person filing reproductions that actually reproduced, and two more came from someone who fixed both himself. That is the only reason any of it got found, and it is worth more than my test suite.
Frequently asked questions
Why do Claude Code hooks fail silently on Windows?
Usually because the hook calls python3, which on Windows is often a Microsoft Store placeholder rather than Python. It exists, prints nothing, and exits with an error, so any code that checks whether it exists will skip the real work and report success.
How do I know if my Claude Code hook is actually running?
Do not check that the interpreter exists. Make the hook write a line of output or a timestamp on every run, and treat a missing line as a failure. If a check cannot execute, it should fail loudly rather than pass quietly.
What is a silent failure in AI tooling?
Any case where the tool returns success while doing the wrong thing or nothing. It costs more than a crash because it looks identical to working, so you keep trusting the output.
Can two AI agents safely edit the same file?
Not without a guard. If both read the file, change it, and write it back, the later write silently erases the earlier one. Confirm the file still holds the bytes you read before writing, and refuse if it does not.
Why does my AI tool flag correct Chinese or Russian text as an error?
Most character checks are written for English and treat non-English punctuation or non-Latin filenames as anomalies. It is a rule written against one alphabet, not a problem with your text. The rule needs a way to be switched off.
Is an open-source tool with thousands of users better tested?
It is better reported, which is not the same thing. Users on machines unlike mine find the failures my test suite cannot reach, and this release exists entirely because four of them wrote detailed reports.
Key takeaways
A check that asks whether a tool exists, rather than whether it did the work, will pass on a placeholder and skip everything silently.
A skipped security check must raise an error. Mine did not, so a note holding an API key wrote clean for ten days on every Windows install.
Reporting success you did not earn is worse than reporting nothing, because it sends people to debug the wrong layer.
A check that fires when nothing is wrong gets ignored within a week, which makes it equivalent to no check at all.
Two agents editing one file will silently lose an edit unless you confirm the bytes are unchanged before writing.
Publishing what you could not verify is cheaper than a version number that implies more testing than you did.