A bounded recall hook is appealing because it makes memory boring in the best way. Four relevant notes and silence when nothing matches is a much safer default than pulling half a second brain into context. The restraint is doing a lot of the engineering work here.
Peter, the cap was the easy part. 4 notes and 900 characters is one constant.
The work was teaching it to say nothing. Early versions always found something "relevant", so unrelated prompts came back carrying notes about last week's meeting.
Logging every match is what fixed it, and the fix was almost always in the note itself, not in the retrieval code.
I like that finding, especially that the fix was usually in the note rather than the retrieval code. Teaching the system to stay silent sounds like the harder engineering problem. If the log preserves the matched note and the reason or score behind the match, you can inspect why memory entered the prompt instead of only seeing that it did. I would want the failures on both sides in that record: irrelevant notes that got through, and relevant notes the system kept silent. Have you started keeping examples where it should have recalled something but did not?
Not systematically yet. The log makes false positives visible because I can inspect what entered the prompt and why. False negatives are harder because silence leaves nothing to inspect.
The missing piece is a small eval set: prompt, expected notes, and expected silence. Without that, I’m measuring precision but not recall.
I like that distinction between visible false positives and silent false negatives. Treating expected silence as a first-class result in the eval set feels right. I would also version the notes and retrieval rules alongside each case, otherwise a change that improves recall could quietly reopen the false-positive problem. Are you planning to run that set on every change, or only when the notes themselves change?
Update since this exchange: the first version of that eval exists now. It samples real notes, has a model write a question whose answer lives in the note while avoiding the title words, then scores whether search returns the right note and how high it ranks. Misses come out as a list, not a feeling. Expected silence isn't in the case set yet, that's still the harder half.
On your question: every retrieval change, always. The run costs seconds, so being selective saves nothing. Notes change daily though, so per note change would be noise. They get a periodic re-run on the same cases instead, to catch drift as the vault grows.
Agreed on versioning. The cases are pinned files, so any recall improvement gets measured against the exact set that caught the old false positives. Change one thing, re-run the same cases, compare. Anything else is guessing with confidence.
Contextual memory beats complex retrieval architectures every time. By keeping the injection small and relevant, you avoid the noise that usually degrades model performance in larger systems. I took this logic further by mapping my entire Obsidian graph to a local file structure that the agent can traverse dynamically. This allows the model to map relationships between notes rather than just pulling static text snippets. It turns the agent from a search tool into a genuine thinking partner that understands the history of your own ideas.
Obsidian is already a local file structure, curious what the mapping changed in practice. What does the agent traverse that plain wikilinks didn't give it?
- no hooks on Cowork, how would you implement it? A skill with Obsidian MCP call executed on session start (Claude.md instruction)?
- reasoning behind 4 files and 900 characters? Is there a hard limit not to pass
- I have workspaces with pre-defined workflows; most execution is done with skills; the skill could have a step 0 with an Obsidian query to load the context right away (encapsulated); the query could even be stored outside the skill (easy to change w/o skill update; thoughts?
- how did you build your Obsidian query in the hook to target the correct 4 files? Trial/error or is there best practices you learned along the way?
Lots of questions I know but this is a fascinating topic!
- Cowork: yeah, no hooks, so you lose the automatic part. A skill that queries the Obsidian MCP at session start is the closest thing. Hook fires on every prompt, skill fires when the skill runs. For workspace sessions that's mostly fine, the first prompt is where context matters anyway.
- The 4/900 numbers aren't science. I wanted memory to be a rounding error next to the task, ~250 tokens. Go bigger and nothing breaks, it just slowly stops being free. The abstain threshold matters more than the cap. Zero notes beats four mediocre ones.
- Your step 0 idea is good, my own commands work like that. A skill knows its domain, so its recall is sharper than my generic hook. Query outside the skill is fine too. Just keep the cap and the abstain rule inside the skill, one config edit shouldn't be able to turn recall into a firehose.
- No clever query in the hook. It throws the prompt text at the vault's normal lexical search, keeps what clears a confidence bar, cuts at 4. What actually needed tuning was the notes: clear titles, dates, one topic per note. When recall picked wrong, I fixed the note, not the search.
Build the Cowork version and tell me how it goes, genuinely curious.
A bounded recall hook is appealing because it makes memory boring in the best way. Four relevant notes and silence when nothing matches is a much safer default than pulling half a second brain into context. The restraint is doing a lot of the engineering work here.
Peter, the cap was the easy part. 4 notes and 900 characters is one constant.
The work was teaching it to say nothing. Early versions always found something "relevant", so unrelated prompts came back carrying notes about last week's meeting.
Logging every match is what fixed it, and the fix was almost always in the note itself, not in the retrieval code.
I like that finding, especially that the fix was usually in the note rather than the retrieval code. Teaching the system to stay silent sounds like the harder engineering problem. If the log preserves the matched note and the reason or score behind the match, you can inspect why memory entered the prompt instead of only seeing that it did. I would want the failures on both sides in that record: irrelevant notes that got through, and relevant notes the system kept silent. Have you started keeping examples where it should have recalled something but did not?
Not systematically yet. The log makes false positives visible because I can inspect what entered the prompt and why. False negatives are harder because silence leaves nothing to inspect.
The missing piece is a small eval set: prompt, expected notes, and expected silence. Without that, I’m measuring precision but not recall.
I like that distinction between visible false positives and silent false negatives. Treating expected silence as a first-class result in the eval set feels right. I would also version the notes and retrieval rules alongside each case, otherwise a change that improves recall could quietly reopen the false-positive problem. Are you planning to run that set on every change, or only when the notes themselves change?
Update since this exchange: the first version of that eval exists now. It samples real notes, has a model write a question whose answer lives in the note while avoiding the title words, then scores whether search returns the right note and how high it ranks. Misses come out as a list, not a feeling. Expected silence isn't in the case set yet, that's still the harder half.
On your question: every retrieval change, always. The run costs seconds, so being selective saves nothing. Notes change daily though, so per note change would be noise. They get a periodic re-run on the same cases instead, to catch drift as the vault grows.
Agreed on versioning. The cases are pinned files, so any recall improvement gets measured against the exact set that caught the old false positives. Change one thing, re-run the same cases, compare. Anything else is guessing with confidence.
Contextual memory beats complex retrieval architectures every time. By keeping the injection small and relevant, you avoid the noise that usually degrades model performance in larger systems. I took this logic further by mapping my entire Obsidian graph to a local file structure that the agent can traverse dynamically. This allows the model to map relationships between notes rather than just pulling static text snippets. It turns the agent from a search tool into a genuine thinking partner that understands the history of your own ideas.
https://cyrilsimonnet.substack.com/p/i-built-a-second-brain-with-claude
Obsidian is already a local file structure, curious what the mapping changed in practice. What does the agent traverse that plain wikilinks didn't give it?
Another brilliant design. Thank you for sharing 🙏
Few questions:
- no hooks on Cowork, how would you implement it? A skill with Obsidian MCP call executed on session start (Claude.md instruction)?
- reasoning behind 4 files and 900 characters? Is there a hard limit not to pass
- I have workspaces with pre-defined workflows; most execution is done with skills; the skill could have a step 0 with an Obsidian query to load the context right away (encapsulated); the query could even be stored outside the skill (easy to change w/o skill update; thoughts?
- how did you build your Obsidian query in the hook to target the correct 4 files? Trial/error or is there best practices you learned along the way?
Lots of questions I know but this is a fascinating topic!
Thanks Fabrice.
- Cowork: yeah, no hooks, so you lose the automatic part. A skill that queries the Obsidian MCP at session start is the closest thing. Hook fires on every prompt, skill fires when the skill runs. For workspace sessions that's mostly fine, the first prompt is where context matters anyway.
- The 4/900 numbers aren't science. I wanted memory to be a rounding error next to the task, ~250 tokens. Go bigger and nothing breaks, it just slowly stops being free. The abstain threshold matters more than the cap. Zero notes beats four mediocre ones.
- Your step 0 idea is good, my own commands work like that. A skill knows its domain, so its recall is sharper than my generic hook. Query outside the skill is fine too. Just keep the cap and the abstain rule inside the skill, one config edit shouldn't be able to turn recall into a firehose.
- No clever query in the hook. It throws the prompt text at the vault's normal lexical search, keeps what clears a confidence bar, cuts at 4. What actually needed tuning was the notes: clear titles, dates, one topic per note. When recall picked wrong, I fixed the note, not the search.
Build the Cowork version and tell me how it goes, genuinely curious.