Context
During the independent review that fixed up openspec/changes/recall-learn-writememory-to-runbook (the write-memory-to-runbook slice of #760), the reviewer found a real detection gap in the audit tooling that is out of scope for that change (design.md's Non-Goals don't mention audit tooling either way) but should be tracked.
The gap
dev/eval/audit/extract.py hardcodes the literal Skill-tool name "write-memory" in two places to detect nested worker-procedure invocations in session transcripts:
- Line ~587:
if skill_name != "write-memory" and current_recall_call is not None: — an exception so a nested Skill(write-memory) call inside an open recall window doesn't wrongly close that window.
- Line ~591:
if skill_name in ("learn", "write-memory"): — records a learn_calls entry when a Skill tool_use event names write-memory.
This logic assumes write-memory is invoked via the Skill tool (Skill{name: write-memory}). Once agent-instructions/skills/write-memory/ is retired, transcripts recorded after the retirement will never emit a Skill(write-memory) event — the worker is instead reached by engram show <basename> in a Bash tool call (per recall/learn's updated call sites naming the promoted runbook 1053.2026-09-22.write-memory-compose-execute-verify by wikilink/basename).
Impact
- The vault WRITE itself is still captured correctly:
extract.py's existing engram learn|amend|activate Bash-command detection (~lines 368-378) is carrier-agnostic and unaffected.
- What silently goes dead is the "worker procedure invoked" signal —
build_scorecard.py's W2 question ("Did a learn/write-memory step actually execute?") and any other consumer of extract.py's learn_calls/nested-call detection that expects a Skill(write-memory) event will under-count or miss post-retirement transcripts, since no such event exists anymore.
Suggested fix (not done here — flagging only)
extract.py needs a second detection path for the post-retirement shape: recognize engram show <write-memory-runbook-basename> (or more generally, an engram show call on a runbook basename that matches known write-memory-worker promotions) as equivalent evidence of a worker invocation, alongside the existing Skill(write-memory) path. dev/eval/audit/test_extract.py will need corresponding test coverage once a fix lands.
References
Context
During the independent review that fixed up
openspec/changes/recall-learn-writememory-to-runbook(thewrite-memory-to-runbook slice of #760), the reviewer found a real detection gap in the audit tooling that is out of scope for that change (design.md's Non-Goals don't mention audit tooling either way) but should be tracked.The gap
dev/eval/audit/extract.pyhardcodes the literal Skill-tool name"write-memory"in two places to detect nested worker-procedure invocations in session transcripts:if skill_name != "write-memory" and current_recall_call is not None:— an exception so a nestedSkill(write-memory)call inside an open recall window doesn't wrongly close that window.if skill_name in ("learn", "write-memory"):— records alearn_callsentry when aSkilltool_use event nameswrite-memory.This logic assumes write-memory is invoked via the
Skilltool (Skill{name: write-memory}). Onceagent-instructions/skills/write-memory/is retired, transcripts recorded after the retirement will never emit aSkill(write-memory)event — the worker is instead reached byengram show <basename>in aBashtool call (perrecall/learn's updated call sites naming the promoted runbook1053.2026-09-22.write-memory-compose-execute-verifyby wikilink/basename).Impact
extract.py's existingengram learn|amend|activateBash-command detection (~lines 368-378) is carrier-agnostic and unaffected.build_scorecard.py's W2 question ("Did a learn/write-memory step actually execute?") and any other consumer ofextract.py'slearn_calls/nested-call detection that expects aSkill(write-memory)event will under-count or miss post-retirement transcripts, since no such event exists anymore.Suggested fix (not done here — flagging only)
extract.pyneeds a second detection path for the post-retirement shape: recognizeengram show <write-memory-runbook-basename>(or more generally, anengram showcall on a runbook basename that matches known write-memory-worker promotions) as equivalent evidence of a worker invocation, alongside the existingSkill(write-memory)path.dev/eval/audit/test_extract.pywill need corresponding test coverage once a fix lands.References
openspec/changes/recall-learn-writememory-to-runbook(write-memory-to-runbook slice of recall/learn/write-memory: promote runbook conversion from eval fixture to real production skill replacement #760)dev/eval/audit/extract.py:587,591(Skill-name detection),:368-378(carrier-agnostic Bash-command detection, unaffected)dev/eval/audit/build_scorecard.py:391(W2 question that consumes this signal)1053.2026-09-22.write-memory-compose-execute-verify