Skip to content

audit/extract.py: write-memory worker-invocation detection breaks after skill retirement #764

Description

@toejough

Context

During the independent review that fixed up openspec/changes/recall-learn-writememory-to-runbook (the write-memory-to-runbook slice of #760), the reviewer found a real detection gap in the audit tooling that is out of scope for that change (design.md's Non-Goals don't mention audit tooling either way) but should be tracked.

The gap

dev/eval/audit/extract.py hardcodes the literal Skill-tool name "write-memory" in two places to detect nested worker-procedure invocations in session transcripts:

  • Line ~587: if skill_name != "write-memory" and current_recall_call is not None: — an exception so a nested Skill(write-memory) call inside an open recall window doesn't wrongly close that window.
  • Line ~591: if skill_name in ("learn", "write-memory"): — records a learn_calls entry when a Skill tool_use event names write-memory.

This logic assumes write-memory is invoked via the Skill tool (Skill{name: write-memory}). Once agent-instructions/skills/write-memory/ is retired, transcripts recorded after the retirement will never emit a Skill(write-memory) event — the worker is instead reached by engram show <basename> in a Bash tool call (per recall/learn's updated call sites naming the promoted runbook 1053.2026-09-22.write-memory-compose-execute-verify by wikilink/basename).

Impact

  • The vault WRITE itself is still captured correctly: extract.py's existing engram learn|amend|activate Bash-command detection (~lines 368-378) is carrier-agnostic and unaffected.
  • What silently goes dead is the "worker procedure invoked" signal — build_scorecard.py's W2 question ("Did a learn/write-memory step actually execute?") and any other consumer of extract.py's learn_calls/nested-call detection that expects a Skill(write-memory) event will under-count or miss post-retirement transcripts, since no such event exists anymore.

Suggested fix (not done here — flagging only)

extract.py needs a second detection path for the post-retirement shape: recognize engram show <write-memory-runbook-basename> (or more generally, an engram show call on a runbook basename that matches known write-memory-worker promotions) as equivalent evidence of a worker invocation, alongside the existing Skill(write-memory) path. dev/eval/audit/test_extract.py will need corresponding test coverage once a fix lands.

References

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions