Conversation
… would not work Investigated the reporter's proposed fix before implementing it. It does not hold: Claude Code hook handlers run in the SESSION's cwd, and the payload's cwd field is that same session directory -- not wherever a `cd` inside the Bash command took the commit. So `git rev-parse --show-toplevel` from the hook resolves to pm even when the commit landed in a paired repo, reports "same repo", and suppresses nothing. A working version would have to parse `cd` / `git -C` out of the command string. Bug 2 is instead covered by the HEAD-subject guard shipped in 0.24.0: a paired-repo commit leaves pm's HEAD untouched, so the parsed subject cannot match. The residual is one narrow slice -- a commit landing elsewhere with NO parseable -m (-am, -F, editor, heredoc) -- which is exactly what autodetour-parser-misses-am-and-f already owns. Linked relates-to rather than duplicating the work. Issue #65 stays OPEN deliberately: "no success check" is not fully true while -am short-circuits the guard. Only the internal epic is archived. Evidence caveat recorded on the issue: the cwd claim is inference from the hook docs plus observed cwd-reset behaviour, not a captured payload from a paired-repo commit. Claude-Session: https://claude.ai/code/session_01WBqyLnBbYEi5hp2noc3cMC
…=3 control) Closes the confound this harness has carried since it was blessed. THE PROBLEM. The corpus measures whether pm's ARTIFACTS make an agent behave correctly. But a headless `claude -p` also loads the operator's user-scope memory, and this maintainer's ~/.claude/CLAUDE.md states pm's lane heuristic verbatim -- the very thing chose_claude_code_lane asserts. Verified live: in an empty directory with every tool disabled, a session still recited a fact existing only in that file. So a pass could not distinguish "pm's rules block worked" from "the operator's memory told it the answer." The bias is ASYMMETRIC, which makes it a parity problem rather than a curiosity: Hermes and Codex read AGENTS.md, not ~/.claude/CLAUDE.md, so Claude Code would enter every comparison with an assist the others never get. materialize() now writes .claude/settings.local.json into the throwaway fixture -- claudeMdExcludes (absolute paths; ~ is NOT expanded) plus autoMemoryEnabled:false, since ~/.claude/projects/<p>/memory/ is written from the operator's own corrections and drifts toward whatever is being tested. SHIP-REAL-SOFTWARE.md is listed explicitly because CLAUDE.md @-imports it and whether excluding a parent drops its imports is undocumented. Nothing in ~/.claude or the repo is touched, so interactive sessions keep their memory and no "testing window" is needed. Two alternatives that look right were rejected in the comments: --bare also kills hooks and keychain auth (removing the artifact under test), and --setting-sources project,local stops pm's SessionStart hook firing because plugin enablement lives in user-scope settings. PROVEN THREE WAYS on a real fixture before trusting it: isolation on -> NO_INSTRUCTIONS (user memory suppressed) local settings ignored -> 8000-8999 (control: memory IS reachable) isolation on -> canary-active (pm's hook still fires) THE RESULT. 3 samples under suppression, gated against the prior baseline: exit 0, all 5 checks STABLE, 0 blocking. pm's own artifacts carry the lane behavior -- the baseline was never an artifact of operator memory. Re-blessed under isolation, so every future Hermes/Codex comparison is measured against a clean reference. 43 turns, $5.10 notional (subscription auth, not billed spend). Wall-clock deliberately NOT re-measured: that run took 387.9s at load 72 on a 16-core box, which is meaningless as a benchmark. The story's earlier "~2x increase" stays UNVERIFIED rather than being restated as fact. 21 pytest, 287 node, all passing. Claude-Session: https://claude.ai/code/session_01WBqyLnBbYEi5hp2noc3cMC
…ded scorer Story 5, shipped as a permanent scorer rather than a one-off probe. Until now the only evidence isolation worked was the agent FAILING to recite something from the operator's memory -- absence of evidence, indistinguishable from the probe simply not landing. An InstructionsLoaded hook now records every instruction file a session loads with its memory_type (User/Project/Local/Managed), observe() surfaces any User-scope loads, and a new `no_operator_memory_loaded` scorer asserts the list is empty on EVERY run. If someone removes the isolation, a scorer fails instead of the harness quietly measuring the wrong thing. CAUGHT BY CHECKING, NOT BY LUCK. The first recorder was `.mjs` and used `require` and `__dirname` -- neither exists in an ES module. It threw ReferenceError, the recorder's own catch swallowed it, the log stayed empty, and the scorer would have passed vacuously forever. The swallow is correct for production (an observability hook must never break the run it observes) but it hides authoring bugs, so the recorder is now `.cjs` AND a test pipes a real payload through it. A second test writes a synthetic User-scope line and asserts observe() surfaces it, proving the scorer can actually FAIL. Verified end to end on a real fixture before trusting it: recorder standalone works; a live session fires the hook and logs 1 Project entry and 0 User entries. The log being NON-empty is the point -- a scorer reading an empty file proves nothing. Re-blessed at 6 checks, 3/3 samples, user_memory_files_loaded=[] on all three. $4.63 notional (subscription auth, not billed spend). 23 pytest, 287 node, all passing. Claude-Session: https://claude.ai/code/session_01WBqyLnBbYEi5hp2noc3cMC
Stories 6-8 are findings rather than work -- marking them done so the epic reads as complete rather than perpetually 5/8. Claude-Session: https://claude.ai/code/session_01WBqyLnBbYEi5hp2noc3cMC
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes
edd-user-memory-confound-control-run(P1) — the confound this harness has carried since it was first blessed.The problem
The corpus measures whether pm's artifacts make an agent behave correctly. But a headless
claude -palso loads the operator's user-scope memory, and this maintainer's~/.claude/CLAUDE.mdstates pm's lane heuristic verbatim — the very thingchose_claude_code_laneasserts.Verified live: in an empty directory with every tool disabled, a session still recited a fact that exists only in that file. So a passing score could not distinguish "pm's rules block worked" from "the operator's memory told it the answer."
The bias is also asymmetric, which makes it a parity problem rather than a curiosity: Hermes and Codex read
AGENTS.md, not~/.claude/CLAUDE.md, so Claude Code would enter every comparison with an assist the others never get.The fix
materialize()writes.claude/settings.local.jsoninto the throwaway fixture —claudeMdExcludes(absolute paths;~is not expanded) plusautoMemoryEnabled: false, since~/.claude/projects/<p>/memory/is written from the operator's own corrections and drifts toward whatever is being tested.Nothing in
~/.claudeor the repo is touched, so interactive sessions keep their memory and no "testing window" is needed.Two alternatives that look right were rejected, and the reasons are in the code:
--barealso disables hooks and keychain auth — it removes the artifact under test.--setting-sources project,localdoes suppress user memory, but plugin enablement lives in user-scope settings, so pm'sSessionStarthook stops firing entirely.The result
3 samples under suppression, gated against the prior baseline: all STABLE, 0 blocking. pm's own artifacts carry the lane behavior — the baseline was never an artifact of operator memory. Re-blessed under isolation, so every future Hermes/Codex comparison is measured against a clean reference.
Proving it positively, not by absence
Until now the only evidence isolation worked was the agent failing to recite something — absence of evidence, indistinguishable from the probe not landing. An
InstructionsLoadedhook now records every instruction file a session loads with itsmemory_type, and a newno_operator_memory_loadedscorer asserts noUser-scope file appears on every run.This caught a real bug in itself. The first recorder was
.mjsand usedrequire/__dirname— neither exists in an ES module. It threwReferenceError, the recorder's owncatchswallowed it, the log stayed empty, and the scorer would have passed vacuously forever. The swallow is correct for production (an observability hook must never break the run it observes) but it hides authoring bugs, so the recorder is now.cjsand a test pipes a real payload through it. A second test writes a syntheticUser-scope line and assertsobserve()surfaces it — proving the scorer can actually fail.Verified end to end before trusting it: recorder standalone works; a live session fires the hook and logs 1
Projectentry, 0Userentries. The log being non-empty is the point — a scorer reading an empty file proves nothing.Deliberately not claimed
The story asked to investigate a "~2x wall-clock increase". Not re-measured, and left unverified rather than restated as fact — the run took 387.9s at load 72 on a 16-core box, which is meaningless as a benchmark.
Scope
No version bump and no Mintlify sync: everything here is under
evals/, which is inert in a consumer install and not something a user or agent reads about. Flagging that explicitly rather than silently skipping the docs-currency check.23 pytest, 287 node, all passing. Baseline now 6 checks, 3/3 samples,
user_memory_files_loaded=[]on all three. $4.63 notional (subscription auth, not billed spend).https://claude.ai/code/session_01WBqyLnBbYEi5hp2noc3cMC