Conversation
…LAUDE.md Closes the second platform seam 0.24.0 created. The engine now writes a per-platform rules-block target, but evals/observe.py still hardcoded CLAUDE.md -- so for any platform whose block lands elsewhere it reported rules_block_present=false. A confident wrong answer, and it would have surfaced on the first Hermes/Codex run as a parity failure that is not real. Mirroring PLATFORM_RULES_CHAIN into Python was rejected: a second copy of platform knowledge only MOVES the drift rather than removing it, which is the same class of bug this epic was filed for. Instead the engine answers. New `rules-target` subcommand: read-only, prints the absolute path of the file the resolved platform's block belongs in, walking that platform's first-match-wins chain. Deliberately does NOT record a platform -- a query must not mutate state the way write-rules does, and there is a test pinning that. observe() now asks the engine and additionally reports `rules_block_file`. Naming the file makes a failure diagnosable without a re-run: "false" and "false, and we were looking at AGENTS.md" are very different findings. adapter's _failure() gained the key too, preserving the same-key-set invariant scorers depend on. MUTATION-TESTED: restoring the hardcoded CLAUDE.md fails the new regression test with exactly `- AGENTS.md / + CLAUDE.md`, so it pins the fix rather than decorating it. Also added a cannot-tell case -- observe must stay scoreable outside a pm project rather than raising, same principle as the adapter's infra guard. Docs per this repo's drift test, which requires every dispatch key in BOTH SKILL.md and README.md (its exclusion set is deliberately empty). VERIFIED WITH A REAL AGENT RUN against the blessed baseline: exit 0, all 5 checks STABLE, 0 blocking. rules_block_file=CLAUDE.md, present=true in the run record -- the new observable flows end to end. 10 turns, $1.64 notional (subscription auth). 287 node tests, 19 pytest, all passing. Claude-Session: https://claude.ai/code/session_01WBqyLnBbYEi5hp2noc3cMC
A new subcommand is a feature, so it takes a minor bump per this repo's release discipline rather than riding along unreleased. Real Numbers recomputed rather than estimated, as that section's own claim requires: 287 tests, 39 releases, 140-line entry point, 26 lib modules. Claude-Session: https://claude.ai/code/session_01WBqyLnBbYEi5hp2noc3cMC
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes
edd-observe-hardcodes-claude-md(P1), the epic 0.24.0 unblocked — and created.The problem 0.24.0 introduced
0.24.0 made the engine write the rules block to a per-platform target. But
evals/observe.pystill hardcodedCLAUDE.md, which quietly turned it into a second, undeclared platform seam: for any platform whose block lands elsewhere, the observer reportedrules_block_present: false.That is a confident wrong answer. It would have surfaced on the first Hermes or Codex run as a parity failure that is not real — the worst kind, because the harness exists to be trusted.
Why not just mirror the chain in Python
Copying
PLATFORM_RULES_CHAINintoobserve.pywas the obvious fix and was rejected: a second copy of platform knowledge only moves the drift rather than removing it, which is the same class of bug this epic was filed for. The engine answers instead.rules-targetA read-only query printing the absolute path of the file the resolved platform's block belongs in, walking that platform's first-match-wins chain:
It deliberately does not record a platform — a query must not mutate state the way
write-rulesdoes, and there's a test pinning exactly that.observe()now asks the engine and additionally reportsrules_block_file. Naming the file makes a failure diagnosable without a re-run: "false" and "false, and we were looking at AGENTS.md" are very different findings.adapter._failure()gained the key too, preserving the same-key-set invariant scorers depend on.Verification
Mutation-tested — restoring the hardcoded
CLAUDE.mdfails the new regression test with exactly- AGENTS.md / + CLAUDE.md, so it pins the fix rather than decorating it. This branch has repeatedly found tests that read as coverage and measure nothing, so every guard added here was mutated.Real agent run against the blessed baseline: exit 0, all 5 checks STABLE, 0 blocking.
rules_block_file=CLAUDE.md,present=truein the run record — the new observable flows end to end. 10 turns, $1.64 notional (subscription auth, not billed spend).Also added a cannot-tell case:
observe()must stay scoreable outside a pm project rather than raising — same principle as the adapter's infrastructural-failure guard, since an eval that raises cannot report a verdict.287 node tests, 19 pytest, all passing. Docs updated per this repo's drift test, which requires every dispatch key in both
SKILL.mdandREADME.md(its exclusion set is deliberately empty). Real Numbers recomputed, not estimated.https://claude.ai/code/session_01WBqyLnBbYEi5hp2noc3cMC