fix(audit): the harness auditor judged absence from a file it had only partly read (closes #5750) - #5764
Conversation
…y partly read Closes #5750. #5750 was filed automatically at severity **high**: "No code path in the OpenClaw/NemoClaw adapter reads NEMOCLAW_TRACE_FILE / the .e2e/traces directory or parses this trace artifact." That code has shipped. `clawmetry/adapters/openclaw.py:1705` reads `NEMOCLAW_TRACE_FILE`, falls back to `NEMOCLAW_TRACE_DIR` and then the harness default, and puts `nemoclawOnboardTraceStatus`, `nemoclawOnboardTraceSpanCount`, `nemoclawOnboardTraceErrors` and `nemoclawOnboardSlowSpans` on the detection record. `clawmetry/adapters/nemo.py` reads it too. REQ-OBS-RSO-034 specifies the whole capability, delivered by PR #5198. The auditor never saw any of it. `_adapter_source` read the first 60,000 characters; `openclaw.py` is 193,340. The reader sits at line 1690, roughly 18k characters past the cut, so **69% of the adapter was invisible** -- and both runtimes this OSS audit covers map to that same file, so every run judged a two-thirds-clipped adapter. Then the prompt told the model: "verify against the FULL adapter above (it is provided in full)". For a task that is entirely about reporting ABSENCE, that sentence converts "I did not see it" into "it is not there". The model did exactly what it was told the evidence supported. This is the second time. The docstring records the first: aider's conditional COST at line ~527 was cut, the audit flagged "no COST", and the cap was raised to 60k in response. Raising a number is not a fix for a file that grows. * the budget is now far above any adapter here, and `_adapter_source` returns whether it trimmed rather than trimming silently; * when it DOES trim, the prompt says so instead of claiming completeness; * `_adapter_index` always carries every `def` and every UPPER_CASE string from the WHOLE file, so an absence claim stays checkable even under trimming. Small, complete, and built from the full source regardless of the body. tests/test_harness_audit_reads_whole_adapter.py auto-discovers from the manifest, so an adapter that outgrows the budget tomorrow fails here instead of quietly filing fiction. Guard proven: restoring the 60k cap and the old completeness claim reds 5 of 6, including both audited runtimes and the `NEMOCLAW_TRACE_FILE`-past-60k witness. The four sibling issues #5746 to #5749 were filed by the same truncated runs and each needs the same check against the full adapter before anyone works it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Xb6A5G74JiMe3zHFs1JZEP
✅ Drift Bot (ClawMetry): no drift detectedDrift Bot analyzed the changed files against this project's blueprints and requirements and found no drift. |
Checked the four siblings — they are NOT false positivesI flagged #5746 to #5749 as suspect because they came from the same truncated runs. I checked each against the full adapter, and that caution was too broad. Correcting it here so nobody dismisses four real gaps on the strength of this PR. The distinction is which region of the file the claim is about. #5750 asserted a specific symbol did not exist, and that symbol sits ~18k past the old cut. The other four reason about functions the model could actually see:
Grepping the full
So #5750 was the only truncation-caused false positive of the five, and PR #5760 against #5746 is doing real work. The fix in this PR still stands on its own: the auditor was judging a two-thirds-clipped file while being told it was complete, and that produced one high-severity fiction. It would have produced more as 🤖 Generated with Claude Code |
…#5765) Claude-Session: https://claude.ai/code/session_01Xb6A5G74JiMe3zHFs1JZEP Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Closes #5750.
#5750 is a false positive, and the cause files more of them
It was filed automatically at severity high: "No code path in the OpenClaw/NemoClaw adapter reads
NEMOCLAW_TRACE_FILE/ the.e2e/tracesdirectory or parses this trace artifact."That code has shipped:
clawmetry/adapters/openclaw.py:1705NEMOCLAW_TRACE_FILE, falls back toNEMOCLAW_TRACE_DIR, then the harness default:1774-1777nemoclawOnboardTraceStatus,nemoclawOnboardTraceSpanCount,nemoclawOnboardTraceErrors,nemoclawOnboardSlowSpansclawmetry/adapters/nemo.py:1040REQ-OBS-RSO-034Why the auditor could not see it
_adapter_sourceread the first 60,000 characters.openclaw.pyis 193,340. The reader sits at line 1690, roughly 18k past the cut — so 69% of the adapter was invisible. Both runtimes the OSS audit covers (openclaw,nemoclaw) map to that same file, so every run judged a two-thirds-clipped adapter.Then the prompt said:
For a task that is entirely about reporting absence, that sentence converts "I did not see it" into "it is not there". The model did what it was told the evidence supported.
This is the second time. The docstring records the first: aider's conditional
COSTat line ~527 was cut, the audit flagged "no COST", and the cap was raised to 60k in response. Raising a number is not a fix for a file that grows.The fix
_adapter_sourcereturns whether it trimmed rather than trimming silently (the caller also warns);_adapter_indexalways carries everydefand every UPPER_CASE string from the whole file, so an absence claim stays checkable even under trimming. Small, complete, and built from the full source regardless of what the body contains.Guard
tests/test_harness_audit_reads_whole_adapter.pyauto-discovers frommanifest.json, so an adapter that outgrows the budget tomorrow fails here instead of quietly filing fiction. One test asserts theNEMOCLAW_TRACE_FILEwitness sits past the old 60k mark, so it keeps proving something.Proven red — restoring the 60k cap and the old completeness claim:
Blast radius beyond this issue
#5746, #5747, #5748 and #5749 were filed by the same truncated runs. Each needs checking against the full adapter before anyone works it. I have not closed them — that is a per-issue verification, not an assumption — but #5746 already has PR #5760 open against it, which is worth a second look given this.
The cost of this bug is not a wasted CI minute. It is a high-severity issue in the tracker that a person or an agent has to read, reproduce and disprove, against code that already shipped.
No-PRD: bug fix against filed issue #5750.
🤖 Generated with Claude Code
https://claude.ai/code/session_01Xb6A5G74JiMe3zHFs1JZEP