Skip to content

fix(audit): the harness auditor judged absence from a file it had only partly read (closes #5750) - #5764

Merged
vivekchand merged 1 commit into
mainfrom
fix/nemoclaw-onboard-trace
Sep 9, 2026
Merged

fix(audit): the harness auditor judged absence from a file it had only partly read (closes #5750)#5764
vivekchand merged 1 commit into
mainfrom
fix/nemoclaw-onboard-trace

Conversation

@vivekchand

Copy link
Copy Markdown
Owner

Closes #5750.

#5750 is a false positive, and the cause files more of them

It was filed automatically at severity high: "No code path in the OpenClaw/NemoClaw adapter reads NEMOCLAW_TRACE_FILE / the .e2e/traces directory or parses this trace artifact."

That code has shipped:

clawmetry/adapters/openclaw.py:1705 reads NEMOCLAW_TRACE_FILE, falls back to NEMOCLAW_TRACE_DIR, then the harness default
same file, :1774-1777 sets nemoclawOnboardTraceStatus, nemoclawOnboardTraceSpanCount, nemoclawOnboardTraceErrors, nemoclawOnboardSlowSpans
clawmetry/adapters/nemo.py:1040 reads it too
REQ-OBS-RSO-034 specifies the whole capability, delivered by PR #5198

Why the auditor could not see it

_adapter_source read the first 60,000 characters. openclaw.py is 193,340. The reader sits at line 1690, roughly 18k past the cut — so 69% of the adapter was invisible. Both runtimes the OSS audit covers (openclaw, nemoclaw) map to that same file, so every run judged a two-thirds-clipped adapter.

Then the prompt said:

verify against the FULL adapter above (it is provided in full)

For a task that is entirely about reporting absence, that sentence converts "I did not see it" into "it is not there". The model did what it was told the evidence supported.

This is the second time. The docstring records the first: aider's conditional COST at line ~527 was cut, the audit flagged "no COST", and the cap was raised to 60k in response. Raising a number is not a fix for a file that grows.

The fix

  • the budget is far above any adapter here, and _adapter_source returns whether it trimmed rather than trimming silently (the caller also warns);
  • when it does trim, the prompt says so instead of claiming completeness;
  • _adapter_index always carries every def and every UPPER_CASE string from the whole file, so an absence claim stays checkable even under trimming. Small, complete, and built from the full source regardless of what the body contains.
openclaw  clawmetry/adapters/openclaw.py  chars=193104  trimmed=False
   NEMOCLAW_TRACE_FILE visible in body: True
   index mentions NEMOCLAW_TRACE_FILE:  True

Guard

tests/test_harness_audit_reads_whole_adapter.py auto-discovers from manifest.json, so an adapter that outgrows the budget tomorrow fails here instead of quietly filing fiction. One test asserts the NEMOCLAW_TRACE_FILE witness sits past the old 60k mark, so it keeps proving something.

Proven red — restoring the 60k cap and the old completeness claim:

FAILED test_the_audited_adapter_is_read_whole[openclaw]
FAILED test_the_audited_adapter_is_read_whole[nemoclaw]
FAILED test_the_prompt_never_claims_completeness_it_does_not_have[openclaw]
FAILED test_the_prompt_never_claims_completeness_it_does_not_have[nemoclaw]
FAILED test_the_index_reaches_symbols_past_the_old_60k_cap
5 failed, 1 passed

Blast radius beyond this issue

#5746, #5747, #5748 and #5749 were filed by the same truncated runs. Each needs checking against the full adapter before anyone works it. I have not closed them — that is a per-issue verification, not an assumption — but #5746 already has PR #5760 open against it, which is worth a second look given this.

The cost of this bug is not a wasted CI minute. It is a high-severity issue in the tracker that a person or an agent has to read, reproduce and disprove, against code that already shipped.

No-PRD: bug fix against filed issue #5750.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Xb6A5G74JiMe3zHFs1JZEP

…y partly read

Closes #5750.

#5750 was filed automatically at severity **high**: "No code path in the
OpenClaw/NemoClaw adapter reads NEMOCLAW_TRACE_FILE / the .e2e/traces
directory or parses this trace artifact."

That code has shipped. `clawmetry/adapters/openclaw.py:1705` reads
`NEMOCLAW_TRACE_FILE`, falls back to `NEMOCLAW_TRACE_DIR` and then the
harness default, and puts `nemoclawOnboardTraceStatus`,
`nemoclawOnboardTraceSpanCount`, `nemoclawOnboardTraceErrors` and
`nemoclawOnboardSlowSpans` on the detection record. `clawmetry/adapters/nemo.py`
reads it too. REQ-OBS-RSO-034 specifies the whole capability, delivered by
PR #5198.

The auditor never saw any of it. `_adapter_source` read the first 60,000
characters; `openclaw.py` is 193,340. The reader sits at line 1690, roughly
18k characters past the cut, so **69% of the adapter was invisible** -- and
both runtimes this OSS audit covers map to that same file, so every run
judged a two-thirds-clipped adapter.

Then the prompt told the model: "verify against the FULL adapter above (it is
provided in full)". For a task that is entirely about reporting ABSENCE, that
sentence converts "I did not see it" into "it is not there". The model did
exactly what it was told the evidence supported.

This is the second time. The docstring records the first: aider's conditional
COST at line ~527 was cut, the audit flagged "no COST", and the cap was raised
to 60k in response. Raising a number is not a fix for a file that grows.

* the budget is now far above any adapter here, and `_adapter_source` returns
  whether it trimmed rather than trimming silently;
* when it DOES trim, the prompt says so instead of claiming completeness;
* `_adapter_index` always carries every `def` and every UPPER_CASE string from
  the WHOLE file, so an absence claim stays checkable even under trimming.
  Small, complete, and built from the full source regardless of the body.

tests/test_harness_audit_reads_whole_adapter.py auto-discovers from the
manifest, so an adapter that outgrows the budget tomorrow fails here instead
of quietly filing fiction. Guard proven: restoring the 60k cap and the old
completeness claim reds 5 of 6, including both audited runtimes and the
`NEMOCLAW_TRACE_FILE`-past-60k witness.

The four sibling issues #5746 to #5749 were filed by the same truncated runs
and each needs the same check against the full adapter before anyone works it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Xb6A5G74JiMe3zHFs1JZEP
@8090-software-factory

Copy link
Copy Markdown

✅ Drift Bot (ClawMetry): no drift detected

Drift Bot analyzed the changed files against this project's blueprints and requirements and found no drift.

@vivekchand

Copy link
Copy Markdown
Owner Author

Checked the four siblings — they are NOT false positives

I flagged #5746 to #5749 as suspect because they came from the same truncated runs. I checked each against the full adapter, and that caution was too broad. Correcting it here so nobody dismisses four real gaps on the strength of this PR.

The distinction is which region of the file the claim is about. #5750 asserted a specific symbol did not exist, and that symbol sits ~18k past the old cut. The other four reason about functions the model could actually see:

function the audit cited position within old 60k cap?
_openclaw_doctor_findings char 4,899 yes
_clawrouter_detect char 6,053 yes
_sandbox_inference_configs char 46,194 yes

Grepping the full clawmetry/adapters/openclaw.py for what each claims is missing:

issue claim full-file grep verdict
#5746 no share/publish state reader no share_url / is_shared / publish-registry reader; the only share hits are "reasoning-token share", "shared filesystem", "shared plugin-SDK monitor" genuine gap
#5747 no meeting-library reader zero hits for meeting genuine gap
#5748 no connected-accounts / account-priority reader zero hits genuine gap
#5749 no update candidate-state / abandoned-update reader only _gateway_migration_warning (migration warnings from gateway log events), nothing about candidate state or abandoned-update recovery genuine, narrowed — some migration signal exists, the update-pipeline states do not

So #5750 was the only truncation-caused false positive of the five, and PR #5760 against #5746 is doing real work.

The fix in this PR still stands on its own: the auditor was judging a two-thirds-clipped file while being told it was complete, and that produced one high-severity fiction. It would have produced more as openclaw.py kept growing.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[obs-gap:nemoclaw] nemoclaw: onboarding OTel trace artifacts (per-phase spans/status/duration) not ingested

1 participant