Repository navigation
STE-39: the model is advisory, never merged into the verdict or the hash - #36
Merged
Merged
Conversation
…ever merged into it Same skill, same bytes, same config, five runs: agentic-payments.x402 answered WARNING/75 once and SAFE/100 four times, dapp.smart-accounts WARNING three times out of five. A verdict the model can move is one a third party re-running the audit cannot reproduce. Option 1 from the ticket, as it recommended: - apply_llm records the answer as report.llm_advisory, with stricter_than_verdict and disagrees_with_verdict for human review, and changes no verdict field in either direction. The policy.tighten merge is gone. - Policy row 5 (asked-but-failed forces WARNING) is gone: whether a gateway answered is a fact about the auditor's network, not the skill. Row numbers are kept so published reasons still read correctly. - The model trail (llm_advisory, llm_notes, llm_used, llm_attempted, llm_model) is excluded from compute_evidence_hash, so the document, and the evidence_hash on chain, is byte-identical with or without a model, whatever it answered. - run_audit used a second copy of the model step; it now calls apply_llm. - tighten and enforce_critical had no caller left and are removed rather than kept as code implying a model can still move a verdict. Part A: stage3_synthesis.md now says an Agent Skill published as markdown has no permission surface, so the absence of declared permissions or tools is not a finding. Every downgrade above cited "declaring no permissions or tools", the category error STE-36 removed from the regexes. Option 3: requests carry temperature 0 and seed 0 (the native Messages backend takes temperature only; None omits either). Measured live after both changes: 5/5 SAFE/100 for each skill, and one distinct document hash across the five model-on runs. Option 1 stands on its own regardless, since five equal answers are not a proof of determinism.
…at reproduces the ledger audit-evidence.md states option 1 and why, the before/after table from the live model, and a correction the investigation turned up: re-running the audit reproduces the 19 published reports byte-exact at the seed commit f36f690 (19/19) but not at main (0/19). Verdicts, scores and findings are identical; only the evidence_hash field inside the document differs, because the internal report it hashes gained stage2.applicable and stage2.detail in STE-40. The commit is now part of the reproduction instructions. The raw runs are in docs/evidence/ste-39-llm-variance-2026-09-15.json.
All 26 corpus entries audited three ways against the live model: the verdict document is byte-identical in every case, the model answered all 26, and every poisoned fixture stays DANGEROUS. The model disagrees with the verdict on exactly two entries, price-checker and cctp, which are the two wallet_op false positives STE-37 fixes; the disagreement is now recorded instead of acted on.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Linear: STE-39 — ACC'd by @axel on 15 Sep. Review: @axel @fable
What changed
llm_advisory(withstricter_than_verdict/disagrees_with_verdict) and changes no verdict field in either direction. Thepolicy.tightenmerge is gone.evidence_hash, so the verdict document is byte-identical with or without a model, whatever it answered, and whether a key is configured.run_auditreusesapply_llminstead of a second copy;tighten/enforce_criticalremoved (no callers).temperature0 andseed0 by default (Noneomits).Evidence
x402WARNING/75 + SAFE ×4 → SAFE/100 ×5;smart-accountsWARNING ×3 of 5 → SAFE/100 ×5; 1 distinct document hash across model-on runs.price-checkerandcctp— the STE-37 false positives.Heads-up for the ledger
Before this, the hashed report included the model's notes, so report bytes depended on config (
use_llm=Falsereproduced 0/25 published reports; default config with no key reproduces 18/25 at6825ae5, the other 7 atf36f690). This removes that dependence permanently and shifts the bytes once more, so the 25 reports need one re-anchor at this commit. That is planned together with the STE-37 re-submission ofcctpandprice-checker. Documented indocs/audit-evidence.md.