Repository navigation
docs: mythos audit - #104
Conversation
Restructure around a plain-language explanation of the witness chain, state explicitly that the bundled agents/tools/LLM backend are deterministic stubs and fixtures, and consolidate the proves / does-not-prove caveats into one place. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
noctem-o
changed the title
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 116fdb701f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| |---|---|---| | ||
| | Bundle self-consistency | `verify --witness <dir>` | Manifest paths resolve under strict confinement (no absolute paths, `..` escapes, or symlink escapes) and the SHA-256 artifact/bundle hashes match the recorded values. | | ||
| | Semantic recompute | `verify --witness <dir> --recompute-witness-root` | The witness root recomputes from the bundle's recorded semantics. Recompute fails closed on incomplete or inconsistent transcripts (missing steps, orphan or duplicate tool calls, trace/transcript mismatches). | | ||
| | Anchored verification | `verify --witness <dir> --expect <root>` | The bundle matches a root you obtained from somewhere you trust. This is the only layer that defends against wholesale bundle replacement. | |
There was a problem hiding this comment.
As documented here, verify --witness <dir> --expect <root> does not actually compare the bundle semantics to the supplied root: verify_cmd returns after drift::verify_witness_bundle whenever --witness is a directory and --recompute-witness-root is absent, so args.expect is never used in that path (src/main.rs:1128-1145). Users following this “only layer” for anchored verification can pass a stale or wrong root and still succeed if the bundle's manifest hashes are self-consistent; the bundle command needs --recompute-witness-root --expect <root>.
Useful? React with 👍 / 👎.
| | Matching witness root under semantic recompute | Bundle semantics match the witnessed commitment boundary. | That the execution was externally anchored in time/public logs. | | ||
| | Anchored verification (`--expect`) | Bundle semantics match an externally provided root. | Hardware/runtime integrity of original execution. | | ||
| | Passing ordeal fixture run | Deterministic fixture wiring stayed stable (trace/transcript/schema-fingerprint path). | Scientific benchmark validity or real tool-semantic quality. | | ||
| A recorded bundle can be replayed (`run --replay <bundle_dir>`): tool responses are substituted from the transcript instead of re-executed, and any divergence between the recorded and replayed runs is localized in `drift_report.json` down to the first mismatching call and field. Deterministic fault injection (`--faults on --fault-profile ci|stress`) is keyed by seed/run/step/call, so fault schedules are identical across runs and never a source of spurious drift. |
There was a problem hiding this comment.
The shorthand run --replay <bundle_dir> fails with the default CLI settings because --runs defaults to 5000, and run_agent rejects replay unless the selected run list has exactly one entry (src/main.rs:771-777). Users copying this example will get --replay requires a single --case or --runs 1, so the documented replay command should include --runs 1 or a --case value.
Useful? React with 👍 / 👎.
| | Matching witness root under semantic recompute | Bundle semantics match the witnessed commitment boundary. | That the execution was externally anchored in time/public logs. | | ||
| | Anchored verification (`--expect`) | Bundle semantics match an externally provided root. | Hardware/runtime integrity of original execution. | | ||
| | Passing ordeal fixture run | Deterministic fixture wiring stayed stable (trace/transcript/schema-fingerprint path). | Scientific benchmark validity or real tool-semantic quality. | | ||
| A recorded bundle can be replayed (`run --replay <bundle_dir>`): tool responses are substituted from the transcript instead of re-executed, and any divergence between the recorded and replayed runs is localized in `drift_report.json` down to the first mismatching call and field. Deterministic fault injection (`--faults on --fault-profile ci|stress`) is keyed by seed/run/step/call, so fault schedules are identical across runs and never a source of spurious drift. |
There was a problem hiding this comment.
This promises that drift_report.json localizes “any divergence” between recorded and replayed runs, but the replay path only calls drift::detect_transcript_drift(expected_transcript, &transcript_record) and extends it with tool mismatches (src/main.rs:980-984); the expected agent_trace.json is loaded only to select the agent and is never compared. If the replayed agent changes its thought/action/final trace while issuing the same tool requests, the semantic root changes but drift_report.json will not report that divergence, so the README should narrow this claim or the code should compare traces too.
Useful? React with 👍 / 👎.
1 participant
Rewrites the README to be easier to read on first contact and to align more tightly with what the project actually is.
Main changes:
clawdbot,ordeal, the tool layer, and thellm.generatebackend are deterministic stubs/fixtures, that no external model or service is contacted, and that integrating a real agent currently means driving the Rust APIs directly.No code changes.
🤖 Generated with Claude Code