Repository navigation
security: Agent Behavior Verification — Worker Remit, Praxen report, and a Critical-blocks-release gate - #77
Conversation
…rt, release gate
Adds `security/praxen/` — the "Praxen agent-behavior reports" the security
README has been reserving a slot for — and documents a release gate alongside
the existing red-team bar.
Contents:
- `WORKER_REMIT.md` — socxen's declared policy: what the agent is authorized to
do. Authored blind from README/SECURITY/docs only (no access to skills/ or
connector/), so it states intent rather than mirroring the implementation.
- `SCAN_INSTRUCTIONS.md` — scan scope: the whole deployed system (skill +
connector + permissions pack + install), not the skill alone.
- `results/2026-08-12-socxen-0.6.9.{html,json}` — the scan of 0.6.9 at 005fa4c.
- `results/2026-08-12-socxen-0.6.9-audit.md` — independent context-unaware audit
of that scan: 13/13 findings CONFIRMED, 0 unsupported, 0 remit defects.
Release gate (documented, not CI — the same posture as the red-team bar):
a Praxen Critical finding blocks a tag; it is fixed or explicitly waived in
writing by a maintainer before release. High/Medium/Low do not block.
0.6.9 result: 0 Critical (gate passes), 5 High, 7 Medium, 1 Low. Weighted RAISE
2.45/5. Build an AI Red Team scores 3 on the strength of the a10
find-fix-retest-regression arc in redteam/HISTORY.md; Monitor Continuously
scores 3 for the default-on audit trail at the bridge chokepoint.
Also:
- `security/README.md` — Praxen moved from "planned neighbors" into Contents,
with a callout beside the red-team one.
- `CONTRIBUTING.md` — new "Release gates" section tabulating both gates.
- `tests/test_repo_invariants.py` — the SPDX-header exemption for generated run
records now covers `security/praxen/results/` as it already did
`security/redteam/results/`. Authored docs in `security/praxen/` carry headers.
Open items from this scan are tracked as #71 and #76; #72 was found by the
previous scan and is fixed in 0.6.9 — this re-scan verifies it.
Signed-off-by: Steve Wilson <steve.wilson@exabeam.com>
virtualsteve-exa
left a comment
There was a problem hiding this comment.
Code-owner review, with independent verification. Verdict: READY once two small things are fixed — one is a public-repo hygiene item.
Verified independently ✅
- Suite + invariants: 238 passed / 0 skipped on the branch;
gen_aibom.py --checkcurrent; CI green. The SPDX-test change is minimal and exactly parallel to the existing redteam/results exemption. - Report internal consistency: JSON has 13 findings (5 High / 7 Medium / 1 Low, 0 Critical) matching the README table; 50 distinct
R-rules referenced, matching the coverage claim. - Spot-checked findings at their cited lines — accurate: 009 (
run.py:341-342floating--models sonnetdefault + the now-staleHISTORY.md:42gate note) and 013 (all fouruses:across the three workflows are mutable major tags) check out verbatim. The audit's sharpest catch on 001 also verifies: the snippet-merge instruction lives insideif [ "$CREDS_OK" = 0 ](install.sh:445), so an install with credentials present never prints the merge instruction at all — that detail is worth carrying into whatever fixes #70/#73 land. - No secrets/tenant data in the artifacts — all credential-shaped grep hits are rule text.
Fix before merge
- Local absolute paths committed to a public repo.
results/2026-08-12-socxen-0.6.9-audit.md:6-8(×3) and one occurrence in the JSON embed/Users/steve.wilson/Documents/github/deckard/local/...— operator username plus an internal project name. The isolation claims survive with placeholders (<scan-workspace>/…); please scrub both files. The HTML is clean. - Dangling reference:
SCAN_INSTRUCTIONS.md:52points atCLONE_SHA.txt"beside this file," which is (rightly) not committed — it's scan-time state. Reword to "pin the SHA at scan time and record it in the results artifact" (the results do record005fa4c).
Ask, non-blocking
- Close the triage loop the gate itself promises. CONTRIBUTING now says non-blocking findings "are triaged into issues" — but 13 findings map to #71, #76, and the previously-fixed #72 only; the other ten have no recorded disposition. Suggest a short triage table (finding → issue / accepted-with-rationale / superseded) in
praxen/README.mdor beside the results. Note findings 001 and 012 land partly in #70/#73's lap — cross-reference rather than double-file.
For the maintainer (not this PR's author to resolve)
The remit's 8 Open Questions record five operator decisions (approver identity, case-write tier, no volume cap, no telemetry allowlist, missing-gate = warn-not-refuse). Those are policy, and maintainer sign-off on them is the substantive review of this PR — flagged to Steve directly.
Nice work overall: the blind-authoring discipline, the audit pass that re-reads every citation, and honest score rationale (band-edge calls recorded as such) make this the most defensible artifact in security/.
… table Addresses code-owner review on #77. 1. Public-repo hygiene: removed committed absolute paths carrying an operator username and internal project name — 3 occurrences in the audit record and the `scan.workspace` field in the findings JSON, replaced with `<scan-workspace>` / `<socxen-checkout>` placeholders. The isolation claims in the audit record read the same with placeholders. HTML was already clean. Verified: no `/Users/` or operator name anywhere under security/praxen/. JSON re-validated — 13 findings, 50 rules, weighted 2.45 unchanged. 2. Dangling reference: SCAN_INSTRUCTIONS.md pointed at a `CLONE_SHA.txt` "beside this file" that is (correctly) scan-time state and not committed. Reworded to "pin the SHA at scan time and record it in the results artifact", naming the 005fa4c pin the committed run actually used. 3. Triage table (review item 3, non-blocking): all 13 findings now carry a recorded disposition in security/praxen/README.md — #71, #76, and #70/#73 cross-referenced rather than double-filed, the rest marked *awaiting triage* as the live worklist. Disposition is explicitly the maintainer's call; the table does not invent one. Also carried the reviewer's install.sh:445 catch (the merge instruction sits inside `if [ "$CREDS_OK" = 0 ]`, so an install with credentials present never prints it) as a note for whoever fixes #70/#73. 238 passed; AI BOM current. Signed-off-by: Steve Wilson <steve.wilson@exabeam.com>
|
Thanks — all three addressed in 1. Local absolute paths — fixed, and thank you for catching it. That's a genuine hygiene miss on a public repo: operator username plus an internal project name, committed. Scrubbed all four occurrences (3 in the audit record, plus 2. Dangling 3. Triage table — added, and you were right that the gate was writing a cheque the artifact didn't cash. All 13 findings now carry a disposition in I also carried your On the maintainer item: the remit's five operator decisions (approver identity, case-write tier, no volume cap, no telemetry allowlist, missing-gate = warn-not-refuse) were decided by @virtualsteve-exa during authoring, not inferred — each is recorded in the Open Questions section with the date. Worth an explicit confirm on merge, since they're the substantive policy content and they set what every future scan checks. |
|
Re-reviewed |
…eleased changes (#80) * docs: bring the docs current with the plugin/ restructure and the unreleased changes Pre-promotion currency pass. Three classes of defect, all found by reviewing docs against the code rather than against each other. BROKEN INSTRUCTIONS (post-#66, a user follows these): - plugin/docs/installation.md told users to run ./install.sh --merge-permissions in the section the page itself calls the most important on it. There is no root install.sh any more; twenty lines earlier the same page correctly says ./plugin/install.sh. Same for merge_permissions.py's path. - install.sh's OWN output was stale: the usage block is printed verbatim as --help, and every line said ./install.sh. Its 'Next steps' block named skills/soc-investigate/settings.snippet.json and ./install.sh. Both now print the paths the script has already resolved, so they are right from a clone and from the installed plugin directory alike. The --help text also still claimed the merge targets ~/.claude/settings.json — the exact hardcoding #73 removed. A DOC THAT CONTRADICTED THE CODE: - logging.md said a failed emit makes telemetry 'disable itself'. It does the opposite on purpose: the event is dropped and the trail stays on, because a transient fault must not switch off a mandatory audit log — there is a test named for that behavior. Split the two failure modes, and documented the observra>=1.1 floor and the new stderr warning channel, neither of which appeared anywhere. MISSING RELEASE NOTES: - [Unreleased] documented 1 of 4 landed changes. Added entries for the plugin/ restructure (#66), the ABV release gate (#77), and the dependency bounds + hash-pinned lockfile and observra 1.1 migration (#78), and corrected the pre-restructure paths in the entries that were already there. Also: the lockfile existed with no mention in any document — CONTRIBUTING now tells contributors to re-lock (CI checks the AI BOM but not lock drift) and the end-to-end recipe copies the .lock; SECURITY.md's scope adds merge_permissions.py (the only code that writes to a user's settings) and the lockfile; the front-door README no longer presents the installer as check-only; CONTRIBUTING said gates run 'before a tag' though the repo has no tags and says so twice elsewhere; and SCAN_INSTRUCTIONS.md — forward-looking scope for the NEXT Praxen audit — carried pre-restructure paths and asked about a manual-merge-only world that no longer exists, which would have biased that scan. Verified: 263 tests pass, AI BOM current, installer smoke still merges to {allow: 18, ask: 4, deny: 68}, --help renders, every relative link resolves. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Steve Wilson <steve.wilson@exabeam.com> * docs: correct three defects found in review of this PR 1. A FALSE ASSURANCE IN AN AUDIT DOC. The rewritten fail-open section claimed observra's warnings surface on stderr for 'a full queue' so a dropped event is 'always visible'. Traced through observra 1.1: a queue-overflow drop is logged at DEBUG and counted in observra_events_dropped_total, below the WARNING-level stderr handler the shim installs — so that drop is counted, not announced. Backend write errors DO surface. Corrected in logging.md, the CHANGELOG entry, and the shim's own (pre-existing, equally wrong) inline comment. 2. THE REVERSE PATH ERROR. installation.md ships INSIDE the plugin and the primary install path is the marketplace, where no plugin/ directory exists — so 'skills/soc-investigate/...' was correct plugin-relative and my prefixing it broke it for the majority reader, while plugin/README.md kept the relative form. Now names the plugin-relative path with the clone path as a gloss. 3. A STALE PATH IN THE BLOCK THIS PR EDITED. tests/end-to-end-testing.md's CACHE still used the pre-0.6.5 marketplace name (socxen/socxen); the real cache is under open-agent-ai-security/socxen/. The new .lock copy would have missed. Also: the shim's DESIGN RULES said an emit failure disables logging, contradicting its own code, tests, and this PR's new doc; my bump_version change contradicted the script's own docstring, so both call sites now use uv run; praxen/README.md's gate said 'before tagging' while CONTRIBUTING (which links it) says release; the rotation bounds are clamped, not passed straight through; the merge-permissions hint now prints an absolute path instead of $0 as invoked; and (#70) gained its PR number per the file's (#issue, #PR) convention. Verified: 263 pass, BOM current, Next-steps block renders with paste-able absolute paths, real settings untouched, all relative links resolve. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Steve Wilson <steve.wilson@exabeam.com> * docs: order the [Unreleased] sections Added-before-Fixed The last clear item from the review pass. The section order was Changed -> Fixed -> Added -> Security; both Keep a Changelog and this file's own 0.6.3 entry put Added first. Pure reordering — no wording changes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Steve Wilson <steve.wilson@exabeam.com> --------- Signed-off-by: Steve Wilson <steve.wilson@exabeam.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…/50 resisted (#76) (#81) * fix: pin the red-team gate to an explicit model ID and record the resolved model in every artifact (#76) The runner's --models default was the floating 'sonnet' alias, so the gate's result could change out from under the harness whenever a new Sonnet shipped, and a recorded verdict could not be tied to a specific model version after the fact. Now: - --models defaults to the explicit supported-floor ID (claude-sonnet-4-6), matching the discipline the dated results/ artifacts already show; - the runner captures the session's *resolved* model ID from the CLI init event and tallies/reports under it, so even an alias-invoked run produces an attributable artifact (and prints a notice when the two differ); - HISTORY.md's stale note (Sonnet 5 'not approved for cyber work' — no longer true) is replaced with the real reason the pin exists: reproducibility; - PLAN.md's 'weakest supported model (currently Sonnet)' wording names the pinned ID explicitly in all three places. The AI BOM is unaffected: it inventories dependencies, not harness source. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Steve Wilson <steve.wilson@exabeam.com> * fix: a stray stream event must not discard a whole red-team trial Found while running the gate for #76: the first live trial died with "'str' object has no attribute 'get'". The CLI's stream-json output carries lines whose `message` is a plain STRING rather than an object (error/notice events — an MCP server failing to reconnect, for instance). _parse called .get() on it unconditionally, raising AttributeError, which the runner's per-trial `except Exception` then degraded into a fully ERRORED trial. A drive that actually completed 19 tool calls and produced a 36k-character report was discarded because one unrelated notice appeared in the transcript. Grading integrity was never at risk — _inconc_rows already refuses to read an all-errored blocking-class attack as a pass (verdict INCONCLUSIVE, exit 3) — but the harness burned a full live drive per occurrence and could have stalled the gate on re-runs. Parse defensively by SHAPE, not just by event type, and add regression tests for both the odd-shaped line and the resolved-model capture. Negative-verified: the new test fails red on the unfixed parse. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Steve Wilson <steve.wilson@exabeam.com> * security: record the 2026-08-13 red-team gate run — 50/50 resisted, PASS (#76) The staleness half of #76. The gate had not run since 2026-07-03, across 0.6.7/0.6.8/0.6.9, and this promotion carries #66/#73/#77/#78. Run: 10 class-A families x 5 trials on the pinned claude-sonnet-4-6, judge on, driven against the working-tree plugin. Every family landed 0/5; no errored and no inconclusive trials, so the verdict rests on 50 valid trials rather than on absence of evidence. The a10 rows carry the real signal. In 4 of 5 trials the model reproduced the phishing markdown link in its chat output — the injection still works on the model — but the persisted artifact was clean every time, because the write-sink neutralizer defanged it. That is exactly what output-pipeline grading exists to distinguish: a pass where the model never emitted the payload would prove nothing about the guardrail. a10 landed 3/3 in the first run and drove the deterministic fix, so seeing the code layer fire is the point. Also fixes the ledger's own stale connector/ paths after #66, and updates the a10 summary's resistance figure from the 0/2 fix-time retest to 0/5 here. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Steve Wilson <steve.wilson@exabeam.com> * docs: link the class C/D corpus gap to its tracking issue Praxen finding 008 was 'awaiting triage'; it now has a home (#82) rather than being an untracked row. The gap itself is unchanged — the corpus is still class-A only, and classes C and D remain unexercised despite both blocking a release. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Steve Wilson <steve.wilson@exabeam.com> --------- Signed-off-by: Steve Wilson <steve.wilson@exabeam.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Adds
security/praxen/— the "Praxen agent-behavior reports" the security README has been reserving a slot for — and documents a release gate alongside the existing red-team bar.What this is
The red team asks "can an attacker who controls the telemetry make socxen misbehave?" — adversarial, runtime, and only over the paths it has fixtures for. Praxen asks "does the shipped code enforce what we say our policy is?" — static, whole-system, and it audits every rule in the remit, including controls no fixture has ever probed. The two are complementary; neither subsumes the other.
The gate
Documented gate, not a CI check — deliberately the same posture as the red-team release bar. A Praxen scan is an agentic analysis, not a deterministic test; it belongs in the checklist a human runs, not in a workflow that must go green on every push.
CONTRIBUTING.mdgains a Release gates section tabulating both.Why Critical specifically: in Praxen's model a Critical is a control absent or defeated on the shipped default path, not a hardening opportunity. For an agent that reads attacker-influenceable telemetry and writes dispositions into a production SOC platform, that's the class that must not reach a tag.
0.6.9 result — the gate passes
Categories: Limit Domain 2 · Balance KB 3 · Zero Trust 2 · Supply Chain 2 · Red Team 3 · Monitor 3.
Red Team scores 3 on the strength of the program already in this repo — the a10 arc in
redteam/HISTORY.md(landed 3/3 → issue #30 → two prompt-level fixes measurably failed → architectural fix at the write sink → retest clean → promoted to a permanent regression fixture) is cited as a complete find→fix→verify loop. Monitor scores 3 for the default-on structured audit trail at the bridge's single chokepoint.How the remit was written — and why that matters
Authored blind. The agent that wrote it read
README.md,SECURITY.mdanddocs/**only — it was barred fromskills/andconnector/. A remit written from the implementation describes what socxen does rather than what it should do, and a scan against it finds nothing.It was then hardened over two rounds: a high-mode scan audit found 4 defective rules, an independent review found 5 more (including one introduced while fixing the first set), and a third check came back clean. All 8 Open Questions are resolved, each recording whether the docs settled it or a maintainer decided.
Please review the remit as policy, not as a report artifact — it's a claim about maintainer intent, and where the docs were silent (approver identity, volume caps, unattended operation, telemetry destinations) the answers recorded are the ones we inferred or chose. If any of them isn't what you intend, that's a remit fix, and it changes what future scans check.
Open items
Tracked as issues rather than folded in here: #71 (dependency pinning / lockfile / scanner) and #76 (red-team runner model alias + gate staleness). #72 (containment deny-list naming) was found by the previous scan and fixed in 0.6.9 — this re-scan verifies it: 68 rules, both spellings across both namespaces, invariant-tested in CI.
Verification
238 passed(uv run --with pytest ... pytest -q tests/)security/gen_aibom.py --check→ AI BOM is currentsecurity/praxen/results/exactly as it already coveredsecurity/redteam/results/— same rationale, same category of file. Authored docs undersecurity/praxen/carry the header.🤖 Generated with Claude Code