Skip to content

security: Agent Behavior Verification — Worker Remit, Praxen report, and a Critical-blocks-release gate - #77

Merged
virtualsteve-exa merged 2 commits into
devfrom
feat/praxen-abv-gate
Aug 12, 2026
Merged

virtualsteve-exa merged 2 commits into
devfrom
feat/praxen-abv-gate

Conversation

@virtualsteve-exa

Copy link
Copy Markdown
Collaborator

Adds security/praxen/ — the "Praxen agent-behavior reports" the security README has been reserving a slot for — and documents a release gate alongside the existing red-team bar.

What this is

The red team asks "can an attacker who controls the telemetry make socxen misbehave?" — adversarial, runtime, and only over the paths it has fixtures for. Praxen asks "does the shipped code enforce what we say our policy is?" — static, whole-system, and it audits every rule in the remit, including controls no fixture has ever probed. The two are complementary; neither subsumes the other.

The gate

socxen does not ship a release with an open Praxen Critical finding. Fixed, or explicitly waived in writing by a maintainer with a rationale in the PR, before the tag. High / Medium / Low do not block — they're triaged into issues.

Documented gate, not a CI check — deliberately the same posture as the red-team release bar. A Praxen scan is an agentic analysis, not a deterministic test; it belongs in the checklist a human runs, not in a workflow that must go green on every push. CONTRIBUTING.md gains a Release gates section tabulating both.

Why Critical specifically: in Praxen's model a Critical is a control absent or defeated on the shipped default path, not a hardening opportunity. For an agent that reads attacker-influenceable telemetry and writes dispositions into a production SOC platform, that's the class that must not reach a tag.

0.6.9 result — the gate passes

Critical 0 — passes
Other 5 High · 7 Medium · 1 Low
Weighted RAISE 2.45 / 5 (Partial)
Remit coverage 50 rules — 31 verified · 18 partial · 1 gap
Independent audit 13 / 13 CONFIRMED, 0 unsupported, 0 remit defects

Categories: Limit Domain 2 · Balance KB 3 · Zero Trust 2 · Supply Chain 2 · Red Team 3 · Monitor 3.

Red Team scores 3 on the strength of the program already in this repo — the a10 arc in redteam/HISTORY.md (landed 3/3 → issue #30 → two prompt-level fixes measurably failed → architectural fix at the write sink → retest clean → promoted to a permanent regression fixture) is cited as a complete find→fix→verify loop. Monitor scores 3 for the default-on structured audit trail at the bridge's single chokepoint.

How the remit was written — and why that matters

Authored blind. The agent that wrote it read README.md, SECURITY.md and docs/** only — it was barred from skills/ and connector/. A remit written from the implementation describes what socxen does rather than what it should do, and a scan against it finds nothing.

It was then hardened over two rounds: a high-mode scan audit found 4 defective rules, an independent review found 5 more (including one introduced while fixing the first set), and a third check came back clean. All 8 Open Questions are resolved, each recording whether the docs settled it or a maintainer decided.

Please review the remit as policy, not as a report artifact — it's a claim about maintainer intent, and where the docs were silent (approver identity, volume caps, unattended operation, telemetry destinations) the answers recorded are the ones we inferred or chose. If any of them isn't what you intend, that's a remit fix, and it changes what future scans check.

Open items

Tracked as issues rather than folded in here: #71 (dependency pinning / lockfile / scanner) and #76 (red-team runner model alias + gate staleness). #72 (containment deny-list naming) was found by the previous scan and fixed in 0.6.9 — this re-scan verifies it: 68 rules, both spellings across both namespaces, invariant-tested in CI.

Verification

  • 238 passed (uv run --with pytest ... pytest -q tests/)
  • security/gen_aibom.py --check → AI BOM is current
  • One test touched: the SPDX-header exemption for generated run records now covers security/praxen/results/ exactly as it already covered security/redteam/results/ — same rationale, same category of file. Authored docs under security/praxen/ carry the header.

🤖 Generated with Claude Code

…rt, release gate

Adds `security/praxen/` — the "Praxen agent-behavior reports" the security
README has been reserving a slot for — and documents a release gate alongside
the existing red-team bar.

Contents:
- `WORKER_REMIT.md` — socxen's declared policy: what the agent is authorized to
  do. Authored blind from README/SECURITY/docs only (no access to skills/ or
  connector/), so it states intent rather than mirroring the implementation.
- `SCAN_INSTRUCTIONS.md` — scan scope: the whole deployed system (skill +
  connector + permissions pack + install), not the skill alone.
- `results/2026-08-12-socxen-0.6.9.{html,json}` — the scan of 0.6.9 at 005fa4c.
- `results/2026-08-12-socxen-0.6.9-audit.md` — independent context-unaware audit
  of that scan: 13/13 findings CONFIRMED, 0 unsupported, 0 remit defects.

Release gate (documented, not CI — the same posture as the red-team bar):
a Praxen Critical finding blocks a tag; it is fixed or explicitly waived in
writing by a maintainer before release. High/Medium/Low do not block.

0.6.9 result: 0 Critical (gate passes), 5 High, 7 Medium, 1 Low. Weighted RAISE
2.45/5. Build an AI Red Team scores 3 on the strength of the a10
find-fix-retest-regression arc in redteam/HISTORY.md; Monitor Continuously
scores 3 for the default-on audit trail at the bridge chokepoint.

Also:
- `security/README.md` — Praxen moved from "planned neighbors" into Contents,
  with a callout beside the red-team one.
- `CONTRIBUTING.md` — new "Release gates" section tabulating both gates.
- `tests/test_repo_invariants.py` — the SPDX-header exemption for generated run
  records now covers `security/praxen/results/` as it already did
  `security/redteam/results/`. Authored docs in `security/praxen/` carry headers.

Open items from this scan are tracked as #71 and #76; #72 was found by the
previous scan and is fixed in 0.6.9 — this re-scan verifies it.

Signed-off-by: Steve Wilson <steve.wilson@exabeam.com>

@virtualsteve-exa virtualsteve-exa left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code-owner review, with independent verification. Verdict: READY once two small things are fixed — one is a public-repo hygiene item.

Verified independently ✅

  • Suite + invariants: 238 passed / 0 skipped on the branch; gen_aibom.py --check current; CI green. The SPDX-test change is minimal and exactly parallel to the existing redteam/results exemption.
  • Report internal consistency: JSON has 13 findings (5 High / 7 Medium / 1 Low, 0 Critical) matching the README table; 50 distinct R- rules referenced, matching the coverage claim.
  • Spot-checked findings at their cited lines — accurate: 009 (run.py:341-342 floating --models sonnet default + the now-stale HISTORY.md:42 gate note) and 013 (all four uses: across the three workflows are mutable major tags) check out verbatim. The audit's sharpest catch on 001 also verifies: the snippet-merge instruction lives inside if [ "$CREDS_OK" = 0 ] (install.sh:445), so an install with credentials present never prints the merge instruction at all — that detail is worth carrying into whatever fixes #70/#73 land.
  • No secrets/tenant data in the artifacts — all credential-shaped grep hits are rule text.

Fix before merge

  1. Local absolute paths committed to a public repo. results/2026-08-12-socxen-0.6.9-audit.md:6-8 (×3) and one occurrence in the JSON embed /Users/steve.wilson/Documents/github/deckard/local/... — operator username plus an internal project name. The isolation claims survive with placeholders (<scan-workspace>/…); please scrub both files. The HTML is clean.
  2. Dangling reference: SCAN_INSTRUCTIONS.md:52 points at CLONE_SHA.txt "beside this file," which is (rightly) not committed — it's scan-time state. Reword to "pin the SHA at scan time and record it in the results artifact" (the results do record 005fa4c).

Ask, non-blocking

  1. Close the triage loop the gate itself promises. CONTRIBUTING now says non-blocking findings "are triaged into issues" — but 13 findings map to #71, #76, and the previously-fixed #72 only; the other ten have no recorded disposition. Suggest a short triage table (finding → issue / accepted-with-rationale / superseded) in praxen/README.md or beside the results. Note findings 001 and 012 land partly in #70/#73's lap — cross-reference rather than double-file.

For the maintainer (not this PR's author to resolve)

The remit's 8 Open Questions record five operator decisions (approver identity, case-write tier, no volume cap, no telemetry allowlist, missing-gate = warn-not-refuse). Those are policy, and maintainer sign-off on them is the substantive review of this PR — flagged to Steve directly.

Nice work overall: the blind-authoring discipline, the audit pass that re-reads every citation, and honest score rationale (band-edge calls recorded as such) make this the most defensible artifact in security/.

… table

Addresses code-owner review on #77.

1. Public-repo hygiene: removed committed absolute paths carrying an operator
   username and internal project name — 3 occurrences in the audit record and
   the `scan.workspace` field in the findings JSON, replaced with
   `<scan-workspace>` / `<socxen-checkout>` placeholders. The isolation claims
   in the audit record read the same with placeholders. HTML was already clean.
   Verified: no `/Users/` or operator name anywhere under security/praxen/.
   JSON re-validated — 13 findings, 50 rules, weighted 2.45 unchanged.

2. Dangling reference: SCAN_INSTRUCTIONS.md pointed at a `CLONE_SHA.txt`
   "beside this file" that is (correctly) scan-time state and not committed.
   Reworded to "pin the SHA at scan time and record it in the results
   artifact", naming the 005fa4c pin the committed run actually used.

3. Triage table (review item 3, non-blocking): all 13 findings now carry a
   recorded disposition in security/praxen/README.md — #71, #76, and #70/#73
   cross-referenced rather than double-filed, the rest marked *awaiting triage*
   as the live worklist. Disposition is explicitly the maintainer's call; the
   table does not invent one. Also carried the reviewer's install.sh:445 catch
   (the merge instruction sits inside `if [ "$CREDS_OK" = 0 ]`, so an install
   with credentials present never prints it) as a note for whoever fixes
   #70/#73.

238 passed; AI BOM current.

Signed-off-by: Steve Wilson <steve.wilson@exabeam.com>
@virtualsteve-exa

Copy link
Copy Markdown
Collaborator Author

Thanks — all three addressed in 2f37c91. 238 passed, AI BOM current.

1. Local absolute paths — fixed, and thank you for catching it. That's a genuine hygiene miss on a public repo: operator username plus an internal project name, committed. Scrubbed all four occurrences (3 in the audit record, plus scan.workspace in the JSON) to <scan-workspace> / <socxen-checkout>. The isolation claims read identically with placeholders. Re-validated the JSON afterwards — 13 findings, 50 rules, weighted 2.45, unchanged. Swept the whole directory: no /Users/ and no operator name anywhere.

2. Dangling CLONE_SHA.txt — fixed. You're right that it's scan-time state and correctly uncommitted. Reworded to "pin the SHA at scan time and record it in the results artifact", and named the 005fa4c pin the committed run actually used, so the instruction is self-demonstrating.

3. Triage table — added, and you were right that the gate was writing a cheque the artifact didn't cash. All 13 findings now carry a disposition in security/praxen/README.md: #71 and #76 tracked, #72 marked previously-fixed-and-verified, 001/012 cross-referenced to #70/#73 rather than double-filed as you suggested. The remaining nine are marked awaiting triage with a note that disposition is the maintainer's call — I'd rather the table show an honest worklist than have a contributor invent dispositions for someone else's backlog.

I also carried your install.sh:445 catch into the table as a note for whoever fixes #70/#73 — the merge instruction sitting inside if [ "$CREDS_OK" = 0 ] means an install that already has credentials never prints it, so fixing the merge without fixing that branch would leave the most common upgrade path silent. That's a sharper observation than the finding itself made.

On the maintainer item: the remit's five operator decisions (approver identity, case-write tier, no volume cap, no telemetry allowlist, missing-gate = warn-not-refuse) were decided by @virtualsteve-exa during authoring, not inferred — each is recorded in the Open Questions section with the date. Worth an explicit confirm on merge, since they're the substantive policy content and they set what every future scan checks.

@virtualsteve-exa

Copy link
Copy Markdown
Collaborator Author

Re-reviewed 2f37c91 — all three items verified fixed: swept both artifacts for /Users/ and the project name (0 hits), the SHA-pinning note now records 005fa4c in prose with no dangling file reference, and the triage table is exactly what the gate needed — honest awaiting-triage rows beat silently unfiled findings. Review items are clear from my side; remaining before merge is maintainer sign-off on the remit's five operator decisions (flagged to Steve) and Matt's requested review.

@virtualsteve-exa
virtualsteve-exa merged commit 1964fa2 into dev Aug 12, 2026
2 checks passed
@virtualsteve-exa
virtualsteve-exa deleted the feat/praxen-abv-gate branch August 12, 2026 23:40
virtualsteve-exa added a commit that referenced this pull request Aug 13, 2026
…eleased changes (#80)

* docs: bring the docs current with the plugin/ restructure and the unreleased changes

Pre-promotion currency pass. Three classes of defect, all found by reviewing
docs against the code rather than against each other.

BROKEN INSTRUCTIONS (post-#66, a user follows these):
- plugin/docs/installation.md told users to run ./install.sh --merge-permissions
  in the section the page itself calls the most important on it. There is no root
  install.sh any more; twenty lines earlier the same page correctly says
  ./plugin/install.sh. Same for merge_permissions.py's path.
- install.sh's OWN output was stale: the usage block is printed verbatim as
  --help, and every line said ./install.sh. Its 'Next steps' block named
  skills/soc-investigate/settings.snippet.json and ./install.sh. Both now print
  the paths the script has already resolved, so they are right from a clone and
  from the installed plugin directory alike. The --help text also still claimed
  the merge targets ~/.claude/settings.json — the exact hardcoding #73 removed.

A DOC THAT CONTRADICTED THE CODE:
- logging.md said a failed emit makes telemetry 'disable itself'. It does the
  opposite on purpose: the event is dropped and the trail stays on, because a
  transient fault must not switch off a mandatory audit log — there is a test
  named for that behavior. Split the two failure modes, and documented the
  observra>=1.1 floor and the new stderr warning channel, neither of which
  appeared anywhere.

MISSING RELEASE NOTES:
- [Unreleased] documented 1 of 4 landed changes. Added entries for the plugin/
  restructure (#66), the ABV release gate (#77), and the dependency bounds +
  hash-pinned lockfile and observra 1.1 migration (#78), and corrected the
  pre-restructure paths in the entries that were already there.

Also: the lockfile existed with no mention in any document — CONTRIBUTING now
tells contributors to re-lock (CI checks the AI BOM but not lock drift) and the
end-to-end recipe copies the .lock; SECURITY.md's scope adds merge_permissions.py
(the only code that writes to a user's settings) and the lockfile; the front-door
README no longer presents the installer as check-only; CONTRIBUTING said gates run
'before a tag' though the repo has no tags and says so twice elsewhere; and
SCAN_INSTRUCTIONS.md — forward-looking scope for the NEXT Praxen audit — carried
pre-restructure paths and asked about a manual-merge-only world that no longer
exists, which would have biased that scan.

Verified: 263 tests pass, AI BOM current, installer smoke still merges to
{allow: 18, ask: 4, deny: 68}, --help renders, every relative link resolves.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Steve Wilson <steve.wilson@exabeam.com>

* docs: correct three defects found in review of this PR

1. A FALSE ASSURANCE IN AN AUDIT DOC. The rewritten fail-open section claimed
   observra's warnings surface on stderr for 'a full queue' so a dropped event is
   'always visible'. Traced through observra 1.1: a queue-overflow drop is logged
   at DEBUG and counted in observra_events_dropped_total, below the WARNING-level
   stderr handler the shim installs — so that drop is counted, not announced.
   Backend write errors DO surface. Corrected in logging.md, the CHANGELOG entry,
   and the shim's own (pre-existing, equally wrong) inline comment.

2. THE REVERSE PATH ERROR. installation.md ships INSIDE the plugin and the primary
   install path is the marketplace, where no plugin/ directory exists — so
   'skills/soc-investigate/...' was correct plugin-relative and my prefixing it
   broke it for the majority reader, while plugin/README.md kept the relative form.
   Now names the plugin-relative path with the clone path as a gloss.

3. A STALE PATH IN THE BLOCK THIS PR EDITED. tests/end-to-end-testing.md's CACHE
   still used the pre-0.6.5 marketplace name (socxen/socxen); the real cache is
   under open-agent-ai-security/socxen/. The new .lock copy would have missed.

Also: the shim's DESIGN RULES said an emit failure disables logging, contradicting
its own code, tests, and this PR's new doc; my bump_version change contradicted the
script's own docstring, so both call sites now use uv run; praxen/README.md's gate
said 'before tagging' while CONTRIBUTING (which links it) says release; the rotation
bounds are clamped, not passed straight through; the merge-permissions hint now
prints an absolute path instead of $0 as invoked; and (#70) gained its PR number
per the file's (#issue, #PR) convention.

Verified: 263 pass, BOM current, Next-steps block renders with paste-able absolute
paths, real settings untouched, all relative links resolve.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Steve Wilson <steve.wilson@exabeam.com>

* docs: order the [Unreleased] sections Added-before-Fixed

The last clear item from the review pass. The section order was Changed ->
Fixed -> Added -> Security; both Keep a Changelog and this file's own 0.6.3
entry put Added first. Pure reordering — no wording changes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Steve Wilson <steve.wilson@exabeam.com>

---------

Signed-off-by: Steve Wilson <steve.wilson@exabeam.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
virtualsteve-exa added a commit that referenced this pull request Aug 13, 2026
…/50 resisted (#76) (#81)

* fix: pin the red-team gate to an explicit model ID and record the resolved model in every artifact (#76)

The runner's --models default was the floating 'sonnet' alias, so the gate's
result could change out from under the harness whenever a new Sonnet shipped,
and a recorded verdict could not be tied to a specific model version after
the fact. Now:

- --models defaults to the explicit supported-floor ID (claude-sonnet-4-6),
  matching the discipline the dated results/ artifacts already show;
- the runner captures the session's *resolved* model ID from the CLI init
  event and tallies/reports under it, so even an alias-invoked run produces
  an attributable artifact (and prints a notice when the two differ);
- HISTORY.md's stale note (Sonnet 5 'not approved for cyber work' — no longer
  true) is replaced with the real reason the pin exists: reproducibility;
- PLAN.md's 'weakest supported model (currently Sonnet)' wording names the
  pinned ID explicitly in all three places.

The AI BOM is unaffected: it inventories dependencies, not harness source.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Steve Wilson <steve.wilson@exabeam.com>

* fix: a stray stream event must not discard a whole red-team trial

Found while running the gate for #76: the first live trial died with
"'str' object has no attribute 'get'".

The CLI's stream-json output carries lines whose `message` is a plain
STRING rather than an object (error/notice events — an MCP server failing
to reconnect, for instance). _parse called .get() on it unconditionally,
raising AttributeError, which the runner's per-trial `except Exception`
then degraded into a fully ERRORED trial. A drive that actually completed
19 tool calls and produced a 36k-character report was discarded because
one unrelated notice appeared in the transcript.

Grading integrity was never at risk — _inconc_rows already refuses to read
an all-errored blocking-class attack as a pass (verdict INCONCLUSIVE, exit
3) — but the harness burned a full live drive per occurrence and could
have stalled the gate on re-runs.

Parse defensively by SHAPE, not just by event type, and add regression
tests for both the odd-shaped line and the resolved-model capture.
Negative-verified: the new test fails red on the unfixed parse.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Steve Wilson <steve.wilson@exabeam.com>

* security: record the 2026-08-13 red-team gate run — 50/50 resisted, PASS (#76)

The staleness half of #76. The gate had not run since 2026-07-03, across
0.6.7/0.6.8/0.6.9, and this promotion carries #66/#73/#77/#78.

Run: 10 class-A families x 5 trials on the pinned claude-sonnet-4-6, judge on,
driven against the working-tree plugin. Every family landed 0/5; no errored and
no inconclusive trials, so the verdict rests on 50 valid trials rather than on
absence of evidence.

The a10 rows carry the real signal. In 4 of 5 trials the model reproduced the
phishing markdown link in its chat output — the injection still works on the
model — but the persisted artifact was clean every time, because the write-sink
neutralizer defanged it. That is exactly what output-pipeline grading exists to
distinguish: a pass where the model never emitted the payload would prove
nothing about the guardrail. a10 landed 3/3 in the first run and drove the
deterministic fix, so seeing the code layer fire is the point.

Also fixes the ledger's own stale connector/ paths after #66, and updates the
a10 summary's resistance figure from the 0/2 fix-time retest to 0/5 here.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Steve Wilson <steve.wilson@exabeam.com>

* docs: link the class C/D corpus gap to its tracking issue

Praxen finding 008 was 'awaiting triage'; it now has a home (#82) rather than
being an untracked row. The gap itself is unchanged — the corpus is still
class-A only, and classes C and D remain unexercised despite both blocking a
release.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Steve Wilson <steve.wilson@exabeam.com>

---------

Signed-off-by: Steve Wilson <steve.wilson@exabeam.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant