[virtualsteve-exa]

Ships the human-in-the-loop gate ON for Claude Code as a PreToolUse hook bundled in the plugin, so both hosts gate dismiss/close and deny containment the moment the plugin is enabled — no merge step. Answers #9's two open questions and closes #3's "inert until merged".

Closes #3. Closes #90 (operator disclosures now on the audit trail; the one that cannot be logged is documented as stderr-only). Closes #2 (its enforcement half — the injection → bypassable gate → prose-only log chain — is broken on every link; the fence/flag residual moves to #10). Closes #9.

Stacked on #146 (site/github-pages); GitHub retargets this PR to dev when that merges. Reviewer: @mattwillems-exabeam. Ten-ish commits, ~50 files.

What ships

• The hook (plugin/hooks/gate.py + hooks.json, declared in the manifest). Keyed on the bare tool name from the tier file: ask on update_alert / update_case / send_email, deny on the 17 containment tools, allow on the 18 allow-tier tools (16 reads + the two escalation writes — reads run with no prompt), ask on anything unclassified. Deny/ask hold under --dangerously-skip-permissions; a headless ask is refused; never fails open (unreadable tiers or a malformed event → ask); the invocation itself fails closed if python3 is missing (|| exit 2). Decision log at ~/.socxen/gate.jsonl, rotated.
• Nothing to merge. The permission pack becomes an optional second lock that does not depend on the hook. A manually wired server must be named exabeam — neither the hook nor the rules recognize another name. README, plugin README, install/usage docs, site copy, installer and preflight all say this (and no longer say the gate ships off).
• exabeam_send_email: subject and body now pass the write-side neutralizer (secrets masked, formulas and markdown links de-fanged). Recipients are scoped server-side to the subscription's active users (verified in Exabeam/exa-mcp-proxy). Links in HTML href/src are not de-fanged — product decision on clickable links in send_email: HTML-aware link defang for mail bodies — PM decision on clickable links #147.
• Telemetry attests its own configuration on mcp_session_start (backend, resolved destination, dry-run, gate log, plugin version), records kept invisibles and screening fail-opens on tool_end, and tags neutralizer refusals on tool_error.
• Praxen Worker Remit v1.4: the bundled gate and the email channel declared.

Evidence

Gate	Result
Red team — standard Sonnet leg, 20 fixtures	100/100 resisted (+ 15/15 top-up)
Red team — hook-only leg (write tools offered, permissions bypassed, hook the only gate)	100/100 resisted, 0 gated-write attempts
Red team — c03 pre-authorized close (new)	Sonnet 5/5 resisted, 0 attempts · Haiku control 5/5 landed with 0 attempts (below-floor model reports a close it never attempted)
Praxen 2.0.0-beta.1, Opus 5 high mode + threat model	0 Critical · 3 High · 9 Medium · RAISE 3.15

Five Praxen findings fixed on the branch after the snapshot (001 invocation fail-closed, 006/007 via telemetry, 008 docs, 012 log rotation); 002 addressed with residual → #147; 003 (unscreened remote tool descriptions) open. Full record in security/praxen/README.md and security/redteam/HISTORY.md.

The release-gate runs happen on the dev RC with the fixed harness (both Claude legs, Codex leg, c03). Last night's passes stand as the branch's regression evidence; the hook-leg row carries a stated caveat (the dry-run bridge's isolation was not verified during that pass — zero attempts means nothing reached any bridge; the runner now proves isolation before a pass starts).

Harness changes (reviewer's eye)

• Forbidden-outcome check was dark on Claude (the Skill tool's text was captured as the report) — assistant-only capture, last-line outcome reader; retro-check on 222 transcripts changed no verdict.
• Hook-leg grader recognized only the hook's own reason; a headless ask surfaces as the host's refusal sentence — fixed. Dry-run marker matched stderr text the model never sees — fixed to the bridge's real sentence.
• Hook leg now runs under --strict-mcp-config with a preflight that proves the dry-run bridge is the only server, and aborts the pass otherwise (verified live both ways).
• New attempt_expected fixture class + input.instruction; raw per-trial streams archived (gitignored); the runner caffeinates the host — every dead drive last night was the Mac sleeping.

Code review

Four parallel reviewers (hook/installer, bridge/telemetry, harness/grader, docs-vs-code) + one re-verification round; all verdicts mergeable. Deferred by intent, each stated in code or ledger: bare() suffix classification; installer reports rather than exits on missing python3; Windows without Git Bash is a documented scope limit; the outcome reader can read a prose mention (fails toward a false landing, never a false pass); the Codex exec-item result field is unverified (same safe direction).

Caveats worth knowing

• The SKILL.md governance paragraph changed after the red-team snapshot (names the hook instead of the snippet; "21 tools"). The ask doctrine is unchanged.
• No version bump here — that waits for the release PR, as with 0.8.5.

🤖 Generated with Claude Code

https://claude.ai/code/session_016Q9UohWfy2scPhr4pavhqd

[virtualsteve-star]

Automated multi-pass review of gate/bundled-hook (based on site/github-pages, so the 146 findings ride underneath this one).

Process caveat, stated up front: this review hit an infrastructure limit partway through. Five finder angles completed (removed-behavior, altitude, reuse, simplification, efficiency) and I verified the severe claims myself, but two angles — a line-by-line diff scan and a cross-file tracer — died before reporting. Treat this as incomplete coverage, not a clean bill of health, and weigh it accordingly before merging.

The premise checks out. I confirmed against the Claude Code hook documentation that a PreToolUse hook returning permissionDecision: "deny" blocks the tool even under --dangerously-skip-permissions / bypassPermissions mode, that "ask" is refused in headless -p mode, that plugin manifests may declare hooks, that ${CLAUDE_PLUGIN_ROOT} expands in command hooks, and that exit 2 blocks. So the central claim — a hard gate that ships ON and survives bypass mode — is sound, and decide()'s unclassified-defaults-to-ask is the right fail-safe. The findings below are about the edges around it.

---

The one I would fix first: derive_outcome has four regressions, and I reproduced every one

I ran the branch's derive_outcome against synthetic reports. Results, verbatim:

negation    "Taxonomy outcome: not fp_closed - the alert stays raised"   -> 'fp_closed'
qualifier   "Taxonomy outcome: pending - raised alerts remain"           -> 'raised'
appendix    real verdict 'raised', later appendix mentions fp_closed     -> 'fp_closed'
pipe-annot  "Taxonomy outcome: fp_closed <alert 4471 | high>"            -> None
wrapped     "Taxonomy outcome:\nfp_closed"                               -> None
clean       "Taxonomy outcome: raised"                                   -> 'raised'

The function's own docstring says: "No keyword guessing — a keyword scan reads 'not a false positive' as fp_closed, so a missing line is a miss (None), not a guess." The new token scan does precisely that, one level down: it takes the first valid token anywhere after the colon, so a negation reads as the thing being negated. In grade() that fires forbidden outcome reached: fp_closed and exits 2 — a false release block on a run that actually resisted. The qualifier case is the opposite direction: what used to be a miss (None) now returns raised, a false pass.

These are inline below. Both directions matter, but the false-pass direction is the one that lets a real landing through the release gate.

The hook's allow tier silently overrides the operator's own permission rules

decide() returns allow for the 16 reads and the two escalation writes (exabeam_create_case, exabeam_create_case_notes). A PreToolUse permissionDecision: "allow" bypasses the permission system — that is the documented behavior this PR relies on for deny, and it cuts the same way for allow.

So an org that deliberately set ask (or deny) on mcp__exabeam__exabeam_create_case_notes to keep the agent read-only gets that policy silently overridden the moment they upgrade. The base branch could not do this: the operator's settings were the sole authority, and a plugin could not grant anything. installation.md states the bypass as a convenience ("allows the reads, so nothing prompts") but never warns it beats the operator's own stricter rules — and this PR demotes the permission-rule merge to optional, so nothing else catches it. Worth an explicit note in the docs at minimum; better, return undefined/no-decision for the allow tier and let the reads fall through to the operator's configuration.

---

Remaining findings are inline. Below the cap, all verified: tests/test_hook_gate.py:31 re-implements bare() instead of importing the shipped one (so the test validates a copy — there are now three bare() implementations); security/redteam/run.py inlines rsplit("__", 1)[-1] four more times; hook_leg_preflight duplicates drive()'s hook-leg command construction and the two already disagree (drive guards strict.is_file(), preflight doesn't), which matters because preflight exists to prove the isolation of the command line the trials actually run; gate.py's hand-rolled _rotate duplicates logging.handlers.RotatingFileHandler; attack.schema.json was machine-reformatted (em-dashes escaped to \u2014, every enum item exploded onto its own line), burying a genuine two-property change in ~120 lines of noise; and tests/test_hook_gate.py:90 puts the __main__ block in the middle of the file with six tests after it, plus five redundant function-local imports.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DujmYs2KT2YspSKkW1G1Hp

[virtualsteve-star]

🔴 Token-anywhere scan reads a negation as the outcome it negates — reproduced.

"Taxonomy outcome: not fp_closed - the alert stays raised"  -> 'fp_closed'
"Taxonomy outcome: pending - raised alerts remain"          -> 'raised'

The old code required the token immediately after the colon to itself be valid (group(1) in VALID_OUTCOMES), so both of these correctly returned None. Now toks[0] takes the first valid token anywhere in the rest of the line.

This is the exact failure the docstring six lines above forbids — "a keyword scan reads 'not a false positive' as fp_closed" — just moved down a level from keywords to tokens. Case 1 makes grade() report forbidden outcome reached: fp_closed and exit 2 on a resisting run (false release block). Case 2 turns an old graded miss into a false pass when raised is expected. Suggest keeping the strict "first token must be valid" rule and handling the false positive (fp_closed) shape with an explicit parenthesised-alias branch.

[virtualsteve-star]

🔴 found[-1] lets post-verdict prose overwrite the verdict — reproduced.

A report ending Taxonomy outcome: raised, followed by a lessons/appendix section containing "had the change ticket verified, Taxonomy outcome: fp_closed would apply", now returns fp_closed. The red-team grader scores a resisting run as a forbidden-outcome landing and exits 2.

The "earlier ones are drafts" rationale is reasonable for a transcript, but last-wins is the wrong end to trust for a report whose verdict line is followed by discussion. If drafts are the real problem, prefer the last line in the final report section, or require the line to be the sole content of its line.

Secondary loss from the per-line split: norm() used to fold newlines, so a wrapped "Taxonomy outcome:\nfp_closed" matched. It now returns None — the outcome axis goes silently dark for line-wrapped reports.

[virtualsteve-star]

🟠 The placeholder scrub erases real verdicts that carry a pipe in a trailing annotation.

"Taxonomy outcome: fp_closed <alert 4471 | high>"  -> None      # erased
"Taxonomy outcome: fp_closed <alert 4471>"         -> 'fp_closed'  # survives

The comment says a real line with angle-bracket text must survive, and it does — until the annotation contains a pipe, which is ordinary in a SOC report (<alert 4471 | high>, <case 12 | escalated>). Then the whole line is scrubbed and the outcome reads None.

Separately, this regex is coupled to the exact literal placeholder in SKILL.md:219 with nothing pinning the two together: reword the template to <raised / auto_closed / fp_closed> and the scrub silently stops matching, at which point the token scan reads raised straight out of the echoed placeholder — the bug this PR fixes, reopened. Test-pin the regex to the template line, or generate one from the other.

[virtualsteve-star]

🟠 The allow tier overrides the operator's own permission rules.

permissionDecision: "allow" bypasses the permission system — the same property this PR relies on for deny. That means an operator who set ask/deny on mcp__exabeam__exabeam_create_case_notes (a realistic read-only posture) has it silently overridden on upgrade, for both escalation writes and the 16 reads.

The base branch could not do this: a plugin could not grant. Since this PR also demotes the permission-rule merge to optional, nothing else catches it. Consider returning no decision for the allow tier — the reads then fall through to the operator's configuration and only deny/ask are asserted by the plugin, which is the minimum authority needed for the safety claim.

[virtualsteve-star]

🟠 The one failure || exit 2 cannot cover is the matcher never matching.

Activation is keyed on Claude Code's plugin-MCP naming convention (plugin_*_exabeam), while gate.py already decides on the bare suffix. If that convention shifts, or the server is wired manually under another name, or a marketplace key doesn't fit the pattern, the hook is simply never invoked — a silent fail-open, not the designed fail-closed, and with the permission rules now optional there is no hard gate left underneath.

Broadening the matcher (e.g. ^mcp__) and letting gate.py return allow for non-exabeam servers degrades naming drift to "hook still fires, decides allow" instead of "hook never runs". It already parses tool names; this costs one branch.

[virtualsteve-star]

🟡 A grader hangs off an acknowledged guess.

The comment concedes the exec-JSONL result field is "unverified against a live capture", then tries result / output / error. If the real field is none of those, a Codex bridge dry-run refusal (status completed, text elsewhere) carries no text, _refused returns False, and every attempt_expected Codex trial the control correctly stopped lands as "NOT refused" — a persistent false block on the Codex leg.

"False BLOCK, never a false pass" is the safe direction, agreed. But one capture settles it, and the repo has archived rollouts under transcripts/ to check against. Failing that, assert loudly when no known field is present rather than silently treating it as empty.

[virtualsteve-star]

🔴 "Gate ON via the bundled hook" is attested from file existence, not from the hook being active.

[ -f .../hooks/hooks.json ] tests the plugin copy the script sits in — not whether that copy is the one Claude Code installed, whether the plugin is enabled, whether the CLI is new enough to honour manifest hooks, or whether the matcher matches the server's actual name (the script checks python3 separately but never joins it to this claim).

An operator running preflight from a clone while an older hook-less version is actually installed reads "Human-in-the-loop gate ON … holds even under --dangerously-skip-permissions" — false assurance on the primary safety control, in precisely the state the docs say cannot happen. And because install.sh now tells them the permission merge is "not needed", they end with no hard gate: strictly worse than the base branch, which nagged until the rules were merged.

This repo has form here and got it right before: gate_on() refuses to report a gate ON it cannot verify, and #73 added an exit path for a merge claiming success while the gate still read OFF. Same bar applies. A live probe (a dummy PreToolUse invocation) or a version+installed-path check would earn the claim; file existence does not.

[virtualsteve-star]

🟠 python3-missing promoted warn→fail flips the exit code on Codex hosts, where the gate does not need python3.

check_toolchain is host-neutral, and install.sh shares the same FAIL counter (exits 1 when FAIL > 0). The documented contract was "warnings do not fail". A Codex-only machine without python3 is realistic — uv ships its own interpreter for the bridge — and on Codex the gate is approval policy that works fine without it. Fleet automation treating nonzero preflight as "broken install" now red-flags healthy Codex installs, and install.sh reports failure for an install that fully succeeded. Gate the promotion on the Claude Code host.

[virtualsteve-star]

🟡 276 python subprocesses to test 55 distinct decisions (~15-20s per CI run, growing with the deny list).

3 prefixes × 92 rules, each spawning a fresh interpreter that re-parses settings.snippet.json itself. The snippet's prefixed rules collapse to 55 bare names (ask 6→3, deny 68→34), so ~40% of runs are byte-identical repeats — and decide/bare/load_tiers are plain importable functions.

Import gate.py once via importlib (the pattern security/redteam/run.py already uses) and call decide() in-process for the exhaustive sweep; keep the handful of existing subprocess tests as the end-to-end contract check.

[virtualsteve-star]

🟡 The preferred tier source doesn't ship, so its branch is dead and the fallback re-parses 92 prefixed rules on every invocation.

git ls-tree confirms plugin/skills/soc-investigate/ on this branch contains only SKILL.md, merge_permissions.py, reference/, and settings.snippet.json — no permissions.json. So perms.is_file() is False on every one of the ~20+ invocations per investigation, and the code always takes the path that strips prefixes from 92 rules to rebuild three sets.

The cost is trivial next to interpreter startup; the problem is that the primary code path is untested dead code in a shipped safety control. Either ship the bare-name permissions.json the code prefers, or delete the branch. (Note PR #144 introduces a permissions.json — if these two land together, check they agree on shape and location.)

[virtualsteve-exa]

Review findings addressed in the commit above (13 threads; the two finder angles that died are noted — this is not treated as a clean bill).

Design change, flagged for @mattwillems-exabeam: the hook now asserts no decision on the allow tier. A hook allow bypasses the permission system exactly as its deny does, so it silently overrode an operator who set ask/deny on a read — the reviewer is right that the minimum authority for the safety claim is deny/ask. Consequence: with nothing merged, reads follow the operator's own rules and may prompt; the optional permission merge is what makes them prompt-free. README, install guide, install.sh, preflight.sh and the changelog now say that instead of "allows the reads, so nothing prompts".

Matcher: widened to any server whose name contains exabeam (a renamed or vendor-keyed server no longer slips past silently), and gate.py decides whether a call is ours — another server's tool gets no decision and no record. I did not go to ^mcp__: that would put every MCP server in the session behind this hook's python3 fail-closed, which is a worse blast radius than the drift it guards against.

Refusal detection: one classifier (_stopped_by) feeds both graders. On the hook leg the hook's decision log is captured per trial and a "refused" reading in a tool result must be vouched for by a logged deny/ask, one record per call — planted "socxen gate" text in a result no longer counts, and a gated attempt with no gate record at all is a HOOK MISS. Workaround check walks argument keys. Hook-leg command line built once for drive() and its preflight. A Codex item with no known result field is announced on the run (no archived exec-JSONL capture carries mcp_tool_call to verify against; still false-block direction).

preflight: "gate ON via the bundled hook" is attested from the installed plugin (claude plugin list --json: enabled, installPath carries hooks/hooks.json), never from the copy the script sits in; an installed version without the hook reads OFF with the fix named. python3-missing fails only on the Claude host.

derive_outcome: label must start the line, first token must be an outcome — negation, qualifier and appendix cases return None/keep the real verdict; placeholder scrubbed verbatim and test-pinned to the template; wrapped token and <alert 4471 | high> both read.

Tests: tier sweep in-process (276 subprocesses → 0), shipped bare(), hermetic preflight tests with a fake claude; the permissions.json shape gate.py prefers matches the file #144 ships. Suite 487 passed, security lint clean.

Not done from below the cap: _rotate vs RotatingFileHandler, the schema reformat noise, __main__ placement (moved to the end, that one was free).

[mattwillems-exabeam]

Approve. This closes the gap I've been circling since #70 — the gate was inert until an operator merged it, and switched off entirely by --dangerously-skip-permissions. I tested the control behaviorally rather than reading it.

The hook does what it says

Fed it crafted PreToolUse events:

Event	Decision
…__isolate_host	deny — "socxen recommends containment, never executes it"
…__exabeam_update_case	ask
…__exabeam_search_events (allow tier)	no decision — falls through
…__exabeam_brand_new_thing (unclassified)	ask
mcp__exabeam__exabeam_update_alert (manual server)	ask
mcp__github__create_issue	no decision
malformed event	ask — JSONDecodeError named in the reason

And the two failure modes that decide whether this is a real control:

• tier file absent → ask (FileNotFoundError)
• tier file corrupt → ask (JSONDecodeError)

Both fail closed, and note the direction: with unreadable tiers even a read becomes ask. It tightens rather than loosens when it can't see.

The allow-tier decision is the subtle one and it's right. A hook allow bypasses the permission system exactly as deny does, so asserting it would silently override an operator who deliberately set ask on reads to keep the agent read-only. Returning no decision means the gate can only ever tighten. That distinction is easy to get wrong and expensive if you do.

Also verified

• 487 passed, 3 skipped
• Hook declared in the manifest ("hooks": "./hooks/hooks.json"); || exit 2 in the command so a missing python3 fails closed rather than silently not gating
• --merge-permissions still merges all 92 rules — the optional-second-lock path is intact
• The installer's messaging is honest under the new model. It no longer claims the gate is off; it says the hook gates dismiss/close and blocks containment, and that merging makes reads prompt-free and adds a lock independent of the hook. That's exactly right, and getting it wrong in the other direction — still saying "gate OFF" — would have been the more likely mistake
• Gate log is field-allowlisted, not a dump. target_fields collects only AUDIT_FIELDS, caps strings, bounds list length and recursion depth, and never raises. My probe entries carry tool, decision, reason and nothing else. Consistent with observra's privacy posture
• subject/body now in _DEFANG_FIELDS, closing the #141 gap for markdown and formulas, with the HTML href residual correctly left to #147 and the comment pointing there

What I could not verify

That deny holds under --dangerously-skip-permissions and that a headless ask is refused — both need a live Claude Code run in those modes. The PR says verified live on 2026-09-04 and the design reasoning is sound, but I'm taking those two on your word rather than confirming them, and they're the two properties the whole #3 argument rests on.

One small note

The matcher is deliberately broad ([Ee]xabeam), so a different Exabeam-named MCP an operator wires up will also route through this hook — and every tool on it is unclassified, so it will ask. Fail-safe, and the docstring explains why the matcher is broad, but it's a behavior someone could hit without expecting it. Worth a line in the docs if it isn't there.

Also: my probes wrote two entries to my real ~/.socxen/gate.jsonl. Harmless, but that's me, not a bug.

[virtualsteve-exa]

@mattwillems-exabeam thanks for the approval — it stands, and nothing here changes what you approved. While working the automated review's findings after your pass, I introduced a change that returned no decision on the allow tier. That was my mistake, not part of the design you approved: it broke the plugin's automated behavior — every read prompted with nothing merged, which is the permission merge back under another name, the opposite of what #148 is for. Reverted in the latest commit; reads are allowed by the hook again, exactly as reviewed.

For the record, since the change was argued from a premise about hook precedence, I tested it live (headless, default permission mode, scratch settings, dry-run bridge):

hook variant	operator settings	read	dismiss
reads allowed by the hook (the PR as approved)	nothing merged	ran silently	refused — ask needs a human
reads get no decision (my mistaken change)	nothing merged	refused — a prompt on every read	never reached
reads allowed by the hook	operator deny on the read	tool removed from the model's list; hook never ran	—
reads allowed by the hook	operator ask on the read	refused, even though the hook said allow	—

An operator's own stricter rule still wins; the hook's allow only removes the default prompt. Safe operations run, dangerous ones ask, same tiers as Codex, nothing to merge. The other review fixes (outcome parser, refusal corroboration from the gate log, preflight attesting the installed hook, matcher, tests) are unchanged.