Skip to content

docs: bring the docs current with the plugin/ restructure and the unreleased changes - #80

Merged
virtualsteve-exa merged 3 commits into
devfrom
docs/pre-promotion-currency
Aug 13, 2026
Merged

virtualsteve-exa merged 3 commits into
devfrom
docs/pre-promotion-currency

Conversation

@virtualsteve-exa

Copy link
Copy Markdown
Collaborator

Pre-promotion currency pass, ahead of the dev → main release. Everything here was found by checking docs against the code, not against other docs.

Broken instructions (post-#66)

A doc that contradicted shipped code

logging.md claimed that if "an event fails to emit, telemetry disables itself." The code does the opposite deliberately: the event is dropped and the trail stays on, because a transient per-event fault must not silently switch off a mandatory audit log — test_single_emit_failure_does_not_disable_the_whole_trail exists to enforce exactly that. Only a configuration-level failure disables. The two cases are now documented separately. Also added the observra>=1.1 floor (the rotation passthrough and emit() both depend on it) and the new stderr warning channel from #78 — neither appeared anywhere.

[Unreleased] covered 1 of 4 landed changes

Added entries for the plugin/ restructure (#66), the ABV release gate (#77), and dependency bounds + hash-pinned lockfile / observra 1.1 (#78); git show --stat confirms neither #77 nor #78 touched the changelog. Fixed the pre-restructure paths inside the existing entries and the file header. Historical entries for 0.6.0–0.6.9 keep their then-current paths, untouched.

The lockfile nobody mentioned

plugin/connector/exabeam-mcp-bridge.py.lock ships 33 hash-pinned packages and a grep across every .md/.py/.sh/.yml found zero references. CONTRIBUTING now tells contributors to re-lock in the same PR (CI's --check guards the AI BOM but not lock drift), and tests/end-to-end-testing.md copies the .lock into the cache — without it a maintainer testing a dependency change resolves a different tree than users get.

Smaller items

  • SECURITY.md scope adds merge_permissions.py (the only code that writes to a user's settings file) and the lockfile.
  • plugin/README.md still framed the installer as check-only; the warning box now shows --merge-permissions with its consent semantics.
  • CONTRIBUTING said the assurance gates run "before a tag" twice, though the repo has zero tags and the same file says so elsewhere; also unified python3 vs uv run for bump_version.py.
  • security/praxen/SCAN_INSTRUCTIONS.md — worth a look, since it's forward-looking scope for the next Praxen audit rather than a dated record. It carried pre-restructure paths and an enforcement question premised on a manual-merge-only world, which would have biased that scan's result.
  • Root README.md's security/ row omitted the ABV program; security/praxen/README.md's note on finding 001 cited a line number that had drifted, and is now marked largely addressed by feat: install.sh --merge-permissions — prompted, backed-up merge of the governance gate (#70) #73.

Verification

263 tests pass · AI BOM current · installer smoke still merges to {allow: 18, ask: 4, deny: 68} · --help renders correctly · every relative markdown link in the touched files resolves.

Not included: the version bump. Worth noting for the release — the version still reads 0.6.9 with four merges on top, and by this project's own precedent (the 0.6.6 entry: "a plugin update is only delivered when the version moves") the promotion needs bump_version.py or the update reaches nobody.

🤖 Generated with Claude Code

…eleased changes

Pre-promotion currency pass. Three classes of defect, all found by reviewing
docs against the code rather than against each other.

BROKEN INSTRUCTIONS (post-#66, a user follows these):
- plugin/docs/installation.md told users to run ./install.sh --merge-permissions
  in the section the page itself calls the most important on it. There is no root
  install.sh any more; twenty lines earlier the same page correctly says
  ./plugin/install.sh. Same for merge_permissions.py's path.
- install.sh's OWN output was stale: the usage block is printed verbatim as
  --help, and every line said ./install.sh. Its 'Next steps' block named
  skills/soc-investigate/settings.snippet.json and ./install.sh. Both now print
  the paths the script has already resolved, so they are right from a clone and
  from the installed plugin directory alike. The --help text also still claimed
  the merge targets ~/.claude/settings.json — the exact hardcoding #73 removed.

A DOC THAT CONTRADICTED THE CODE:
- logging.md said a failed emit makes telemetry 'disable itself'. It does the
  opposite on purpose: the event is dropped and the trail stays on, because a
  transient fault must not switch off a mandatory audit log — there is a test
  named for that behavior. Split the two failure modes, and documented the
  observra>=1.1 floor and the new stderr warning channel, neither of which
  appeared anywhere.

MISSING RELEASE NOTES:
- [Unreleased] documented 1 of 4 landed changes. Added entries for the plugin/
  restructure (#66), the ABV release gate (#77), and the dependency bounds +
  hash-pinned lockfile and observra 1.1 migration (#78), and corrected the
  pre-restructure paths in the entries that were already there.

Also: the lockfile existed with no mention in any document — CONTRIBUTING now
tells contributors to re-lock (CI checks the AI BOM but not lock drift) and the
end-to-end recipe copies the .lock; SECURITY.md's scope adds merge_permissions.py
(the only code that writes to a user's settings) and the lockfile; the front-door
README no longer presents the installer as check-only; CONTRIBUTING said gates run
'before a tag' though the repo has no tags and says so twice elsewhere; and
SCAN_INSTRUCTIONS.md — forward-looking scope for the NEXT Praxen audit — carried
pre-restructure paths and asked about a manual-merge-only world that no longer
exists, which would have biased that scan.

Verified: 263 tests pass, AI BOM current, installer smoke still merges to
{allow: 18, ask: 4, deny: 68}, --help renders, every relative link resolves.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Steve Wilson <steve.wilson@exabeam.com>
1. A FALSE ASSURANCE IN AN AUDIT DOC. The rewritten fail-open section claimed
   observra's warnings surface on stderr for 'a full queue' so a dropped event is
   'always visible'. Traced through observra 1.1: a queue-overflow drop is logged
   at DEBUG and counted in observra_events_dropped_total, below the WARNING-level
   stderr handler the shim installs — so that drop is counted, not announced.
   Backend write errors DO surface. Corrected in logging.md, the CHANGELOG entry,
   and the shim's own (pre-existing, equally wrong) inline comment.

2. THE REVERSE PATH ERROR. installation.md ships INSIDE the plugin and the primary
   install path is the marketplace, where no plugin/ directory exists — so
   'skills/soc-investigate/...' was correct plugin-relative and my prefixing it
   broke it for the majority reader, while plugin/README.md kept the relative form.
   Now names the plugin-relative path with the clone path as a gloss.

3. A STALE PATH IN THE BLOCK THIS PR EDITED. tests/end-to-end-testing.md's CACHE
   still used the pre-0.6.5 marketplace name (socxen/socxen); the real cache is
   under open-agent-ai-security/socxen/. The new .lock copy would have missed.

Also: the shim's DESIGN RULES said an emit failure disables logging, contradicting
its own code, tests, and this PR's new doc; my bump_version change contradicted the
script's own docstring, so both call sites now use uv run; praxen/README.md's gate
said 'before tagging' while CONTRIBUTING (which links it) says release; the rotation
bounds are clamped, not passed straight through; the merge-permissions hint now
prints an absolute path instead of $0 as invoked; and (#70) gained its PR number
per the file's (#issue, #PR) convention.

Verified: 263 pass, BOM current, Next-steps block renders with paste-able absolute
paths, real settings untouched, all relative links resolve.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Steve Wilson <steve.wilson@exabeam.com>
@virtualsteve-exa

Copy link
Copy Markdown
Collaborator Author

Review pass — three real defects found and fixed (609650b)

Ran an adversarial review agent against this PR, specifically pointed at the claims most likely to be wrong. It re-verified the numeric and behavioral claims independently (the observra>=1.1,<2 pin, all five bounded deps, 33 lockfile packages, plugin/install.sh:534 after this PR's header growth, and every Praxen figure against results/2026-08-12-socxen-0.6.9.json) — those hold. Three things did not:

1. A false assurance in an audit doc. My rewritten fail-open section said observra's warnings surface on stderr for "a full queue," so a dropped event is "always visible." Traced through observra 1.1, that's wrong in the half that matters most: a queue-overflow drop is logged at DEBUG and counted in observra_events_dropped_total, below the WARNING-level stderr handler the shim installs. Backend write errors do surface. Queue overflow is the likeliest silent-drop mode, so this was an audit-trail doc promising visibility it doesn't have. Corrected in logging.md, the CHANGELOG entry, and the shim's own inline comment — which was already making the same wrong claim before this PR.

2. The reverse path error — the one I explicitly asked the reviewer to hunt for. installation.md ships inside the plugin, and the primary install path is the marketplace, where no plugin/ directory exists. skills/soc-investigate/settings.snippet.json was correct plugin-relative, and prefixing it broke the page for the majority reader — while plugin/README.md, edited in this same PR, correctly kept the relative form, so the shipped payload had become internally inconsistent. Now names the plugin-relative path with the clone path as a gloss.

3. A stale path inside the block this PR edited. tests/end-to-end-testing.md's CACHE still used the pre-0.6.5 marketplace name (socxen/socxen/); the real cache is under open-agent-ai-security/socxen/, verified against this machine. Adding the .lock copy to a recipe whose $CACHE doesn't resolve would have been a fix that still didn't work.

Also folded in: the shim's DESIGN RULES docstring claimed an emit failure disables logging — contradicting its own code, its tests, and this PR's new doc; my bump_version change contradicted the script's own docstring, so both call sites now use uv run; praxen/README.md's gate said "before tagging" while the CONTRIBUTING section linking to it says release; rotation bounds are clamped, not passed straight through; the merge hint prints an absolute path rather than $0 as invoked; and (#70) gained its PR number per the file's (#issue, #PR) convention.

Deliberately not changed: WORKER_REMIT.md's citations and praxen/README.md's blind-authoring narrative. Those are version-stamped evidence of what the docs said when the remit was written — rewriting line numbers inside a dated standard misrepresents it. If the security team wants them refreshed, that's a remit version bump, their call. security/redteam/HISTORY.md also has root-relative connector paths; it's being touched on the #76 branch, so it gets fixed there to avoid a conflict.

Verification after the fixes: 263 tests pass, AI BOM current, the Next-steps block renders with paste-able absolute paths, the real ~/.claude/settings.json is untouched by the smoke, and every relative link resolves. Both checks green on 609650b.

One item the reviewer flagged that isn't fixable here: the CHANGELOG's git-subdir claim depends on the external marketplace index. It's true only if that index flips in lockstep at release — worth holding the two together at promotion time.

The last clear item from the review pass. The section order was Changed ->
Fixed -> Added -> Security; both Keep a Changelog and this file's own 0.6.3
entry put Added first. Pure reordering — no wording changes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Steve Wilson <steve.wilson@exabeam.com>
@virtualsteve-exa
virtualsteve-exa merged commit b685e5d into dev Aug 13, 2026
2 checks passed
@virtualsteve-exa
virtualsteve-exa deleted the docs/pre-promotion-currency branch August 13, 2026 20:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant