Skip to content

docs: accuracy remediation — reconcile documentation with shipped code - #14

Merged
SALhik merged 20 commits into
mainfrom
docs/documentation-accuracy-remediation-2026-07-01
Jul 13, 2026
Merged

SALhik merged 20 commits into
mainfrom
docs/documentation-accuracy-remediation-2026-07-01

Conversation

@SALhik

@SALhik SALhik commented Jul 3, 2026 •

Copy link
Copy Markdown
Owner

Summary

Fixes documentation drift across docs/ so the reference and operations material describes the shipped code rather than an aspirational or older system. Executed in six reviewed waves from the 2026-07-01 accuracy audit and remediation plan.

What changed

  • Version: single-sourced at 3.4.0 (pentestgpt/_version.py), pyproject.toml synced, drift-guard test added.
  • Dev docs: CI/CD, CONTRIBUTING, CODE_QUALITY reconciled to the real 2-job pipeline (Poetry 2.2.1 / Python 3.12, real Makefile targets, no isort/phantom jobs).
  • Config: env-var reference matched to AppSettings (16 phantom vars removed, real fields added, DB default fixed).
  • API reference: regenerated from live OpenAPI via new scripts/dump_openapi.py; 204 real routes documented; phantom /auth/* and hyphenated paths removed; nonexistent SDK docs deleted.
  • Operations: DEPLOYMENT.md rewritten against the real runtime (no invented HTTP server or v1.0.0 build).
  • Guides: INTEGRATION and PLUGINS fixed to the real module layout (PluginInterface, communication hub, OrchestratorFactory).
  • Architecture: stealth/MITRE paths corrected, technique count 50+ to 38, network subsystem documented.
  • CLI: full 20-flag reference generated from pentestgpt --help.
  • Housekeeping: fixed index links, retired the stale roadmap, indexed the orphaned testing guide.
  • Guard: new scripts/check_docs.py (links, module paths, CLI flags, API paths) wired into quality.yml to fail CI on future drift. It already caught 4 real stale imports in INTEGRATION.md.
  • Dependency: declared the previously-undeclared python-nmap (imported by the network subsystem).

Verification

  • mypy . clean (615 files); ruff check . clean; pytest 5688 passed, 11 skipped
  • scripts/dump_openapi.py and scripts/check_docs.py run clean; pentestgpt --version prints 3.4.0; --help matches the documented 20 flags

Follow-ups (surfaced, not fixed here)

  • testing_api.app registers 0 routes (include_router precedes the route decorators)
  • monitoring API /api/v1/monitoring duplication (documented per decision; code untouched)
  • os.getenv in two config classes vs the AppSettings convention; *_BASE_URL are code-set, not env-overridable
  • cosmetic "v3.0" banners (deferred batch); two isort lines in CODE_QUALITY.md; a SECURITY.md CLI example mismatch

Note on commit count

origin/main is currently 3 commits behind local main, so this PR temporarily shows 3 pre-existing commits (c56a6f3, 9db0605, 079c7ef) on top of the 13 remediation commits. They drop off once main is synced.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Documentation

    • Refreshed setup, getting-started, architecture, API, configuration, CI/CD, and contributing guides to match current behavior and CLI options.
    • Updated stealth and MITRE ATT&CK documentation, clarified Stealth Level naming, and revised API route overviews; removed the SDK examples index content.
    • Replaced the roadmap with a shorter pointer-based version and expanded documentation entry points.
  • Chores

    • Updated the example environment configuration and tightened CI documentation validation with dedicated accuracy checks.
    • Added a local pre-check that blocks required gstack usage when the tool isn’t installed.

SALhik and others added 16 commits June 24, 2026 10:07
results_verifier.py: also catches FuturesTimeoutError so LLM call timeouts
from thread-pool executors propagate correctly.
tool_executor.py: also catches asyncio.TimeoutError alongside the builtin
so async command execution timeouts are handled uniformly.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Documents the gstack skills available to Claude Code agents on this project, including the /browse skill for web browsing and the full list of available skills.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adopt 3.4.0 as the one product version (matches CHANGELOG + git release):
- pentestgpt/_version.py is the source of truth (3.0.0 -> 3.4.0)
- pyproject.toml literal synced to 3.4.0; drop "v3.0" from the description
- new tests/test_version_consistency.py fails CI if the two ever drift
- docs/README.md and README.md H1 banners de-versioned

Wave 0 decisions (documentation-accuracy remediation):
- D1: adopt 3.4.0, sourced from _version.py (recommended)
- D2: leave subsystem version fields and v3.0 docstrings alone (recommended)
- D3: delete the nonexistent SDK docs (recommended)
- D4: rewrite operations/DEPLOYMENT.md from scratch (recommended)
- D5: document the monitoring API split; change neither module (no code change)
- D6: generate the API reference from a live OpenAPI dump script (recommended)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…SDK docs

Rewrites docs/api/README.md and docs/api/AGENT_API.md from the 11
generated OpenAPI schemas in docs/api/_generated/. Removes the
fictional SDK_EXAMPLES.md (no pentestgpt_sdk package exists).

Key corrections:
- Describes the real architecture: 11 standalone FastAPI apps/routers,
  no shared aggregator or /api/v1 mount point.
- Removes phantom auth endpoints; describes actual bearer-token middleware.
- Fixes detection resistance prefix to /detection_resistance/ (underscore).
- Fixes cost-optimization prefix to /api/ (not a doubled path).
- Documents the monitoring split: monitoring_api (19 paths, active app)
  vs monitoring_api_secure (6 paths, router not currently mounted).
- All route tables sourced directly from the _generated JSON files.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
pentestgpt/network/segmentation_traversal.py does `import nmap` at module
level, so pentestgpt.api.network_api cannot be imported without python-nmap.
It was previously undeclared in both requirements.txt and pyproject.toml.
Add it (0.7.1) so a clean install can import the network API and CI can
regenerate/verify the API docs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…untime

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace the example-only CLI coverage with a complete options table in the
getting-started guide, generated from `pentestgpt --help` (all 20 flags,
including --v2-mode/--v3-mode which the audit's grep missed). Add a --help
pointer to the top-level README. No fictional flags; matches --help exactly.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… subsystem)

- Replace every pentestgpt/stealth/ path with canonical sources:
  pentestgpt/detection_resistance/evasion_engine.py (canonical evasion
  engine) and pentestgpt/monitoring/detection_monitor.py; mark
  pentestgpt/utils/evasion_engine.py as a deprecated shim
- Fix all pentestgpt.stealth.* Python imports to canonical modules
- Repoint MITRE examples to canonical pentestgpt/mitre/ package
  (taxonomy.py, planner.py, kb.py); mark utils/mitre_*.py as deprecated
  re-export shims slated for v4.0.0 removal
- Remove nonexistent pentestgpt/utils/test_mitre.py reference
- Correct technique count from "50+" to 38 (confirmed by taxonomy log)
- Fix StealthLevel pseudocode: STEALTH -> MAXIMUM in all four files;
  remove phantom Minimal row from DETAILS.md stealth table
- Document pentestgpt/network/ subsystem (protocol_evasion,
  segmentation_traversal, traffic_analyzer, traffic_obfuscation_manager)
  in DETAILS.md
- Fix stray pentestgpt/utils/mitre_planner.py path in README.md

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- docs/README.md: remove the broken SDK_EXAMPLES link (file deleted in the
  API rewrite), drop the redundant CHANGELOG_V3 index line (it is a redirect
  stub, kept on disk as a tombstone), and add the orphaned Community Beta
  Testing Guide under Getting Started.
- docs/ROADMAP.md: replace the year-stale v3.0-v3.2 roadmap with a short
  pointer to the changelog and GitHub milestones/issues.
- Removed the empty docs/superpowers/ directory (untracked .DS_Store only).

Index now: 33 links all resolve; the only unreferenced doc is the intentional
CHANGELOG_V3 tombstone.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Introduces scripts/check_docs.py with four deterministic, offline checks:
1. Internal links in docs/ and README.md resolve to existing files.
2. pentestgpt.* imports in fenced code blocks (non-archive) map to a real
   .py or __init__.py on disk.
3. --flags in getting-started/README.md and README.md are a subset of
   `pentestgpt --help`.
4. HTTP-method+path pairs in docs/api/*.md exist in docs/api/_generated/*.json
   (skipped when no JSON files are present).

Replaces the placeholder "Validate API documentation" step in quality.yml with
a real check that fails CI on drift.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The new doc-lint guard flagged four stale imports the Phase 5 grep-based
verification missed (it searched slash-form paths, not dotted imports in
fenced blocks):
- api.auth_middleware.get_current_user -> auth is middleware-based
  (AuthenticationMiddleware); routes do not declare a per-route dependency
- database.repositories.session_repository.SessionRepository -> no such
  module/class; models are per-domain *_models.py, sessions via SessionManager
- monitoring.alert_correlation_engine -> monitoring.alert_correlation
- cost_optimization.budget_manager.BudgetManager ->
  intelligent_budget_manager.IntelligentBudgetManager

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jul 3, 2026 •

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@SALhik, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 54 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 83e68fa6-cb76-478c-a169-ffa2018fb895

📥 Commits

Reviewing files that changed from the base of the PR and between fd69ace and 55133ab.

⛔ Files ignored due to path filters (1)
  • poetry.lock is excluded by !**/*.lock
📒 Files selected for processing (2)
  • CLAUDE.md
  • config/stealth_default.json
📝 Walkthrough

Walkthrough

This PR adds Claude Code gstack enforcement, updates CI documentation validation and environment examples, and revises project, API, architecture, configuration, development, and getting-started documentation for current module paths, stealth terminology, workflows, and CLI usage.

Changes

Claude tooling and configuration

Layer / File(s) Summary
gstack hook and settings
.claude/hooks/check-gstack.sh, .claude/settings.json, CLAUDE.md
Adds a Skill pre-tool hook that checks for gstack and documents the required browsing workflow.
Documentation validation
.github/workflows/quality.yml
Runs scripts/check_docs.py with fixed test environment variables instead of checking only for docs/api/.
Environment example cleanup
.env.example
Removes obsolete provider, storage, security, rate-limit, upload, and payload settings and adds OAuth2 configuration.

Documentation overhaul

Layer / File(s) Summary
Project entry points and index
README.md, docs/README.md
Updates project naming, Quick Start guidance, documentation headings, and index links.
Roadmap and API documentation
docs/ROADMAP.md, docs/api/AGENT_API.md, docs/api/SDK_EXAMPLES.md
Condenses roadmap and API documentation around current references and generated schemas; removes SDK examples.
Architecture and MITRE documentation
docs/architecture/DETAILS.md, docs/architecture/DIAGRAMS.md, docs/architecture/MITRE_INTEGRATION.md, docs/architecture/README.md
Documents canonical MITRE and network packages, updates architecture material, and changes MITRE examples to current import paths.
Stealth implementation documentation
docs/architecture/STEALTH.md, docs/architecture/STEALTH_IMPLEMENTATION.md
Renames the maximum stealth enum member, updates canonical module paths, and revises related examples, tests, and deployment snippets.
Configuration references
docs/configuration/ENVIRONMENT_VARIABLES.md, docs/configuration/REFERENCE.md
Aligns environment and configuration documentation with current loading, provider, OAuth2, rate-limiting, feature-flag, and stealth settings.
Development workflow documentation
docs/development/CI_CD.md, docs/development/CODE_QUALITY.md, docs/development/CONTRIBUTING.md
Updates workflow, quality, setup, Ruff, Bandit, Makefile, and Claude workflow guidance.
Getting-started CLI guidance
docs/getting-started/README.md
Adds command-line options and reformats configuration preset documentation.

Estimated code review effort: 3 (Moderate) | ~30 minutes

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the PR’s main goal: fixing documentation to match the shipped code.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch docs/documentation-accuracy-remediation-2026-07-01

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)
docs/development/CONTRIBUTING.md (1)

265-267: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Replace the stale isort checklist item.

The surrounding guidance now standardizes on Ruff for import sorting, so the checklist should not point contributors back to isort.

♻️ Proposed fix
- [ ] Imports are properly sorted (isort)
+ [ ] Imports are properly sorted (Ruff)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/development/CONTRIBUTING.md` around lines 265 - 267, The checklist item
in CONTRIBUTING.md still references the old isort workflow, but the
import-sorting guidance should now align with Ruff. Update the checklist entry
near the import-ordering guidance to remove the isort mention and replace it
with the Ruff-based import sorting/check item, keeping the surrounding
contributor instructions consistent with the current tooling.
docs/architecture/STEALTH.md (1)

2495-2508: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Keep stealth_level as StealthLevel here. _increase_stealth assigns the raw string "maximum", but this section treats stealth_level as a StealthLevel enum everywhere else. That leaves current_level.value comparisons brittle and can break the next transition.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/architecture/STEALTH.md` around lines 2495 - 2508, Keep
component.stealth_level as a StealthLevel enum in _increase_stealth and
_reduce_stealth, since the current code assigns a raw string while later logic
relies on current_level.value comparisons. Update the assignments in these
methods to use the enum members consistently, and ensure all transition checks
in _increase_stealth and _reduce_stealth compare against StealthLevel values
instead of string literals.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.env.example:
- Around line 55-58: The OAuth2 environment variables in the .env.example block
are out of the order expected by dotenv-linter. Reorder the entries under the
OAuth2 section so that OAUTH2_CLIENT_ID and OAUTH2_CLIENT_SECRET appear before
OAUTH2_ENABLED, keeping the rest of the section unchanged.

In `@docs/api/AGENT_API.md`:
- Around line 14-15: The API table descriptions for the `/agents/start` and
`/agents/stop` rows are too broad; update them to describe actions on a single
agent rather than the whole system. Use the existing agent handler semantics
from the `agent_api.py` endpoints that accept an `agent_id`, and revise the
wording in the `AGENT_API.md` table so the `start` and `stop` entries clearly
reflect per-agent behavior.

In `@docs/architecture/DIAGRAMS.md`:
- Around line 361-364: The flow label for the highest stealth branch is
misaligned with the `StealthLevel.MAXIMUM` preset. Update the diagram in the
branch handling `B -->|Maximum 10| ...` so it routes to `E[Maximum Stealth]`
instead of `F[Research Mode]`, keeping the `Aggressive Stealth`, `Balanced
Stealth`, and `Maximum Stealth` labels consistent with the existing `B` decision
mapping.

In `@docs/architecture/MITRE_INTEGRATION.md`:
- Around line 23-26: The deprecation note is too broad and should apply only to
the MITRE re-export shims, not the entire utils area. Update the
MITRE_INTEGRATION documentation around the utils section to clearly mark only
mitre_attack.py, mitre_planner.py, and mitre_kb.py as deprecated shim files,
while leaving the rest of pentestgpt/utils as active modules. Keep the wording
tied to those three filenames and their “re-export shim” role so the deprecation
scope is unambiguous.

In `@docs/architecture/README.md`:
- Line 9: The version displayed in the architecture README is outdated and
should be brought in sync with the current project release. Update the Current
Version text in the README content that includes the AutoPentester paper
reference so it shows 3.4.0 instead of 3.0.0, keeping the rest of the line
unchanged.

---

Outside diff comments:
In `@docs/architecture/STEALTH.md`:
- Around line 2495-2508: Keep component.stealth_level as a StealthLevel enum in
_increase_stealth and _reduce_stealth, since the current code assigns a raw
string while later logic relies on current_level.value comparisons. Update the
assignments in these methods to use the enum members consistently, and ensure
all transition checks in _increase_stealth and _reduce_stealth compare against
StealthLevel values instead of string literals.

In `@docs/development/CONTRIBUTING.md`:
- Around line 265-267: The checklist item in CONTRIBUTING.md still references
the old isort workflow, but the import-sorting guidance should now align with
Ruff. Update the checklist entry near the import-ordering guidance to remove the
isort mention and replace it with the Ruff-based import sorting/check item,
keeping the surrounding contributor instructions consistent with the current
tooling.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 03c17402-cc8e-47a2-8853-9cedf48e7a76

📥 Commits

Reviewing files that changed from the base of the PR and between c5aeb47 and 3781f81.

⛔ Files ignored due to path filters (11)
  • docs/api/_generated/agent_api.json is excluded by !**/_generated/**
  • docs/api/_generated/cost_optimization_api.json is excluded by !**/_generated/**
  • docs/api/_generated/detection_resistance_api.json is excluded by !**/_generated/**
  • docs/api/_generated/effectiveness_api.json is excluded by !**/_generated/**
  • docs/api/_generated/knowledge_api.json is excluded by !**/_generated/**
  • docs/api/_generated/monitoring_api.json is excluded by !**/_generated/**
  • docs/api/_generated/monitoring_api_secure.json is excluded by !**/_generated/**
  • docs/api/_generated/network_api.json is excluded by !**/_generated/**
  • docs/api/_generated/plugin_api.json is excluded by !**/_generated/**
  • docs/api/_generated/stealth_api.json is excluded by !**/_generated/**
  • docs/api/_generated/testing_api.json is excluded by !**/_generated/**
📒 Files selected for processing (35)
  • .claude/hooks/check-gstack.sh
  • .claude/settings.json
  • .env.example
  • .github/workflows/quality.yml
  • CLAUDE.md
  • README.md
  • docs/README.md
  • docs/ROADMAP.md
  • docs/api/AGENT_API.md
  • docs/api/README.md
  • docs/api/SDK_EXAMPLES.md
  • docs/architecture/DETAILS.md
  • docs/architecture/DIAGRAMS.md
  • docs/architecture/MITRE_INTEGRATION.md
  • docs/architecture/README.md
  • docs/architecture/STEALTH.md
  • docs/architecture/STEALTH_IMPLEMENTATION.md
  • docs/configuration/ENVIRONMENT_VARIABLES.md
  • docs/configuration/REFERENCE.md
  • docs/development/CI_CD.md
  • docs/development/CODE_QUALITY.md
  • docs/development/CONTRIBUTING.md
  • docs/getting-started/README.md
  • docs/guides/INTEGRATION.md
  • docs/guides/PLUGINS.md
  • docs/operations/DEPLOYMENT.md
  • docs/operations/MONITORING.md
  • pentestgpt/_version.py
  • pentestgpt/agents/results_verifier.py
  • pentestgpt/agents/tool_executor.py
  • pyproject.toml
  • requirements.txt
  • scripts/check_docs.py
  • scripts/dump_openapi.py
  • tests/test_version_consistency.py
💤 Files with no reviewable changes (1)
  • docs/api/SDK_EXAMPLES.md

Comment thread .env.example Resolved
Comment thread docs/api/AGENT_API.md Resolved
Comment thread docs/architecture/DIAGRAMS.md Resolved
Comment thread docs/architecture/MITRE_INTEGRATION.md Resolved
Comment thread docs/architecture/README.md Resolved
SALhik added 4 commits July 13, 2026 14:06
- .env.example: reorder OAuth2 keys so CLIENT_ID/CLIENT_SECRET precede
  ENABLED (dotenv-linter ordering)
- AGENT_API.md: /agents/start and /agents/stop act on a single agent
  (agent_id in body), not the whole system
- DIAGRAMS.md: route Maximum 10 to Maximum Stealth instead of Research
  Mode, which select_stealth_level() only reaches via a separate
  research_mode override, never via security maturity; drop the
  resulting orphaned Research Mode node/edge
- MITRE_INTEGRATION.md: scope the deprecation note to the 3 named
  shim files, not all of pentestgpt/utils/
- architecture/README.md: version 3.0.0 -> 3.4.0, matching _version.py
- STEALTH.md: adaptation controller example assigns StealthLevel enum
  members instead of raw strings, matching the enum's usage elsewhere
  in the doc
- CODE_QUALITY.md: checklist item isort -> Ruff (CodeRabbit cited
  CONTRIBUTING.md, but that file has no isort references; the stale
  line was here)
@SALhik
SALhik merged commit d203db5 into main Jul 13, 2026
2 of 5 checks passed
@SALhik
SALhik deleted the docs/documentation-accuracy-remediation-2026-07-01 branch July 13, 2026 08:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant