Harden secrets, metrics, and reporting coverage

.github/workflows/ci-matrix.yml

@@ -0,0 +1,64 @@
+name: ci-matrix
+
+on:
+  pull_request:
+  push:
+    branches: ["main", "master"]
+  workflow_dispatch:
+
+env:
+  PIP_DISABLE_PIP_VERSION_CHECK: "1"
+  PYTHON_KEYRING_BACKEND: "keyring.backends.null.Keyring"
+
+jobs:
+  build-test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.9", "3.11"]
+    steps:
+      - uses: actions/checkout@v4
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: "pip"
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r requirements.txt
+          pip install -r requirements-dev.txt
+      - name: Lint
+        run: |
+          ruff check .
+          black --check .
+      - name: Run unit tests
+        run: pytest -m "not integration" --maxfail=1 --ff
+      - name: Upload coverage data
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: coverage-${{ matrix.python-version }}
+          path: ./.coverage*
+          if-no-files-found: ignore
+
+  integration:
+    runs-on: ubuntu-latest
+    needs: build-test
+    environment: integration
+    if: github.event_name == 'workflow_dispatch' && github.event.inputs.integration == 'true'
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+          cache: "pip"
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r requirements.txt
+          pip install -r requirements-dev.txt
+      - name: Run integration tests
+        env:
+          INTEGRATION_SCANME: ${{ secrets.INTEGRATION_SCANME || 'false' }}
+        run: pytest -m integration --maxfail=1 --ff

Dockerfile

@@ -7,6 +7,8 @@ ENV PYTHONDONTWRITEBYTECODE=1 \
 
 WORKDIR /app
 
+ARG INCLUDE_DEV_KEYS=false
+
 COPY requirements.txt ./
 
 RUN python -m venv /opt/venv \
@@ -21,6 +23,8 @@ ENV PYTHONDONTWRITEBYTECODE=1 \
 
 WORKDIR /app
 
+ARG INCLUDE_DEV_KEYS=false
+
 RUN apt-get update \
     && apt-get install --no-install-recommends -y \
         libcairo2 \
@@ -38,10 +42,14 @@ RUN apt-get update \
 COPY --from=builder /opt/venv /opt/venv
 COPY . .
 
-RUN chown -R reconscript:reconscript /app
+RUN if [ "$INCLUDE_DEV_KEYS" != "true" ]; then rm -f keys/dev_*; fi \
+    && chown -R reconscript:reconscript /app
 
 USER reconscript
 
 EXPOSE 5000
 
+HEALTHCHECK --interval=30s --timeout=5s --start-period=30s --retries=3 \
+    CMD python -c "import urllib.request; urllib.request.urlopen('http://127.0.0.1:5000/healthz', timeout=3)"
+
 CMD ["python", "start.py"]

README.md

@@ -39,10 +39,17 @@ python -m pip install -r requirements-dev.txt
 
 ## Quick Start
 ### Launch the Web UI
+Before starting the Flask UI, generate deployment-specific secrets and point the application at them:
+
 ```bash
+export FLASK_SECRET_KEY_FILE=/secure/path/flask_secret.key
+export ADMIN_USER=security-admin
+export ADMIN_PASSWORD='replace-with-strong-passphrase'
+export CONSENT_PUBLIC_KEY_PATH=/secure/path/consent_ed25519.pub
+export REPORT_SIGNING_KEY_PATH=/secure/path/report_ed25519.priv
 python start.py
 ```
-The launcher checks dependencies, loads environment variables from `.env` if present, and starts the Flask server on <http://127.0.0.1:5000>. Use `start.sh`, `start.bat`, or `start.ps1` for platform-specific wrappers.
+The launcher checks dependencies, loads environment variables from `.env` if present, and starts the Flask server on <http://127.0.0.1:5000>. Use `start.sh`, `start.bat`, or `start.ps1` for platform-specific wrappers. Set `ALLOW_DEV_SECRETS=true` only for local demos that intentionally reuse the sample keys in `keys/`.
 
 ### Run a CLI Scan
 ```bash
@@ -57,7 +64,10 @@ A Docker Compose definition is provided for isolated demonstrations:
 docker compose up --build
 ```
 
-Mount the `results/` directory when running containers so generated artefacts persist outside the container lifecycle.
+Mount the `results/` directory when running containers so generated artefacts persist outside the container lifecycle. Override the required secrets via environment variables or secrets managers at runtime; the container image omits the developer keys unless built with `--build-arg INCLUDE_DEV_KEYS=true`.
+
+### Observability
+ReconScript exposes Prometheus-compatible metrics at `/metrics` and a readiness probe at `/healthz`. Scrape the metrics endpoint to monitor scan durations, completion counts, and open-port histograms.
 
 ## Validation and Quality Checks
 The project includes automation scripts and workflows to keep contributions consistent:
@@ -71,7 +81,7 @@ pip-audit --requirement requirements.txt
 pytest
 ```
 
-The `.github/workflows/lint.yml` pipeline mirrors these steps and is configured to surface security findings without blocking the build unless a critical error occurs.
+Continuous integration is handled by `.github/workflows/ci-matrix.yml`, which caches Python dependencies, runs Ruff, Black, and pytest on Python 3.9 and 3.11, and uploads coverage artefacts for inspection.
 
 ## Troubleshooting
 - **Missing system packages:** PDF export requires additional system libraries; review `docs/HELP.md` before enabling that pathway.

ROADMAP.md

@@ -1,36 +1,17 @@
 # Roadmap
 
-## Planned Enhancements
-- [ ] Publish automated consent manifest testing guidance in `docs/` so operators can rehearse pre-flight reviews.
-- [ ] Extend the report pipeline with configurable scheduling to support recurring scans.
-- [ ] Introduce optional role-based access control for the web UI to align with enterprise access policies.
-- [ ] Design a REST API that mirrors CLI capabilities for remote orchestration.
+## 30-Day Objectives
+- [x] Enforce environment-provided secrets for the Flask UI and consent signing flow.
+- [x] Consolidate GitHub Actions into a cached matrix workflow covering Ruff, Black, and pytest.
+- [x] Repair Markdown exporter fallback logic and add regression coverage for CLI reporting.
+- [ ] Publish an onboarding checklist that walks operators through secret provisioning and CI expectations.
 
-## Deferred Work and Investigations
-- [ ] Evaluate integrating Shodan and Censys enrichment services without compromising the read-only posture.
-- [ ] Update Docker and base operating system images to the latest slim Python releases and refresh lockfiles afterwards.
-- [ ] Review validation of user-supplied targets throughout the `reconscript` package to ensure strict input handling before any production deployment.
-- [ ] Revisit helper scripts that invoke subprocesses to confirm arguments are sanitised and environment-aware.
+## 60-Day Objectives
+- [ ] Automate signing-key rotation with documentation for Vault/Secrets Manager integrations.
+- [ ] Expand Prometheus metrics to include per-stage durations and scrape examples for popular platforms.
+- [ ] Add integration tests that exercise PDF generation, metrics endpoint responses, and RBAC toggles.
 
-## Documentation and Operational Tasks
-- [ ] Capture refreshed UI screenshots for `docs/screenshots/` once the interface updates stabilise.
-- [ ] Align README, HELP, and CLI reference content whenever major features ship to prevent drift.
-- [ ] Add explicit environment variable tables to `docs/HELP.md` covering Docker, CLI, and web deployments.
-
-## Security and Compliance Notes
-- `bandit -r reconscript`: **Not executed** in this audit environment. Recommend running locally; prioritise findings involving input handling or unsafe subprocess usage.
-- `pip-audit --requirement requirements.txt`: **Not executed**. Review reported CVEs promptly and pin patched versions.
-- Docker images should include metadata labels (maintainer, version, description) and consider enabling a container health check in future iterations.
-- No hardcoded secrets were identified during the documentation review; continue relying on environment variables and manifest files for sensitive values.
-
-## Dependency and Compatibility Considerations
-- Runtime dependencies remain pinned in `requirements.txt` and `pyproject.toml`. Monitor Flask, Requests, and urllib3 for security updates; refresh pins quarterly.
-- Development dependencies now include `bandit`, `ruff`, and `pip-audit` to align with the CI workflow.
-- The project targets Python 3.9 through 3.13. Validate support for upcoming Python releases annually and update classifiers accordingly.
-
-## Audit Summary
-- Dependencies checked and aligned with pinned versions for repeatable installs.
-- Deprecated packaging fields replaced with modern SPDX-compatible settings per Python.org guidance for 2026.
-- Security tools recommended but not executed; see notes above for follow-up actions.
-- CI workflow updated to run linting, tests, and security audits for continuous review.
-- No functional code changes were introduced during this audit.
+## 90-Day Objectives
+- [ ] Build Grafana dashboards/alerts around the exposed Prometheus metrics and publish SLO targets.
+- [ ] Draft a security playbook covering credential rotation, incident response, and consent manifest audits.
+- [ ] Complete an accessibility audit of the UI templates, including keyboard-only navigation reviews and contrast testing.