Skip to content

ci: improve evidence scan diagnostics and action pinning - #1589

Merged
Teingi merged 5 commits into
masterfrom
codex/ci-diagnostics
Sep 15, 2026
Merged

ci: improve evidence scan diagnostics and action pinning#1589
Teingi merged 5 commits into
masterfrom
codex/ci-diagnostics

Conversation

@AlexStocks

@AlexStocks AlexStocks commented Sep 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Related to #1587.

Rationale for this change

The CI hardening in #1587 made replay evidence fail-closed, but the scanner step discarded all output and treated TruffleHog findings and scanner infrastructure failures identically. The repository also had no automated guard or update configuration for its pinned GitHub Actions.

What changes are included in this PR?

  • Keep TruffleHog JSON and stderr in runner temporary storage, derive bounded sanitized diagnostics, and upload only summary.txt when the scenario or scan fails.
  • Classify exit code 183 as findings; classify other non-zero results as infra_error after one retry.
  • Upgrade the pinned TruffleHog image to the verified v3.97.4 digest.
  • Add a repository checker that rejects mutable third-party Action references and run it from make check.
  • Add weekly Dependabot updates for GitHub Actions.
  • Add contract and fake-Docker tests for pinning, scan classification, retry behavior, and redaction.

Are there any user-facing changes?

No product or API changes. CI failures now include actionable sanitized diagnostics while raw scanner output remains excluded from artifacts.

How was this change tested?

  • uv run python -m pytest tests/test_workflow_actions.py tests/test_e2e_workflow_contract.py tests/test_scan_e2e_evidence.py tests/test_api_contract.py tests/test_js_operations.py -q (57 passed)
  • uv run ruff check and uv run ruff format --check on changed Python files
  • Actions pin checker: 67 references checked
  • Bash syntax check, YAML parsing, and git diff --check
  • make contract-test (48 passed)
  • make check reaches the existing ty gate but remains blocked by pre-existing type errors in src/powercontext/builtin/runtime/artifact_processing.py and tests/builtin/runtime/test_artifact_processing.py.

AI usage statement

Implemented with Codex using repository-local tests and review evidence.

AlexStocks and others added 4 commits September 14, 2026 11:46
@AlexStocks OmX
ci: improve evidence scan diagnostics and action pinning
f7ab0e2 
Lore: Make replay-scan failures distinguish secret findings from infrastructure errors while keeping raw scanner output out of uploaded artifacts.

Constraint: Preserve the network-disabled, read-only evidence scan and enforce immutable third-party Action references across workflow and composite-action files.

Tested: 57 focused and contract tests passed; action pin scan, Ruff, shell syntax, YAML parsing, and diff checks passed.

Not-tested: Full make check remains blocked by pre-existing ty errors in artifact_processing.py and test_artifact_processing.py.

Co-authored-by: OmX <omx@oh-my-codex.dev>
@AlexStocks OmX
ci: harden diagnostic redaction and YAML action checks
5aef900 
Lore: Harden uploaded E2E diagnostics against arbitrary scanner text and parse YAML action references semantically.

Constraint: Keep raw scanner JSON and stderr in runner-local files while exposing only bounded, fixed diagnostics and requiring 40-character third-party action pins.

Tested: uv run --locked python -m pytest tests/test_scan_e2e_evidence.py tests/test_workflow_actions.py tests/test_e2e_workflow_contract.py; uv run --locked ruff check; uv run --locked ruff format --check; make contract-test; bash -n scripts/scan_e2e_evidence.sh; uv run --locked python scripts/check_workflow_actions.py .github/workflows .github/actions; git diff --check.

Not-tested: make workflow-actions-check reaches the repository's pre-existing ty baseline errors in artifact_processing.py and its tests.

Co-authored-by: OmX <omx@oh-my-codex.dev>
@AlexStocks OmX
ci: harden scanner inputs and YAML validation
5c250c6 
Lore: Keep diagnostic headers and scanner inputs bounded, and make YAML action validation deterministic under malformed or aliased documents.

Constraint: Raw scanner output stays runner-local; uploaded diagnostics contain only fixed fields, immutable image references, and allowlisted findings.

Tested: uv run --locked python -m pytest tests/test_scan_e2e_evidence.py tests/test_workflow_actions.py tests/test_e2e_workflow_contract.py; uv run --locked ruff check scripts/check_workflow_actions.py tests/test_scan_e2e_evidence.py tests/test_workflow_actions.py; uv run --locked ruff format --check scripts/check_workflow_actions.py tests/test_scan_e2e_evidence.py tests/test_workflow_actions.py; bash -n scripts/scan_e2e_evidence.sh; uv run --locked python scripts/check_workflow_actions.py .github/workflows .github/actions; make contract-test; git diff --check.

Not-tested: make workflow-actions-check remains blocked only by the repository's pre-existing ty diagnostics in artifact_processing.py and its tests.

Co-authored-by: OmX <omx@oh-my-codex.dev>
@AlexStocks OmX
ci: harden scanner inputs and YAML validation
fc606e5 
Lore: Keep diagnostic headers and scanner inputs bounded, and make YAML action validation deterministic under malformed or aliased documents.

Constraint: Raw scanner output stays runner-local; uploaded diagnostics contain only fixed fields, immutable image references, and allowlisted findings.

Tested: uv run --locked python -m pytest tests/test_scan_e2e_evidence.py tests/test_workflow_actions.py tests/test_e2e_workflow_contract.py; uv run --locked ruff check scripts/check_workflow_actions.py tests/test_scan_e2e_evidence.py tests/test_workflow_actions.py; uv run --locked ruff format --check scripts/check_workflow_actions.py tests/test_scan_e2e_evidence.py tests/test_workflow_actions.py; bash -n scripts/scan_e2e_evidence.sh; uv run --locked python scripts/check_workflow_actions.py .github/workflows .github/actions; make contract-test; git diff --check.

Not-tested: make check remains blocked only by the repository's pre-existing ty diagnostics in artifact_processing.py and its tests; the new checker warning was fixed by explicit BaseResolver import.

Co-authored-by: OmX <omx@oh-my-codex.dev>
@AlexStocks
AlexStocks requested a review from PsiACE September 15, 2026 03:06
@AlexStocks

AlexStocks commented Sep 15, 2026

Copy link
Copy Markdown
Contributor Author

@Teingi @PsiACE please review this PR.

@AlexStocks

AlexStocks commented Sep 15, 2026

Copy link
Copy Markdown
Contributor Author

Addressed the latest scanner diagnostics review comments in 5598e4d:

  • Finding paths from scanner output are no longer copied into the uploaded summary; non-empty paths are emitted as [REDACTED], covering credential-shaped paths such as ghp_*/replay.json.
  • The findings cap now applies to the complete JSONL stream with jq -nr 'limit(20; inputs | ...)', so multi-record scans stay bounded.
  • Added regressions for credential-shaped finding paths and a 45-finding scan that must emit exactly 20 finding lines.

Local validation:

  • .\.venv\Scripts\python.exe -m pytest tests/test_e2e_workflow_contract.py tests/test_scan_e2e_evidence.py -> 11 passed
  • D:\programs\git\bin\bash.exe -n scripts/scan_e2e_evidence.sh -> passed
  • git diff --check -> passed

@Teingi
Teingi merged commit 160587e into master Sep 15, 2026
18 checks passed
@PsiACE
PsiACE deleted the codex/ci-diagnostics branch September 28, 2026 03:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@PsiACE PsiACE Awaiting requested review from PsiACE PsiACE is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@AlexStocks