tests: property-based action-pin assertions (unbreak Dependabot bumps) - #90
Conversation
Replace the four frozen action-pin fixtures (exact gitleaks SHA; three '# v4' major-version regexes for codeql init/analyze and upload-artifact) with anchored property assertions: a real 'uses:' line, a full 40-hex commit SHA, and a version comment (dotted forms accepted). Every Dependabot actions-group PR (#9/#29/#67) failed CI by construction because Dependabot cannot update fixtures that mirror the workflow it bumps. Full-SHA pinning stays enforced repo-wide by test_every_external_action_is_pinned_to_a_full_commit; WHICH SHA runs is governed by CODEOWNERS review of workflow diffs. Anchoring to uses: lines is a strictness increase over the old substring assertions (Codex cross-family review, DISAGREE-MINOR H, both concerns integrated; its proposed regex was corrected empirically to accept dotted version comments, which Dependabot writes). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ff5db0c489
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| r"(?m)^\s*(?:-\s*)?uses:\s*" | ||
| r"gitleaks/gitleaks-action@[0-9a-f]{40}\s+# v\d+(?:\.\d+)*\s*$", |
There was a problem hiding this comment.
When an inline version comment is removed or moved, Python's \s+ can consume the newline, so a pinned uses: line followed by a separate # v4 comment still satisfies this assertion; ^\s* similarly allows matching an action-looking line inside a YAML block scalar. The same construction is used for all four new assertions, so they do not actually guarantee a real uses: line with an inline version comment as documented. Use horizontal whitespace such as [ \t] around the tokens so the multiline anchors remain line-local.
Useful? React with 👍 / 👎.
1 participant
Summary
Dependabot actions-group PRs (#9, #29, #67) fail CI by construction: two test files freeze exact action-pin details (one exact Gitleaks SHA; three
# v4major-version regexes for CodeQL init/analyze and upload-artifact) that Dependabot legitimately bumps — and Dependabot cannot update the fixtures that mirror the workflow it changes. This PR makes those four assertions property-based so a routine pin bump passes while every security property stays enforced.What changed
tests/test_ci_security_contract.py(test_secret_scan_combines_local_and_full_history_scanners): the exact-SHAassertIntoken is replaced by an anchored regex requiring a realuses:line forgitleaks/gitleaks-action, pinned to a full 40-hex SHA, with a version comment.tests/test_provider_plugin_retirement.py(test_public_security_workflows_are_present): the three@[0-9a-f]{40} # v4regexes are replaced by the same anchoreduses:-line form with# v\d+(?:\.\d+)*(Dependabot writes dotted comments like# v3.0.0).Security invariants (unchanged or strengthened)
test_every_external_action_is_pinned_to_a_full_commit— floating tags still fail.uses:lines, so a comment or string mentioning an action no longer satisfies the assertion (strictness increase over the old substring checks; Codex review recommendation)./.github/workflows/plus review of workflow diffs. The frozen fixtures only mirrored the workflow — any PR editing a workflow could edit the fixture in the same PR — so no real enforcement is lost. Accepted trade-off (flagged by Codex): a workflow-only SHA change no longer trips a test; it is caught by the workflow diff review itself.Verification
python3 -m unittest discover -s tests -t .— 660/660 OKpython3 -m unittest discover -s scripts -p 'test_*.py'— 327/327 OKactions-c1735b5799): all four match its bumped workflows; negative cases (floating tag, comment-only mention, missing version comment) correctly rejected.python3 scripts/build-changelog.py --dry-runOK;python3 scripts/check_release_consistency.pyOK;git diff --checkclean.Boundary declaration
Generated and release surfaces
changelog.d/fragment present; generatedCHANGELOG.mduntouched.author: claude
author_model: claude-fable-5 (effort: default)
standing_directives: operator-instructed (2026-08-04, "draft the durable fix as a PR and proceed through merge"); adversarial-architecture review folded into Step-2 per 2026-07-03 directive; fragment-only changelog
tier: 2
cross_check: PROCEED after integration (codex gpt-5.5, effort medium, hardened codex_exec wrapper repo-review mode; verdict DISAGREE-MINOR confidence H; both concerns integrated: uses:-line anchoring adopted, accepted-trade-off drift-tripwire loss documented above; Codex's proposed regex empirically corrected to accept dotted version comments)
post_condition: 660/660 + 327/327 OK; four regexes match live Dependabot branch actions-c1735b5799; negative cases rejected; changelog dry-run OK; release-consistency OK; git diff --check clean
mcp_coverage_gap: NONE
contributor_rights: OWNER-AUTHORED
operator_reserved: no
plugin_affected: NONE — test infrastructure, no distributed plugin content
readme_refresh_status: README unaffected (no release, no version bump)
🤖 Generated with Claude Code