Skip to content

test(ci): verify affected validation before rollout - #57

Closed
santi020k wants to merge 2 commits into
release/v1.4.0from
test/affected-ci-verification
Closed

santi020k wants to merge 2 commits into
release/v1.4.0from
test/affected-ci-verification

Conversation

@santi020k

@santi020k santi020k commented Oct 25, 2026 •

Copy link
Copy Markdown
Owner

Completed the ordinary-PR verification for the affected-validation pilot in PR #56. This PR used real pull_request events and actual base 448f4d8a62c506c8e5bb96a3a21e41d1841f2166 on release/v1.4.0; neither the base nor full-mode flag was overridden.

Revision Hosted result Selected checks
2fa570bb2daa05675e729564b4c43da2351e6799 — Action test comment CI passed, validation 24.95 s Action workspace only: all 12 tests, build, bundle verification; browser installation/build/tests skipped, planner job passed
84f8337ce2cdb610ff73e957ccd10e76a0399b23 — site-content comment CI passed, validation 8.91 s Site workspace tests/build; browser required and passed 63 tests with one expected skip in 40.2 s

Both quality-ci-plan artifacts report mode: affected, the expected workspace, correct browser selection, and exit code zero. Both revisions retained live npm audit, all 32 repository script checks, and a passing Changeset check against the actual release base.

For context, the release revision's full validation task took 67.76 s. These runs have different cache and runner conditions; this is selection/coverage evidence, not a controlled end-to-end speedup claim. Local controlled benchmarks remain in docs/affected-ci-pilot.md.

Closed without merging. The harmless probe changes are not release content. The branch is retained for reproducibility. No release or wider rollout was performed.

Exercise Action-only selection and browser skipping without changing behavior.
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review 🔄 Running since 2026-10-25T20:44:02.832985Z 2fa570b PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Replace the Action-only probe with a site-content comment.
@santi020k santi020k closed this Oct 25, 2026
santi020k added a commit that referenced this pull request Oct 25, 2026
Check the standalone cargo-audit version and wait for enlarged text before measuring page width.
Record successful Action and site validation from ordinary PR #57.
santi020k added a commit that referenced this pull request Oct 25, 2026
Prepare Quality 1.4.0 with CLI check previews, safer file handling,
refreshed documentation, and an affected-validation pilot in this
repository.

- Add `quality check --dry-run` / `--explain`; preserve file permissions
during atomic writes and scope changed-file discovery beneath a nested
`--root`.
- Refresh documentation with Lumen 4, accessible navigation and
examples, page transitions, and production browser coverage.
- Use existing Quality analyzers and Turbo affected-package selection
for ordinary PRs. Shared or unknown inputs fall back to full validation.
Release branches, manual CI, pre-push, and publication retain full
validation.
- Fix the 16 recorded npm advisories and regenerate the shipped Action
bundle. Replace deprecated `serde_yaml` / unmaintained `unsafe-libyaml`
with maintained `yaml_serde 0.10.7` and `libyaml-rs`.
- Run pinned cargo-audit 0.22.2 directly with `--deny warnings`,
read-only permissions, no advisory exceptions, and a fresh advisory
database. This removes the old Action's Node 20 and Checks API reporting
warnings. Check the standalone executable's version explicitly.
- Add CodeQL scans on pushes to main. Restrict publication and site
deployment to merged main. Reconcile all included Changesets into
CLI/Action 1.4.0 and site 0.3.0.

### Validation and review

- `pnpm validate`: passed locally, including seven analyzers, package
tests/builds, playground verification, all 32 repository script tests,
release metadata, Action bundle reproducibility, and zero known npm
vulnerabilities.
- Rust 1.85 compilation and YAML regressions passed: aliases, quoted
scalars, multiline commands, tags, workflow triggers, serialization,
invalid/ambiguous input, and existing CLI behavior.
- Live Rust audit with `--deny warnings`: passed. The original lockfile
fails the same command on RUSTSEC-2024-0449, confirming the warning is
enforced.
- Production browser suite: 63 passed, one expected desktop skip. Fix
the homepage heading's overflow with wider platform fonts by wrapping
within its container, without shrinking enlarged text. Browser
regressions verify the requested font size, heading containment, and the
strict 320px page boundary; ten repeated focused checks and
mobile/desktop visual comparisons also passed.
- Independent reviews of the complete release and audit/layout
follow-ups found no outstanding actionable issues.
- All six hosted workflows passed on
`d5e6011a0ea5b4387d47c61a942862dfe42ec1bd`: [full validation and browser
checks](https://github.com/santi020k/quality/actions/runs/40233506179),
[Linux/macOS/Windows](https://github.com/santi020k/quality/actions/runs/40233506250),
[CodeQL](https://github.com/santi020k/quality/actions/runs/40233506267),
[Rust
audit](https://github.com/santi020k/quality/actions/runs/40233506167),
[Dependency
Review](https://github.com/santi020k/quality/actions/runs/40233506165),
and [PR
title](https://github.com/santi020k/quality/actions/runs/40233506174).
- The saved release report confirms `mode: full`, no skipped checks, and
exit code zero: 52.28 s validation, with all 63 browser checks passing
separately in 48.4 s. Rust Audit has no warning annotations. The only
remaining warning is the upstream Scorecard signal below; GitHub runner
migration/capacity notices are informational.
- The automatic GitHub Codex review completed on the latest commit with
no findings. No review submissions or unresolved inline threads were
present at the final check; this task does not merge the release.

### Ordinary-PR pilot verification

[PR #57](#57) used ordinary PR
events against the actual unmerged release base, without overriding
change selection. It was closed without merging its harmless comments.

| Scenario | Result |
| --- | --- |
| [Action-only
change](https://github.com/santi020k/quality/actions/runs/40232145573) |
Passed, 24.95 s validation; all 12 Action tests, build and bundle
verification; browser installation/build/tests skipped |
| [Site-only
change](https://github.com/santi020k/quality/actions/runs/40232368546) |
Passed, 8.91 s validation; site tests/build; browser required and passed
63 checks in 40.2 s |

Both saved reports confirm affected mode and the expected workspace.
Live npm audits, all 32 repository script checks, and Changeset checks
against the actual PR base passed in both cases. Release validation
remains full.

The [benchmark and hosted
evidence](https://github.com/santi020k/quality/blob/release/v1.4.0/docs/affected-ci-pilot.md)
retain the controlled local measurements: warm deterministic validation
improved by 79% for README edits, 61% for Action tests, 77% for site
content, and 16% for Rust source. Hosted samples have different
cache/runner conditions and exclude setup and queueing; they prove
selection and retained coverage, not a controlled end-to-end speedup.
The failed initial benchmark audit and intermediate hosted failures
remain documented rather than counted as successful release gates.

### Remaining upstream warning

Dependency Review still reports upstream `http-cache-semantics 4.3.0`
OpenSSF Scorecard **2.9**, below the existing threshold of 3. It is the
latest stable version, and current Astro still depends on it. Quality
cannot fix the upstream repository's review, branch protection, security
policy, fuzzing, and static-analysis controls. No exclusion, lower
threshold, or advisory suppression was added. The vulnerability audit
passes; this upstream maintenance signal remains visible and should be
reassessed before wider rollout.

### Compatibility and recovery

Public 1.x CLI, configuration, reports, exit codes, and Action inputs
remain compatible; Rust 1.85 remains supported. The private development
workspace requires Node >=22.19.0 to match an existing dependency;
contributors on 22.18 must update. The Action stays on Node 24. No
persisted-data migration is needed.

To disable affected CI, set `QUALITY_CI_FULL=true` in both validation
jobs; `pnpm validate` and `pnpm test:site` remain available. Publication
must use the GitHub release workflow after merge. If a published release
fails, prepare a patch through that workflow without moving published
tags; recover the site through a reviewed follow-up deployment from
main.

This task does not merge the release, publish packages, deploy the site,
or roll the pilot out to other repositories.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant