Skip to content

fix(ci): clear the fixable Scorecard findings and add frontend coverage to Codecov - #3005

Merged
vavallee merged 7 commits into
mainfrom
fix/ci-scorecard-and-coverage
Oct 5, 2026
Merged

vavallee merged 7 commits into
mainfrom
fix/ci-scorecard-and-coverage

Conversation

@vavallee

@vavallee vavallee commented Oct 5, 2026 •

Copy link
Copy Markdown
Owner

Summary

Fixes the four OpenSSF Scorecard findings that can be fixed in code, and adds frontend coverage to Codecov so the badge stops describing Go internal/ only. Scoring rules below are from the Scorecard source (ossf/scorecard main, which vendors osv-scanner v2.3.2), not from the docs.

Check Before Expected after Change Rule (scorecard source)
Token-Permissions 0 10 dev-preview.yml: top level is now contents: read; contents: write, actions: write, pull-requests: read moved onto the assemble job checks/evaluation/permissions.go: a top level contents/actions write costs 10 points each (reduceBy). A job level write only deducts when the same file's top level is write (score to 0) or undeclared (minus 0.5)
Signed-Releases 0 2 at v1.40.1, rising to 10 after five releases goreleaser job uploads the attest step's Sigstore bundle as bindery_<version>.intoto.jsonl probes/releasesHaveProvenance: only .intoto.jsonl counts as provenance (10 per release); .sigstore.json would only count as a signature (8). Score is the floor of the average over the last 5 releases, and older releases cannot be backfilled
Pinned-Dependencies 9 10 semgrep installed with --require-hashes from a pip-compile lockfile; Python pinned to 3.12 via SHA pinned actions/setup-python; Dependabot pip entry keeps it current checks/raw/shell_download_validate.go isUnpinnedPipInstall: --require-hashes makes a pip install pinned
Vulnerabilities 9 10 root osv-scanner.toml ignores GO-2026-5932 with a reason clients/osv.go calls osvscanner.DoScan with no config override, and osv-scanner loads osv-scanner.toml from the directory holding go.mod

Notes per item

Token-Permissions. ci.yml deploy-prod keeps its job level statuses: write. On its own it is only a warning: ci.yml declares contents: read at the top, so the job level case in permissions.go never deducts. Nothing about that job changed.

Signed-Releases ordering and failure. The release is public before goreleaser runs (the CHANGELOG step creates it), so the upload only adds one asset afterwards. It retries three times, uses --clobber so a rerun is idempotent, and is continue-on-error with an ::error annotation. Failing the goreleaser job there would skip deploy-prod and the Discord post for a release whose binaries are already published and attested; the asset can be re-added by hand from gh attestation download. The bundle is one JSON line (actions/attest writes JSON.stringify(bundle) + EOL), which gh attestation verify --bundle accepts. The id: attest sits below with: so this merges cleanly with Dependabot's #2966 bump (checked with git merge-tree); bundle-path is still an output in v4.2.2.

Vulnerabilities. No fix exists: the advisory is introduced: 0 with no fixed version, and x/crypto v0.57.0 is both what we use and @latest. go list -deps ./... shows no openpgp package and the pinned govulncheck reports "Your code is affected by 0 vulnerabilities".

Pinned-Dependencies sweep. Workflows and the three Dockerfiles had no other unpinned pip, npm install -g, go install @latest or curl | sh. The make security target had two @latest installs; Scorecard does not read the Makefile, but they are pinned now too (gosec v2.29.0, govulncheck to the existing GOVULNCHECK_VERSION).

Codecov. validate-frontend (PRs) and test (main and development) now run npm run test:coverage (same suite plus v8 coverage) and upload web/coverage/lcov.info with flags: frontend and disable_search: true. lcov paths are repo relative (web/src/...) through the reporter's projectRoot, so no Codecov path fixing is needed. The single project status moved into flag_management: each flag compares against its own base with the same auto target and 1% threshold, with carryforward, so a frontend upload cannot fail the backend status. Patch stays at 70%. Tests, test helpers, locale JSON and config files are excluded both in vitest and in .codecov.yml. cmd/** stays ignored: go test -cover ./cmd/... gives 25.2% for cmd/bindery (wiring), 0% and 38.7% for the two separate services. None of the Codecov statuses are required checks on main.

Removing web/** from the ignore list also puts frontend lines into codecov/patch: a PR that changes web code now needs 70% of its changed frontend lines covered, same target as Go. codecov/patch is not a required check, so this shows up as a red status, not a blocked merge.

On a v* tag the test job gates the image and goreleaser, so there it runs the plain Go and frontend suites with no coverage and no uploads (the tagged commit already reported on main). That keeps coverage's extra runtime and its timeout risk out of the release gate. Backend got the same split.

The README badge is the shields.io overall number, so once main has a frontend upload it reports Go plus web combined. Codecov's Go metric still counts partially covered lines as misses; that was left alone on purpose.

What can only be verified at the next tag

  • The goreleaser job attaching bindery_1.40.1.intoto.jsonl to the v1.40.1 release, and gh attestation verify bindery_1.40.1_linux_amd64.tar.gz --repo vavallee/bindery --bundle bindery_1.40.1.intoto.jsonl passing against it.
  • Scorecard picking it up: Signed-Releases should read 2 after v1.40.1 and climb by 2 per release.
  • The changelog fragment only lands in v1.40.1 if the open release PR chore(release): assemble the v1.40.1 changelog #2991 is re-assembled after this merges; otherwise it waits for the next release (and the "From v1.40.1" lines in docs/DEPLOYMENT.md and SECURITY.md need adjusting).
  • Token-Permissions, Pinned-Dependencies and Vulnerabilities need only the next Scorecard run on main, not a tag.
  • First Codecov uploads with both flags happen on this PR's CI; the per flag statuses and the badge settle after the first main push.

Checklist

  • Tests added or updated (no test code changed, by design)
  • Doc-update gate cleared: docs/DEPLOYMENT.md and SECURITY.md describe verifying with the bundle
  • Changelog fragment changelog.d/3005-scorecard-and-coverage.md (Security)

Test plan

  • actionlint on all workflows (1.7.12)
  • goreleaser check
  • make go-version-check
  • cd web && npm ci && npm run test:coverage: 105 files, 1220 tests pass, lcov written with web/src/... paths. Frontend coverage 79.9% lines, 77.82% statements, 71.17% branches, 68.97% functions
  • cd web && npm test, npm run build, npm run typecheck, npm run lint (0 errors)
  • .codecov.yml validated with https://codecov.io/validate
  • pip install --require-hashes -r .github/requirements/semgrep.txt into a clean Python 3.12 venv, semgrep --version prints 1.179.0
  • osv-scanner v2.3.2 (Scorecard's version) over a copy of the tree, default plugins and Scorecard's plugin set: GO-2026-5932 filtered by the new file, nothing else reported. Also scanned semgrep.txt on its own: no issues
  • govulncheck at the pinned revision: 0 vulnerabilities reached
  • Release upload step exercised for real (tag only)
  • Codecov frontend flag rendering on this PR (pending CI)

One run of the coverage suite had a single 5s test timeout (HistoryPage pagination) while the box was at load 16 on 6 cores; it passed alone with coverage and on the rerun. Coverage adds about 25% to suite time locally (72s to 90s). Worth watching on the first CI runs.

🤖 Generated with Claude Code

https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9

vavallee and others added 7 commits October 4, 2026 23:58
The workflow granted contents: write and actions: write at the top level,
which Scorecard's Token-Permissions check scores as a full ten point
deduction each (checks/evaluation/permissions.go, reduceBy). Declare
contents: read at the top and give the write scopes to the only job that
force pushes development and dispatches ci.yml. Behaviour is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>
security.yml installed semgrep with a bare pip install, the only finding
behind Scorecard's Pinned-Dependencies score of 9. Pin semgrep and its
transitive dependencies with pip-compile --generate-hashes, install with
--require-hashes (which Scorecard treats as pinned, see
isUnpinnedPipInstall in checks/raw/shell_download_validate.go), pin the
interpreter the lockfile was compiled for, and let Dependabot recompile it.

Also pin the two @latest installs in make security to fixed versions;
Scorecard does not read the Makefile, but local and CI tooling should not
drift apart.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>
The goreleaser job already attests the release archives, but the bundle
only lives in GitHub's attestation store. Scorecard's Signed-Releases
check reads release assets and counts provenance only for names ending in
.intoto.jsonl (probes/releasesHaveProvenance), so every release scored 0.

Upload the attest step's bundle as bindery_<version>.intoto.jsonl after
goreleaser has published. The upload retries, uses --clobber so reruns
are idempotent, and is continue-on-error with an ::error annotation:
failing the job after the release is public would only skip deploy-prod
and the Discord post. Document verifying against the file with --bundle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>
GO-2026-5932 flags the deprecated golang.org/x/crypto/openpgp package at
module level with no fixed version, so it matches every x/crypto release
(v0.57.0 is current). Bindery imports no openpgp package and govulncheck
reports no call. It was the one finding behind Scorecard's
Vulnerabilities score of 9.

Scorecard runs osv-scanner's DoScan without a config override
(clients/osv.go), and osv-scanner loads osv-scanner.toml from the
directory holding each lockfile, so a root level IgnoredVulns entry is
honoured. Checked locally with osv-scanner v2.3.2, the version Scorecard
vendors: the advisory is filtered and nothing else is reported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>
.codecov.yml ignored web/**, so the badge only ever described Go code in
internal/ and tools/. Collect v8 coverage from the existing vitest run
(new test:coverage script, lcov written to web/coverage/ with repo
relative paths) in validate-frontend and test, and upload it with
flags: frontend and disable_search so it cannot pick up coverage.out.

Project status moves into flag_management: each flag compares against
its own base with the old auto target and 1% threshold, and carryforward
covers a commit that uploads only one flag, so frontend coverage can
never fail the backend status. Patch stays at 70%. Tests, test helpers,
locale JSON and config files stay out; cmd/** stays ignored (25% wiring
in cmd/bindery plus two separate services).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>
Dependabot rewrites the attest-build-provenance uses: line (#2966); an id
inserted right above it would conflict with that bump on merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>
On a v* tag the test job gates the image and goreleaser, so run the Go
and frontend suites there without coverage and skip both Codecov
uploads. Coverage adds about a quarter to the runtime, and the tagged
commit already reported coverage on main. Main and development keep
coverage and uploads. Name the changelog fragment after the PR (#3005).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>
@vavallee
vavallee marked this pull request as ready for review October 5, 2026 03:19

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clean CI/tooling PR. Nothing blocking; a couple of small notes.

Token-Permissions (dev-preview.yml:37-40): write scopes correctly scoped to the assemble job. Top-level contents: read is the right Scorecard-safe default.

Pinned-Dependencies (security.yml, semgrep.txt): --require-hashes with a pip-compile lockfile is the correct fix. SHA-pinned actions/setup-python, Dependabot pip entry — pattern is solid.

Vulnerabilities (osv-scanner.toml): ignore is well-documented with a reason and an implicit removal cue ("osv-scanner warns about unused ignores"). Appropriate given no fixed version and no reachable call path.

Codecov flags (.codecov.yml): project: false + per-flag flag_management is the right call to avoid a spurious regression on the first two-flag upload. carryforward: true handles partial-upload days correctly.

ci.yml upload retry loop (ci.yml:~648-658): Two minor issues in the bash:

for attempt in 1 2 3; do
  if gh release upload ...; then exit 0; fi
  echo "upload attempt ${attempt} failed; retrying"  # ← misleading on attempt 3
  sleep $((attempt * 10))                             # ← sleeps 30s after the final failure
done

On the third iteration the loop prints "retrying" when there is no retry, then waits 30 seconds before the ::error:: annotation fires. Not a correctness bug (the continue-on-error: true and three-attempt logic are fine), but worth tidying: gate the sleep on attempt < 3 or move it to the top of the loop body.

@vitest/coverage-v8 is not visible in the web/package.json diff, but npm run test:coverage --coverage requires it. The PR description's test run confirms it is already in devDependencies — just noting it so the next reader doesn't have to wonder.

— 🤖 Bindery triage bot (automated). Reply to correct me; a human will see it.

@vavallee
vavallee merged commit 2eeba27 into main Oct 5, 2026
41 of 43 checks passed
@vavallee
vavallee deleted the fix/ci-scorecard-and-coverage branch October 5, 2026 03:27
vavallee added a commit that referenced this pull request Oct 5, 2026
…gelog

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>
vavallee added a commit that referenced this pull request Oct 5, 2026
* chore(release): assemble the v1.40.1 changelog

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>

* chore(release): add #2942, #2944, #2987 and #2989 to the v1.40.1 changelog

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>

* chore(release): add #2998 to the v1.40.1 changelog

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>

* chore(release): add #2665, #2926, #2937 and #3005 to the v1.40.1 changelog

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016fJcCVbNnKsmj2MAAMwWj9
Signed-off-by: vavallee <vavallee@protonmail.com>

---------

Signed-off-by: vavallee <vavallee@protonmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant