Validate or retire the suspected shared-baseline cause of Dependabot CI failures #481

Status: low-confidence investigation with counter-evidence

The earlier Dependabot sweep proposed a shared failure affecting unrelated updates, mentioning libc, clap, toml, camino, tar and tokio. That inventory was not reliable enough to establish live PR identities or a common compiler/test error. This issue does not assert that those PRs currently exist or fail.

A fresh repo:leynos/whitaker is:open author:app/dependabot status:failure search returned no results. That is counter-evidence for an active incident, although it is not a substitute for inspecting current PR check runs. The first task is therefore to verify or retire the suspected cluster, not to implement a speculative CI change.

The inspected Dependabot configuration already includes an rstest-bdd family group and minor/patch groups for Cargo, GitHub Actions and workflow-test Python tooling. Do not duplicate existing grouping fixes.

Related existing concern: #271 describes an installer that can obtain a mutable lint suite despite a pinned installer version. Treat this as one possible provenance hazard only if the affected job actually exercises that path. This investigation must not duplicate #271 or assume its mechanism explains these alleged failures.

Hypotheses

H1, shared baseline: the first actionable failure comes from code, a workflow, a toolchain, a lint suite, or another CI input shared by multiple dependency PRs and occurs without the proposed dependency updates.

H2, bot-context baseline: an equivalent PR with no dependency changes fails because its event/permission/environment context differs from the successful default-branch workflow.

Alternatives include stale branches, independently incompatible package versions, a genuinely coupled update family, nondeterministic tests, cache contamination, infrastructure failure, and an already resolved or incorrectly identified historical cluster. Several causes can coexist.

Investigation and falsification plan

1. Validate the incident before spending build time. Enumerate current open Dependabot PRs directly and inspect their actual latest check suites, required jobs and run attempts. Do not depend exclusively on the search status:failure qualifier. Resolve any relevant historical PRs from real metadata, not the earlier guessed numbers. Distinguish cancelled/skipped/missing checks, review gates, external services and infrastructure failures from compilation or test failures.
2. If no affected PRs remain and accessible historical evidence does not establish an unresolved problem, record the evidence and close this issue as not reproduced/resolved. No configuration patch is required to satisfy this ticket.
3. For each genuine candidate, record dependency delta and actual lockfile resolution, base/head/merge SHA, workflow revision, runner image, toolchain, relevant tool/lint-suite revisions, feature flags, command, cache key, event/permission context and a stable fingerprint of the first actionable error. Red status or a shared final exit code is not a fingerprint.
4. Choose a bounded representative sample, ideally three unrelated updates if available, and one fixed current base. Run the same smallest failing command for the unchanged control and each isolated update under equivalent conditions. Use a clean-cache check where it tests a specific hypothesis. Record actual checked-out commits; do not compare different feature matrices or a privileged push job with a restricted bot PR and call that a control.
5. Test H2 using equivalent permitted PR context and trusted code; do not execute untrusted code with elevated credentials to simulate a control. If the failure occurs before dependency compilation or test execution, inspect the failing setup/permission contract first.
6. Support H1 when unchanged controls reproduce the same first error and a specific baseline-only correction removes it from both controls and representative update branches. Identify the responsible revision/input, not merely the job name.
7. Falsify H1 for a particular failure when matched unchanged controls pass, the isolated dependency delta reliably introduces that error, and reverting only that delta removes it. Falsify H2 when equivalent no-change PR context passes and the dependency delta alone causes the error. Inaccessible logs, inconsistent runs or unmatched environments yield an inconclusive result, not a diagnosis.
8. Re-test beyond the first blocker. Fixing a common setup failure can expose independent package incompatibilities; do not infer that one baseline fix makes the whole queue safe to merge.

Conditional rectification

• Common code/workflow/toolchain fault: repair it once in its owning component, add a regression test or pinned-input/contract check, then refresh affected PRs and run the normal gates.
• Mutable lint-suite provenance: link the exact installed revision and invocation to whitaker-installer installs an unpinned lint suite (rolling/default branch) #271, contribute the evidence there, and reuse its fix rather than opening a second implementation issue.
• Bot-context fault: make the workflow explicitly handle that context with least privilege. Do not turn required tests into success-on-skip or grant broader secrets/token access as a shortcut.
• Drift: retain necessary dependency changes on a current base; close only changes proven superseded, preserving human commits before any recreation.
• Coupled dependencies or a single-version incompatibility: split out a minimal evidence-backed grouping or migration issue. Distinct error fingerprints deserve distinct diagnoses.
• Flakiness/infrastructure: establish a bounded reproducer or service evidence and address the source; do not repeatedly rerun until a lucky green result masks it.

Acceptance criteria

• A verified live/historical inventory replaces the unreliable sweep, or the concern is explicitly retired with evidence.
• The report includes exact inputs, representative log links, error fingerprints and a matched-control comparison.
• H1 and H2 have separate supported/falsified/inconclusive outcomes; any residual failures have named follow-ups.
• Any implemented repair has a regression/provenance check and passes the existing relevant CI gates.
• No blanket dependency grouping/ignores, disabled quality gates or unrelated migrations are required merely to close this investigation.

Evidence sweep addendum — 7 October 2026

Direct enumeration establishes different active failures, not the original cluster

The full open-pull listing and actual head checks find active failing action updates even though the earlier status-filtered search was empty. This does not rehabilitate the unverified libc/clap/toml/camino/tar/tokio cluster.

• PR #480, uploader a5765019912a8ab6882b12db049c7cde635f3a85 → ff1dd759dfffc0db3459e30e833f52437ee62b57: run 36961435573 attempt 1/job 110695783583, pull_request, tested checkout 4302465, head c33b0d6. The suite reports 1 failed/724 passed. codescene_uploader_contract_test.py:116 demands the old approved uploader SHA.
• PR #476, grouped taiki-e/install-action 18b1216eba7f8039b0f8d131d5473787f0edce68 → 4cef1412cce204788f482e778a0b9187f9626a29 and setup-uv 20cfd1bf945f4377ade1205e4dbc17946fc9a30d → c18668ad3cf93ea998bef934396af7bb5c839dc7 update: run 37249299130 attempt 1/job 111573442043, pull_request, checkout 2e59ed8, head 297a487, base 0b2973b, Python 3.14.5/pytest 9.0.2. 2 failed/760 passed: the release-binstall and Ubicloud provenance contracts demand the prior reviewed installer SHA. The diff retains fallback none; the error is approved-pin mismatch, not proof that source fallback occurred.

E, high confidence for these static approval transitions. They are not evidence of a common compiler/application defect or of #271's mutable lint-suite hazard. No matched old/new runtime control establishes whether the action updates themselves are suitable. Do not change #271 on this evidence.

Falsification: isolate each action delta on one fixed base and run only the existing contract fixture first; distinguish approved SHA provenance from semantic properties (locked/prebuilt installation, exact tool version, fallback none, credentials and permissions). Review the new action's published source. If a companion is irrelevant, revert it in the isolated fixture. Earlier blockers leave runtime comparisons inconclusive.

Conditional rectification: if the new action revision is reviewed and its security/cost contracts hold, update the canonical approval fact and dependent assertions together in a small deterministic operation. If it changes behaviour, retain the guard and track that migration separately. Do not auto-accept every SHA or remove provenance/security tests. Acceptance requires rejection of unreviewed pins, retention of fallback none/prebuilt constraints and execution of consumer gates after explicit approval. No source change, fresh build or PR action was performed. Root executable/static policy work belongs to shared-actions#588/concordat#270; consumer approval remains Whitaker-owned.