feat(gate): configurable advisory check-runs so a non-resolving external status never blocks or stalls the gate forever

[JSONbored]

Context

Any GitHub App a repo installs alongside gittensory (a security scanner, a contributor-trust analyzer, a license/CLA bot, etc.) can post its own check-run with a conclusion outside GitHub's normal pass/fail vocabulary — most commonly action_required on a status: completed run, used as a deliberate "a human needs to act on MY dashboard" terminal state. gittensory's live CI aggregate (src/github/backfill.ts:2456-2463) already special-cases action_required, but only for ONE specific cause: a fork PR awaiting GitHub's own "Approve and run" workflow gate (a state that resolves itself once a maintainer clicks Approve). To fix that case, action_required was excluded from CI_FAILING_CONCLUSIONS entirely and falls through to "pending" instead.

That exclusion doesn't distinguish "transient, self-resolving" (the Approve-and-run case) from "durable, external-system-resolving" (a third-party app's own completed, terminal action_required that will never flip on its own — it needs a human to clear it in that app's UI, not in GitHub). gittensory treats both identically, so a repo running any such app gets its contributor PRs stuck:

• The disposition planner has no staleness override. src/settings/agent-actions.ts:710-712,739 — ciPending = ciState === "pending" || ciHasPending === true; while true, planAgentMaintenanceActions returns immediately with no merge, no close, no manual-review label. Nothing ever flips ciHasPending back, so this is a permanent hold, not a temporary one.
• The review-trigger's 30-minute staleness cap (src/queue/processors.ts:3604-3649, STUCK_CI_DEFER_MS at :3714) only fires when the stuck check is NOT a required branch-protection context. If it IS required (or the repo has zero required contexts configured, see below), ci.hasVisiblePending short-circuits the cap at :3617 and the PR defers with no escape, ever.
• A repo with no branch-protection required-status-checks configured at all hits this every time: enforceRequiredOnly is false, so isRequired() (backfill.ts:2762) returns true for every check unconditionally — meaning any app's stuck check is treated as a required blocker even though nothing was ever configured to require it.

Live evidence (verified via gh api against this org's own two repos, both running Superagent — superagent.sh — as a contributor-trust/security scanner)

• gittensory (branch protection requires 2 contexts, validate + Superagent Security Scan; Superagent's separate Contributor trust check is NOT one of them): Contributor trust sits at action_required indefinitely. The 30-min cap does eventually force exactly one review through (capped at 1 per head SHA / 7-day lookback, CI_STUCK_FINALIZE_MAX_PER_SHA at :3708) — e.g. #4346 got a single "approve/merge recommended" review 3.6h after opening — but the disposition never resolves: no manual-review label, no merge, no close, permanently. #4343/#4344/#4349 (5+ hours old at time of writing) haven't even gotten that one review yet.
• metagraphed (zero required contexts configured): the same Contributor trust check makes hasVisiblePending true, which blocks the 30-min cap outright. #4479 (2.5+ hours old) has zero gittensory activity of any kind — not even one review — while sibling PRs opened around the same time were fully reviewed and labeled within ~1-1.5h.

To be clear about scope: Superagent's own contributor:flagged/pr:flagged labels are working as intended and are NOT the problem — they should keep being applied exactly as they are today. This issue is only about how gittensory's own gate reacts to Superagent's (or any other installed app's) check-run conclusion.

In both cases: no auto-close (correct — this isn't a real CI failure), but also no review, no label, no terminal state, no signal to the maintainer that anything is waiting on a decision. The contributor's PR just looks abandoned.

Requirements

This must be entirely config-driven and generic — gittensory must not hardcode the name of any specific third-party app or check-run anywhere in behavior-triggering logic. This self-hosted engine is used by other maintainers running their own repos with their own set of installed GitHub Apps; whatever we ship here has to be something any operator can opt into via .gittensory.yml for whichever app(s) they run, not something wired specifically to one vendor. Any example app/check name may appear only in documentation/comments, never as a literal compared against in the gate/disposition code path.

1. New config field, e.g. gate.advisoryCheckRuns (final name up to whoever implements — match existing vocabulary; readiness.mode: advisory already establishes "advisory" as this project's term for "surfaced, never blocking"). A list of { name, appSlug } pairs — mirror the existing cla.checkRunName / cla.checkRunAppSlug pattern (config/examples/gittensory.full.yml:237-245), which already solves "match a check-run by name, trust it only when produced by a specific app slug." Empty/omitted list ⇒ byte-identical to today's behavior for every repo that doesn't configure it.
2. A listed check-run must never gate CI pass/fail and must never count toward "still running." Once its status is completed, it is settled — regardless of what its conclusion string is — and should be excluded from the CI aggregate the same way BOT_OWNED_CHECK_NAMES/isOwnGitHubAppCheckRun are already excluded (backfill.ts:2464-2486). This fixes both the permanent disposition hold and the review-trigger stall.
3. But it must not be silently swallowed either. When a listed check-run resolves to something other than a normal pass (i.e. not success/neutral/skipped), route the PR to the existing manual-review hold path (heldForManualReview in agent-actions.ts:771-775) instead of letting it either block CI or merge straight through a flag a maintainer installed a whole app to raise. Surface which check/app triggered the hold in the review comment/audit event so it's actually actionable, not just another silent label.
4. Fix (or explicitly scope out with a follow-up filed) the zero-required-contexts case: a repo with no branch-protection required-status-checks configured should not have every check implicitly treated as required (backfill.ts:2762, isRequired()) — this is what makes repo B's case worse than repo A's for the exact same upstream signal. If fixing this properly is too large to bundle here, file the follow-up before closing this issue rather than dropping it.
5. Document the new field in config/examples/gittensory.full.yml (matching the expectedCiContexts/cla.* documentation style — semantics, default, config-as-code-only note) and keep .gittensory.yml.example in sync.
6. Unit tests: CI-aggregate exclusion (configured vs. unconfigured, both directions), manual-review routing for a non-passing conclusion, and a regression test using a synthetic app/check name (not any real vendor name) so the test suite itself never hardcodes a specific third party either.

Deliverables

• .gittensory.yml schema field for advisory check-runs (list of name/appSlug pairs), deep-merge precedence consistent with every other gate field
• CI aggregate (backfill.ts) excludes a matched, completed check-run from ciState/hasPending entirely
• Disposition planner routes a matched check-run's non-passing conclusion to the existing manual-review hold, with the triggering check surfaced in the audit/comment
• Zero-required-contexts isRequired() behavior fixed, or a scoped follow-up issue filed for it
• config/examples/gittensory.full.yml + .gittensory.yml.example documentation
• Unit tests covering both the exclusion and the manual-review routing, using a synthetic (non-vendor-specific) fixture

Expected outcome

Any repo-installed third-party check-run that reports a durable, non-standard "needs a human" conclusion — a security scanner, a contributor-trust analyzer, whatever a given self-host operator runs — can be declared advisory per-repo. It never fails CI, never freezes the gate indefinitely, and never gets silently ignored either: it resolves promptly to an explicit, labeled manual-review hold a maintainer can act on. No behavior changes for any repo that doesn't opt in.

Effort

L

[linear-code]

JSO-3318

[JSONbored]

Re-verified against current `origin/main` (`src/github/backfill.ts` `reduceLiveCiAggregate`, ~lines 2790-2860) — this is partially, not fully, superseded. Worth updating scope before anyone picks this up, so it isn't re-solved from scratch:

Already resolved generically (not hardcoded to any vendor — matches this issue's own "must be config-driven and generic" spirit, just via a blanket heuristic instead of an opt-in allowlist):

• Zero-required-contexts case (requirement 4): `isConfirmedRequired()` uses positive evidence ("is this context actually in branch protection's required list") rather than `isRequired()`'s fail-safe "assume required unless proven otherwise" default — a repo with no required-status-checks configured no longer treats every third-party check as blocking.
• Non-required third-party `action_required` (part of requirement 2/3): routed to `nonRequiredFailingDetails`, which never feeds `ciState`/close decisions — visible but non-blocking, not silently swallowed either.

Still genuinely open:

• No `gate.advisoryCheckRuns`-style config field exists anywhere (requirement 1) — the current behavior is an unconditional heuristic (any non-github-actions app's completed `action_required`), not a per-repo opt-in list. Whether that's sufficient or a real config field is still wanted is a product call, not just an implementation gap.
• Requirement 3's core ask is NOT implemented: a third-party `action_required` check that IS a confirmed-required context still lands in `failingDetails` (blocking/auto-close-eligible) today, not routed to the existing `manual-review` hold path. This is the actual remaining gap — the required-context case still fails the way this issue originally described, it just no longer mis-fires on the unrequired-context case (that part was fix(github): treat completed third-party action_required checks as failing/manual-hold #4414/fix(github): third-party action_required check still auto-closed a PR on a repo with no branch-protection required contexts (#4812) #4735-era regression fix, not a full requirement-3 implementation).
• Requirement 6 (regression tests using a synthetic, non-vendor-specific fixture) — worth confirming test coverage exists for the required-context path specifically.

Recommend narrowing this issue's scope to just the still-open piece (manual-review routing for a confirmed-required third-party `action_required` check, plus deciding whether an explicit config allowlist is still wanted on top of the existing generic heuristic) rather than closing it outright.