feat: add Triage Self-Hosted CI Failure template

[mayankpande88]

What this adds

triage_self_hosted_ci_failure — when a GitHub Actions run fails on self-hosted Kubernetes
runners, decide whether it was the code or the runner infrastructure, and offer a gated
re-run only when the failure was transient infra.

webhook ─ fetch_job_meta ── extract_runner ── snapshot ─┬─ rca ─ notify_summary ─ notify_evidence
        └ fetch_failed_log ────────────────────────────┘                              │
                                                         approve_remediation ─────────┘
                                                              └ retry_run ─ notify_retry

Why it is webhook-triggered and not manual

Every existing template is type: manual. This one cannot be, and that is the whole point.

The decisive evidence is perishable. Runner pods are deleted the moment their job ends, and
kubectl events age out in roughly an hour. By the time a human notices CI is red and opens an
assistant, describe pod returns NotFound and the event stream is gone — the investigation is
reduced to metric archaeology. Firing on the webhook captures that state while it still exists.

Trade-off: the trigger is inert until the tenant creates a workflow_webhook integration named
github_ci_webhook and points a repository webhook at it with the workflow_run event. There is
no error to signal this — it simply never fires. Setup steps are in a comment above the trigger.
webhook is already in TRIGGER_TYPES in scripts/validate.py, so CI accepts it.

The AI picks an action, never writes one

rca returns remediation.action from a closed list — rerun_failed or none — and the workflow
runs a pre-written command for the chosen branch. The model never emits shell.

This matters because the approval would otherwise be theatre: scm.github.cli executes arbitrary
bash with an org GitHub token, so a human approving a command they did not read is privilege
escalation with extra steps. rerun_failed also requires infra and transient, so a code failure
gets no button — re-running one wastes a runner and hides the defect.

Bugs carried forward from live testing

Each of these was found by running the workflow against real failures, and each is commented in the
template so the next author does not rediscover it:

• | from_json on every payload reference. The webhook body reaches tasks as a JSON string,
  not an object, so Inputs.webhook_payload.repository.full_name errors with Can't use Getitem on None. The filter sees parsed JSON; tasks do not.
• | int on the run id. JSON numbers decode as floats, so the id renders 30280501006.0 and
  gh run view fails.
• ; not && in the snapshot command. A missing runner pod is the NORMAL case, so chaining
  with && let describe pod abort the two event queries behind it — losing precisely the
  perishable evidence the task exists to capture.
• status in ['approve','approved']. core.approval emits the chosen approval_options value
  (approve), not approved. Comparing against approved skips the retry silently: the approval
  succeeds, the workflow completes, and the human who clicked Approve gets nothing.

Validation

python3 scripts/gen_manifest.py
python3 scripts/validate.py   →  OK: 25 template(s) valid.

Also exercised end-to-end on a dev cluster against a real GKE spot preemption: the runner node was
preempted mid-job, the workflow classified it infra / node / transient with confirmed evidence
from pod events, and proposed rerun_failed.

Notes for review

• Timeout is 4h. The validator sums task timeouts (185m, dominated by the 90m approval window)
  against the workflow budget. I raised the ceiling rather than shrink the approval — a retry prompt
  that expires in 30 minutes is useless overnight.
• message_thread_id on core.approval needs nudgebee-enterprise#35003, which added the
  passthrough. Older runbook-server ignores the param; the prompt just posts top-level.
• Snapshot payload is large — namespace-wide plus Node events ran ~69KB into the RCA prompt in
  testing. Narrowing it with --field-selector would cut tokens at the cost of neighbouring-pod
  context the agent sometimes uses. Left broad deliberately; worth revisiting.
• labels are limited to the schema enum, so this is tagged diagnostic + deploy; there is
  no ci or approval label available.

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.