feat(ci): introduce merge queue for main

[ggallen]

Copied from upstream: NVIDIA/OpenShell#1946

Problem Statement

main can break when a pull request passes CI on a stale branch head, then GitHub creates a final merge commit against a newer main that was never tested by the PR checks.

We saw this with PR NVIDIA/OpenShell#1870 and PR NVIDIA/OpenShell#1577. PR NVIDIA/OpenShell#1870 passed Branch Checks on stale head 29d57cc, whose merge-base did not include NVIDIA/OpenShell#1577. The final merge commit ff028ce0 combined NVIDIA/OpenShell#1870 TLS reload shutdown handling with NVIDIA/OpenShell#1577 compute watcher shutdown handling, introduced a duplicate shutdown_tx binding, and caused main Rust lint to fail in https://github.com/NVIDIA/OpenShell/actions/runs/27656754843/job/81792472668.

PR NVIDIA/OpenShell#1945 fixes that immediate break, but the integration gap remains.

Proposed Design

Enable GitHub merge queue for the protected main branch and require queued merge groups to pass the same gates required for normal PRs.

Implementation outline:

• Enable Require merge queue for the main branch protection/ruleset.
• Add the merge_group trigger to workflows that publish required PR gate inputs, especially:
  • .github/workflows/branch-checks.yml
  • .github/workflows/branch-e2e.yml
  • .github/workflows/helm-lint.yml
• Confirm .github/workflows/required-ci-gates.yml can evaluate and publish required gate statuses for merge-group runs, or update it so the required contexts are reported for merge queue validation.
• Keep the required contexts aligned with the existing PR gate contexts:
  • OpenShell / Branch Checks
  • OpenShell / E2E
  • OpenShell / GPU E2E
  • OpenShell / Helm Lint
• Document the expected maintainer workflow for adding a PR to the merge queue instead of merging directly.

GitHub documentation notes that merge queues validate PR changes applied to the latest target branch and any earlier queued changes, and that GitHub Actions workflows used as required checks must include the merge_group event: https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/configuring-pull-request-merges/managing-a-merge-queue

Alternatives Considered

Require PR branches to be up to date before merging.

That would likely have caught the NVIDIA/OpenShell#1870/NVIDIA/OpenShell#1577 interaction too, because NVIDIA/OpenShell#1870 would have had to rerun checks after updating to include NVIDIA/OpenShell#1577. However, it pushes more manual branch-update work onto contributors and maintainers. Merge queue is a better fit for a busy main branch because it validates the final integration state without forcing every PR author to repeatedly rebase or merge main by hand.

Rely only on push CI after merge.

This detects breakage after main is already broken, which is what happened here.

Agent Investigation

• Loaded watch-github-actions to inspect failing run 27656754843 / job 81792472668.
• Failure was cargo clippy --workspace --all-targets -- -D warnings rejecting unused variable shutdown_tx in crates/openshell-server/src/lib.rs.
• git blame and history showed the production duplicate channel came from the interaction of:
  • feat(gateway): add reconciler lease for HA multi-replica deployments NVIDIA/OpenShell#1577 / e73745f1 adding compute watcher shutdown handling.
  • feat(server): support TLS certificate hot-reload NVIDIA/OpenShell#1870 / ff028ce0 adding TLS reload shutdown handling.
• PR feat(server): support TLS certificate hot-reload NVIDIA/OpenShell#1870 checks passed on head 29d57cc, which did not include feat(gateway): add reconciler lease for HA multi-replica deployments NVIDIA/OpenShell#1577. The broken code only existed in the final merge commit onto newer main.
• Opened PR fix(server): share gateway shutdown channel NVIDIA/OpenShell#1945 to fix the immediate CI failure by sharing the existing gateway shutdown channel.

---

Comment from @elezar (maintainer)

Additional example: NVIDIA/OpenShell#1872 is another case this issue should cover.

NVIDIA/OpenShell#1872 fixed the server build after NVIDIA/OpenShell#1566 introduced gRPC rate limiting code that used the private tonic::body::BoxBody alias. The PR checks for NVIDIA/OpenShell#1566 passed before the final integrated state on main exposed the tonic 0.14 incompatibility. A merge queue, or a strict up-to-date-branch requirement, would have forced the final branch-plus-current-main state through the required Rust checks before merge instead of relying on push CI after main was already broken.

This reinforces that the work is not only a repository setting change. The required CI workflows need to support merge queue validation explicitly. In the current workflow set, .github/workflows/branch-checks.yml, .github/workflows/branch-e2e.yml, and .github/workflows/helm-lint.yml trigger on push to pull-request/[0-9]+ plus workflow_dispatch, but not merge_group. .github/workflows/required-ci-gates.yml also keys its aggregation around pull_request_target and completed workflow runs from those branch workflows. The implementation should update these workflows so the required contexts are reported for merge-group SHAs.

🤖 Finished Triage · ✅ Success · Started 8:34 PM UTC · Completed 8:36 PM UTC
Commit: 4e6610d · View workflow run →

Triage Summary

Severity: High — stale-branch merges have broken main multiple times.
Category: Feature (CI infrastructure)

Problem

PR CI runs on branch heads that may be stale relative to current main. When the final merge commit combines changes that were never tested together, main breaks. This has occurred at least three times with real-world impact.

Root Cause

The CI gate workflows (branch-checks.yml, branch-e2e.yml, helm-lint.yml) trigger on push to pull-request/[0-9]+ branches and workflow_dispatch only — they lack the merge_group trigger. The required-ci-gates.yml workflow aggregates results using pull_request_target and workflow_run events, which do not cover merge-group SHAs.

Implementation Scope

1. Add merge_group trigger to branch-checks.yml, branch-e2e.yml, and helm-lint.yml
2. Update required-ci-gates.yml to resolve PR metadata and publish gate statuses for merge_group events (the current logic assumes github.event.pull_request context)
3. Ensure concurrency groups and the pr_metadata jobs handle merge_group event payloads correctly (e.g., github.event.merge_group.head_sha)
4. Enable "Require merge queue" in the main branch protection ruleset
5. Document the maintainer workflow for queue-based merging

Proposed Test

Scenario: Two PRs modify overlapping code
1. Open PR-A and PR-B, both touching the same file
2. Add both to the merge queue
3. Verify PR-B is tested against main + PR-A (the merged state)
4. Confirm required-ci-gates publishes status contexts for merge-group SHAs

Edge cases:
- pr_metadata resolution when github.event.pull_request is absent
- Concurrency group uniqueness under merge_group events
- E2E label-gated jobs (gpu, kubernetes-ha) under merge_group context

This issue is well-specified and ready for implementation.

---

Labels: CI/GitHub Actions infrastructure feature request for merge queue enablement.

[ggallen]

/fs-code

🤖 Finished Code · ❌ Failure · Started 8:40 PM UTC · Completed 8:45 PM UTC
Commit: 4e6610d · View workflow run →

An open PR already addresses this issue — skipping automated implementation.

• chore(deps): bump tj-actions/changed-files from 42.1.0 to 47.0.6 #2 by @app/dependabot

To override, comment /fs-code --force on this issue.

_{Posted by fullsend pre-code check}

⚠️ Post-code script failed (exit code 1)

The code agent completed, but the post-code script failed while pushing the branch or creating the PR.

Workflow run: https://github.com/ggallen-sandbox/.fullsend/actions/runs/27718262923

Please check the workflow logs for details and retry with /fs-code if appropriate.