CI: investigate conda-cpp-tests jobs hitting the 40m timeout

[rgsl888prabhu]

Context

The `conda-cpp-tests` job in the `pr` workflow has been hitting the outer 40-minute `timeout` wrapper around `./ci/run_ctests.sh` (in `ci/test_cpp.sh`) and failing CI as a result.

Example failing run (PR #1035):

• Run: https://github.com/NVIDIA/cuopt/actions/runs/24721054667/job/72360181090?pr=1035
• Matrix: CUDA 13.1.1, Python 3.14, amd64, Ubuntu 24.04, H100, latest driver, latest deps
• Duration: 2026-04-21T16:55:08Z → 2026-04-21T17:40:09Z (~45 min wall-clock, killed by the outer `timeout 40m`)
• Conclusion: failure
• Runner: `609e-l-amd-g-h100-l-1-9rdw2-runner-v49qn`

As a short-term unblock, PR #1130 bumps the outer timeout from 40m to 50m. This issue tracks the underlying investigation — we should understand why the job is now running longer, not just raise the ceiling indefinitely.

Investigation items

• Break down the ctest run-time per test binary (e.g. `ctest --output-on-failure -V --timeout N` or parse the `Test.xml`) to identify which suites have regressed.
• Compare wall-clock of `conda-cpp-tests` on `main` over the last ~30 days — is this a recent regression or a gradual drift?
• Check whether slow tests are concentrated in one area (MILP, LP, routing GES, grpc integration) so we can target the fix.
• Confirm the regression is not specific to one runner pool or GPU (H100 vs. others in the matrix).
• Decide: (a) reduce test runtime at the source, (b) split into parallel jobs, (c) gate slow tests behind a nightly job, or (d) accept a permanently higher ceiling.

Acceptance criteria

• Root cause is documented on this issue.
• Either the offending tests are sped up / skipped on PRs, or a CI topology change is proposed and merged, such that the PR `conda-cpp-tests` job consistently finishes well under the outer timeout.

Related

• ci: bump C++ test timeout from 40m to 50m #1130 — bump the outer timeout from 40m to 50m as a stop-gap

[github-actions]

🔔 Hi @anandhkb, this issue has had no activity for 21 days. Please update or let us know if it can be closed. Thank you!

If this is an "epic" issue, then please add the "epic" label to this issue.
If it is a PR and not ready for review, then please convert this to draft.
If you just want to switch off this notification, then use the "skip inactivity reminder" label.