Caused a red run on xmsgrid today. One-flag fix, and it removes an external dependency CI does not use.
Symptom
Visual Studio 17 (Release, 3.13, Windows) failed with four configurations dying on:
ERROR: Permission denied for user: 'None': 403: Forbidden. [Remote: conancenter]
Nothing else failed — no compile error, no test failure (100% tests passed, 0 tests failed out of 1). The identical commit passed in the pull_request run minutes earlier, and every other leg of the same matrix passed. Two runs of 15 jobs each hitting conancenter simultaneously is a plausible throttle.
The four failing configurations were all pybind=False — ABI-independent, so this had nothing to do with the Python version.
Cause
github-ci.yaml.jinja passes --remove-conancenter on mac only:
| job |
conan setup |
| mac |
xmsconan_conan_setup --remote-url … --login --remove-conancenter |
| linux |
xmsconan_conan_setup --remote-url … --login |
| windows |
xmsconan_conan_setup --remote-url … --login |
So linux and windows leave conancenter in the remote list, and Conan consults it during resolution. When it answers 403, the build fails on a registry that supplies nothing.
Why removing it is safe
Every dependency resolves from the Aquaveo remote — boost/1.86.0, zlib/1.3.1, cxxtest/4.4, pybind11/3.0.1. The mac job already removes conancenter and builds the same matrix successfully on every release, which is the proof: nothing in the graph needs conancenter.
Suggested fix
Add --remove-conancenter to the linux and windows setup steps, matching mac. Three template lines, and it makes all three platforms consistent.
Worth checking the GitLab template for the same asymmetry while in there.
Why it matters beyond one flake
An external registry in the resolution path means every CI run in all seven repos can fail for reasons outside Aquaveo's control, on a dependency set that never comes from there. The 3.14 fan-out roughly doubled the job count per repo, so the exposure roughly doubled too — this flake surfaced on the first repo after that change, which is unlikely to be a coincidence.
Caused a red run on xmsgrid today. One-flag fix, and it removes an external dependency CI does not use.
Symptom
Visual Studio 17 (Release, 3.13, Windows)failed with four configurations dying on:Nothing else failed — no compile error, no test failure (
100% tests passed, 0 tests failed out of 1). The identical commit passed in thepull_requestrun minutes earlier, and every other leg of the same matrix passed. Two runs of 15 jobs each hitting conancenter simultaneously is a plausible throttle.The four failing configurations were all
pybind=False— ABI-independent, so this had nothing to do with the Python version.Cause
github-ci.yaml.jinjapasses--remove-conancenteron mac only:xmsconan_conan_setup --remote-url … --login --remove-conancenterxmsconan_conan_setup --remote-url … --loginxmsconan_conan_setup --remote-url … --loginSo linux and windows leave
conancenterin the remote list, and Conan consults it during resolution. When it answers 403, the build fails on a registry that supplies nothing.Why removing it is safe
Every dependency resolves from the Aquaveo remote —
boost/1.86.0,zlib/1.3.1,cxxtest/4.4,pybind11/3.0.1. The mac job already removes conancenter and builds the same matrix successfully on every release, which is the proof: nothing in the graph needs conancenter.Suggested fix
Add
--remove-conancenterto the linux and windows setup steps, matching mac. Three template lines, and it makes all three platforms consistent.Worth checking the GitLab template for the same asymmetry while in there.
Why it matters beyond one flake
An external registry in the resolution path means every CI run in all seven repos can fail for reasons outside Aquaveo's control, on a dependency set that never comes from there. The 3.14 fan-out roughly doubled the job count per repo, so the exposure roughly doubled too — this flake surfaced on the first repo after that change, which is unlikely to be a coincidence.