ci: exclude faulty Phoenix node atl1-1-01-002-28-0 - #1865
Conversation
Node atl1-1-01-002-28-0 produced illegal-instruction (SIGILL) faults in pre_process across all case-optimization benchmarks (run 34611616646) and correlated numerical failures (tolerance mismatches + non-physical phasic density MPI_ABORT) in the full test suite (run 34673679046). Both jobs landed on this node; master was intermittently green on the same commit, indicating a bad node rather than a code regression. Add it to the static Phoenix GPU exclude list. Done with Claude Code.
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Pull request overview
Updates Phoenix GPU CI job submission to avoid scheduling on a known-faulty node that is causing intermittent CI failures.
Changes:
- Adds
atl1-1-01-002-28-0to the static Phoenix GPU--excludenode list.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1865 +/- ##
=======================================
Coverage 61.26% 61.26%
=======================================
Files 84 84
Lines 22330 22330
Branches 3265 3265
=======================================
Hits 13680 13680
Misses 6207 6207
Partials 2443 2443 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Description
Node
atl1-1-01-002-28-0on Phoenix is faulty. It caused two distinct CI failures, both landing on that same node:5eq_rk3_weno3_hllc,viscous_weno5_sgb_acoustic,hypo_hll,ibm,igr) died inpre_processwith SIGILL ("Illegal instruction: illegal operand", exit 132) at the very first init routine (s_assign_default_values_to_user_inputs,m_global_parameters.fpp:322) — identical across unrelated cases.MPI_ABORTcrashes.masterwas intermittently green on the same commit lineage, so this is a bad node rather than a code regression. This PR adds the node to the static Phoenix GPU--excludelist. The runtime preflight doesn't catch it because the faults happen during the actual run, not preflight.Type of change
Scope
How Has This Been Tested?
CI-config only; no code paths changed. The existing
$node_excluderender (#SBATCH --exclude=...) is unchanged in mechanism.Checklist
./mfc.sh formaton modified files — N/A (shell/CI script, no Fortran/Python source changed)Done with Claude Code.