Skip to content

ci: exclude faulty Phoenix node atl1-1-01-002-28-0 - #1865

Merged
sbryngelson merged 1 commit into
MFlowCode:masterfrom
sbryngelson:ci/exclude-bad-phoenix-node
Sep 13, 2026
Merged

ci: exclude faulty Phoenix node atl1-1-01-002-28-0#1865
sbryngelson merged 1 commit into
MFlowCode:masterfrom
sbryngelson:ci/exclude-bad-phoenix-node

Conversation

@sbryngelson

Copy link
Copy Markdown
Member

Description

Node atl1-1-01-002-28-0 on Phoenix is faulty. It caused two distinct CI failures, both landing on that same node:

  • Case-optimization (gpu-omp), run 34611616646: all 5 benchmarks (5eq_rk3_weno3_hllc, viscous_weno5_sgb_acoustic, hypo_hll, ibm, igr) died in pre_process with SIGILL ("Illegal instruction: illegal operand", exit 132) at the very first init routine (s_assign_default_values_to_user_inputs, m_global_parameters.fpp:322) — identical across unrelated cases.
  • Full Test Suite (gpu-acc), run 34673679046: 23 failures — tolerance mismatches plus "Pressure relaxation produced a non-physical phasic density" MPI_ABORT crashes.

master was intermittently green on the same commit lineage, so this is a bad node rather than a code regression. This PR adds the node to the static Phoenix GPU --exclude list. The runtime preflight doesn't catch it because the faults happen during the actual run, not preflight.

Type of change

  • Something else (CI/infrastructure)

Scope

  • This PR comprises a single, focused change (one node added to the exclude list).

How Has This Been Tested?

CI-config only; no code paths changed. The existing $node_exclude render (#SBATCH --exclude=...) is unchanged in mechanism.

Checklist

  • I ran ./mfc.sh format on modified files — N/A (shell/CI script, no Fortran/Python source changed)
  • This PR does not change CFD results

Done with Claude Code.

Node atl1-1-01-002-28-0 produced illegal-instruction (SIGILL) faults in
pre_process across all case-optimization benchmarks (run 34611616646) and
correlated numerical failures (tolerance mismatches + non-physical phasic
density MPI_ABORT) in the full test suite (run 34673679046). Both jobs
landed on this node; master was intermittently green on the same commit,
indicating a bad node rather than a code regression. Add it to the static
Phoenix GPU exclude list.

Done with Claude Code.
Copilot AI lite review requested due to automatic review settings September 12, 2026 17:02

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Pull request overview

Updates Phoenix GPU CI job submission to avoid scheduling on a known-faulty node that is causing intermittent CI failures.

Changes:

  • Adds atl1-1-01-002-28-0 to the static Phoenix GPU --exclude node list.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread .github/scripts/submit-slurm-job.sh
@codecov

codecov Bot commented Sep 12, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 61.26%. Comparing base (dc0aec1) to head (aaca082).

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #1865   +/-   ##
=======================================
  Coverage   61.26%   61.26%           
=======================================
  Files          84       84           
  Lines       22330    22330           
  Branches     3265     3265           
=======================================
  Hits        13680    13680           
  Misses       6207     6207           
  Partials     2443     2443           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@sbryngelson
sbryngelson merged commit 03d5bbf into MFlowCode:master Sep 13, 2026
86 of 87 checks passed
@sbryngelson
sbryngelson deleted the ci/exclude-bad-phoenix-node branch September 13, 2026 00:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants