NGEN Hanging on Error with MPI (NGWPC-10757)

[idtodd]

When running NGEN with MPI, the program would hang when an uncaught python error was expected to abort the simulation. This seems to be from rethrowing the pybind11::error_already_set exception after shutting down the interpreter. A separate catch for python exceptions is added that will copy the error text into a std::runtime_exception and throw that after shutting down the interpreter.

Additions

• Catch for run_ngen that allows copying the python error text into a C++ object before throwing a new exception. This prevents trying to interact with the python Exception object after the interpreter has been shut down.

Removals

Changes

Testing

1. MPI run using RTE. T-route's update method was modified to immediately throw an exception. The exception was logged to stderr and the run (all MPI processes) was aborted.

Screenshots

Notes

Todos

Checklist

• PR has an informative and human-readable title
• Changes are limited to a single goal (no scope creep)
• Code can be automatically merged (no conflicts)
• Code follows project standards (link if applicable)
• Passes all existing automated tests
• Any change in functionality is tested
• New functions are documented (with a description, list of inputs, and expected output)
• Placeholder code is flagged / future todos are captured in comments
• Project documentation has been updated (including the "Unreleased" section of the CHANGELOG)
• Reviewers requested with the Reviewers tool ➡️

Testing checklist (automated report can be put here)

Target Environment support

• Linux

[jswade-rtx]

I tested and confirmed that this branch fixes the issue with Ngen hanging on a pybind:error_already_set error.