[Test] Comparison of Rust test runners: cargo test vs cargo-nextest vs other (JUnit XML output support)

[dcalavrezo-qorix]

We need to evaluate and compare Rust test runners, particularly focusing on the ability to generate JUnit XML format reports for integration with CI systems.

Context:

• We currently use cargo test for running Rust tests.

• There is interest in cargo-nextest, a next-generation test runner, which claims to offer enhanced features and JUnit XML output.

• The main goal is to determine which tool is the best fit for our workflows, especially considering requirements for JUnit XML reports and any implications for tool qualification or certification.

Tasks:

Compare the features of cargo test and cargo-nextest, with emphasis on:

• JUnit XML output support and compatibility

• CI/CD integration ( GitHub Actions)

• Performance and reliability

• Any considerations for qualification/certification in regulated environments

• Try out cargo-nextest in one of our Rust projects and generate a sample JUnit XML report.

• Summarize pros/cons and recommend the best approach for our needs.

[PiotrKorkus]

Please find analysis done by @arkjedrz some time ago:
qorix-group/testing_tools#45

[a-zw]

Alternative proposal: Adapt our process that unit tests shall always be green. Then there is no point in using any test reports.

[PiotrKorkus]

Actually test reports might be helpful, I would be happy if they were not needed but there are some cases:

• traceability - virtual sphinx needs for tests are created from xml test report with a script introduced in: XML Test parsing & testlinks docs-as-code#197 and Add pytest decorator plugin tooling#65
• as an option to make sure that tests were indeed executed, when someone want's to merge to SCORE - some discussions we had together with @AlexanderLanin

[dcalavrezo-qorix]

Hi @dcalavrezo-qorix,

Is this issue still relevant? If not, please add the trash label so it can be marked for deletion.

[PiotrKorkus]

It was decided to stick to built-in cargo test runner together with bazel