include test results

[AlexanderLanin]

just copy paste of discussion for now. open point: parse xml from a sphinx-extension or generate needs.json from xml beforehand.

Parse test code

Pro:

• No need to execute the tests
• Possible to integrate it into custom LSP to get IDE Diagnostics (not very much usable benefit)

Con:

• Python specific: It's difficult to detect skipped tests reliably, as there are so many ways to skip tests in Python.
• Do not get test results (without executing tests which negates the first pro)
• Language specific parsing required
• No detection if test was ever executed (same as with links into source code!!)

What exactly we get:

• File and Line number of test cases ✔️
• parameters like TestType and DerivationTechnique, links to requirements, etc. ✔️
• skipped tests 🟡 (not reliably)
• test results (passed, failed) ❌

Parse test results (XML)

Pro:

• One parser for all (supported) languages, due to JunitXML standard

Problems:

• Need to run tests to get files to parse

What exactly we get:

• File and Line number of test cases 🟡 (python: not native, but extendable)
• parameters like TestType and DerivationTechnique, links to requirements, etc. 🟡 (not for skipped/disabled tests)
• skipped tests ✔️
• test results (passed, failed) ✔️

[MaximilianSoerenPollak]

Not yet all in there, but a bit further refined:

­Parse Test Results from XML with Source Code Linking

­ Goal

Implement a test result parser that extracts test information from JUnit XML files and links test cases back to their source code locations for traceability and documentation purposes.

---

Current Situation

Currently we have no way of automatically interpreting / parsing test results, nor pairing it up to requirements.

---

­ Solution

Implement a JUnit XML parser that extracts test case information.

Approach:

• Parse JUnit XML files to extract:
  • Test case names and results (passed/failed/skipped)
  • Test parameters and metadata where available
  • Test execution status

Expected outcome:

• Language-agnostic test parsing via JUnit XML standard
• Reliable detection of skipped tests and test results
• Traceability from test results back to source code
• Foundation for test documentation and requirement linking

---

­ Concrete Tasks

• Implement JUnit XML parser:
  • Parse basic test case information (name, status, duration)
  • Extract test results (passed, failed, skipped)
  • Handle test parameters and custom properties
  • Gather all parsed information in a 'workable' format.
• Link tests (file & lineNr) to the appropriate requirement
• Document usage and configuration options

---

­ Considered Alternatives

Alternative 1: Parse test source code directly

Pros:

• No need to execute tests to get information
• Could integrate into custom LSP for IDE diagnostics
• Direct access to test parameters and metadata

Cons:

• Language-specific parsing required for each supported language
• Difficult to reliably detect skipped tests (many ways to skip in Python)
• No test execution results without running tests
• Cannot detect if tests were actually executed

Alternative 2: Use XML parsing only (chosen approach)

Pros:

• Language-agnostic via JUnit XML standard
• Reliable detection of skipped tests and results
• Standardized format across different test frameworks

Cons:

• Requires test execution to generate XML files
  - File/line number information not native to XML format (Can get this integrated via properties)
• Limited metadata for skipped/disabled tests

---

­ Notes

• XML parsing provides reliable test results and skip detection
• Consider caching source code mapping to improve performance

---

­References

• JUnit XML Format Specification
• pytest JUnit XML documentation
• XML parsing libraries documentation

[MaximilianSoerenPollak]

With #197 This is now closed.