The native JUnit tests cannot be executed without the prior run of JVM tests

[vjovanov]

Is your feature request related to a problem? Please describe.

In projects where test suites run for hours, or days, it is unrealistic to demand that native testing is preceded with the execution on the JVM.

Additionally, the developer experience is not ideal: when testing locally, one needs to wait for both tests to see the result of their work.

Describe the solution you'd like
We need to modify JUnit in such a way that test discovery can be executed without executing the actual tests. That way we could execute the discovery at build time from the Native Build Tools plugin.

[melix]

Some context on why this is done that way:

• the native tests need the list of (internal) JUnit test ids
• these test ids are only known when running the tests (as far as I know, it is possible for a test suite to generate new tests at runtime, so the list of test ids cannot be statically determined)
• therefore we do a run of the JVM tests before running the native tests

Ideally, we'd want to have some kind of "dry run" of the JUnit tests so that we can determine the list of test ids without actually running the tests. There must be something available, since for example Develocity's distributed test feature probably needs something similar, but I'm not sure (let's check with s/o from the JUnit team).

There's also an implementation trouble: say we want to run a particular native test from CLI, e.g running something like ./gradlew nativeTest --tests com.foo.MyAwesomeTest. Then the list of test ids to be generated should be filtered. We need the list of test ids which correspond to that particular test/test filter. Today this works because we implicitly run the JVM tests before, but if we want to remove that dependency, then we need some kind of API/tooling which would run beforehand, with the appropriate classpath etc.

re "when testing locally, one needs to wait for both tests to see the result of their work", I would argue that today it would be fairly rare for someone to run the native tests locally, users most likely rely on CI feedback.