Non-deterministic tests (inverse of fragile)

[hasufell]

There are some tests that occasionally seem to fail due to real issues. In that sense they're the inverse of "fragile": they sometimes show false-positive results.

This has happened recently with these:

• dynbrk002: https://github.com/stable-haskell/ghc/actions/runs/20055419160/job/57537549583#step:10:101
• T16700: https://github.com/stable-haskell/ghc/actions/runs/20055419160/job/57525357752
• T12035j: https://github.com/stable-haskell/ghc/actions/runs/20016330211/job/57397959662
• T7386: https://github.com/stable-haskell/ghc/actions/runs/20086631166/job/57629609123

For these tests we should add something to the test driver so that we can repeat them a couple of times.

[angerman]

So:

• fragile means: fails every once in a while, and we think this is ok?
• nondet means: this most of the time succeeds, but we've observed it occasionally fail?

I'm still super confused about the point of having fragile tests :-)

Ideally we'd want to be able to run each test 100+ times and always get the same output.

I'm a bit concerned about nondet potentially being non-determanistic due to somehow building a broken compiler, which we then use to build that test? E.g. the test is only broken with that specific compiler?

[hasufell]

I'm a bit concerned about nondet potentially being non-determanistic due to somehow building a broken compiler, which we then use to build that test? E.g. the test is only broken with that specific compiler?

Yes? That's the entire point of the test to find a broken compiler.

[angerman]

But if the test is only broken because the compiler is broken, re-running the test 3 or more times will always result in deterministically broken tests.

My concern is that one specific ghc compiler build results in one specific test being broken. Now if we rebuild the compiler in ci, the same test would pass. We'd basically have baked in some compilation defect into the compiler.

[hasufell]

But if the test is only broken because the compiler is broken, re-running the test 3 or more times will always result in deterministically broken tests.

I don't follow. What is broken? A broken test means there is a bug in the test (well, that sucks). A broken compiler means the test may fail as a result. Compilation of the test is part of the test.

My concern is that one specific ghc compiler build results in one specific test being broken. Now if we rebuild the compiler in ci, the same test would pass.

What? Why?

[angerman]

Because the compiler itself has a non-deterministic bug when building itself that surfaces only for this one test.

[angerman]

In any case if we notice tests that fail every once in a while, we should just make sure we run them N times to see if they are failing non-deterministically.

[hasufell]

Because the compiler itself has a non-deterministic bug when building itself that surfaces only for this one test.

Then we're cooked anyway.