Lifetime problem with Xunit: InvalidOperationException: There is no currently active test.

[StefanBertels]

I get reproducable runtime error on one of two tests if I run all tests of the following test class / project. If I run just one test (e.g. manually in IDE Rider) everything looks good (no error).

Just guessing: maybe related to #657

Exception

System.InvalidOperationException: There is no currently active test.

System.InvalidOperationException
There is no currently active test.
   at Xunit.Sdk.TestOutputHelper.GuardInitialized() in /_/src/xunit.execution/Sdk/Frameworks/TestOutputHelper.cs:line 52
   at Xunit.Sdk.TestOutputHelper.QueueTestOutput(String output) in /_/src/xunit.execution/Sdk/Frameworks/TestOutputHelper.cs:line 61
   at Xunit.Sdk.TestOutputHelper.WriteLine(String message) in /_/src/xunit.execution/Sdk/Frameworks/TestOutputHelper.cs:line 109
   at TestProject1.UnitTest1.Test2() in TestProject1\UnitTest1.cs:line 18
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)

Here is my C# project:

UnitTest1.cs

public class UnitTest1(Xunit.Abstractions.ITestOutputHelper testOutputHelper)
{
    [FsCheck.Xunit.Property]
    public void Test1()
    {
        testOutputHelper.WriteLine($"{nameof(Test1)}");
    }

    [Xunit.Fact]
    public void Test2()
    {
        testOutputHelper.WriteLine($"{nameof(Test2)}");
    }
}

TestProject1.csproj

<Project Sdk="Microsoft.NET.Sdk">
    <PropertyGroup>
        <TargetFramework>net8.0</TargetFramework>
        <IsTestProject>true</IsTestProject>
    </PropertyGroup>
    <ItemGroup>
        <PackageReference Include="FsCheck.Xunit" Version="3.0.0" />
        <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.12.0" />
        <PackageReference Include="xunit" Version="2.9.3" />
        <PackageReference Include="xunit.runner.visualstudio" Version="3.0.1" />
    </ItemGroup>
</Project>

Reproducable by just running dotnet test in project folder.

No error if I downgrade to xunit 2.8.1 / FsCheck.Xunit 2.16.6.

<Project Sdk="Microsoft.NET.Sdk">
    <PropertyGroup>
        <TargetFramework>net8.0</TargetFramework>
        <IsTestProject>true</IsTestProject>
    </PropertyGroup>
    <ItemGroup>
        <PackageReference Include="FsCheck.Xunit" Version="2.16.6" />
        <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.12.0" />
        <PackageReference Include="xunit" Version="2.8.1" />
        <PackageReference Include="xunit.runner.visualstudio" Version="3.0.1" />
    </ItemGroup>
</Project>

[kurtschelfthout]

I can't reproduce this...

[arialdomartini]

I could replicate the issue. Please, find a minimal project to replicate it.

It actually throws either if run via Rider or via dotnet test.

[arialdomartini]

I am not sure it is related to #657: I have run code against a version of FsCheck where #657 is supposedly fixed (notice: PR #703 is not merged), but this did not fix the problem.

[arialdomartini]

A hint. Interestingly, the same tests do not fail if there is no dependency on ITestOutputHelper:

using Xunit;

public class UnitIssue692NoTestOutputHelper()
{
    [FsCheck.Xunit.Property]
    bool Test1()
    {
        return true;
    }

    [Fact]
    void Test2()
    {
        Assert.True(true);
    }
}

[arialdomartini]

Even more interestingly: the exception is raised only if the instance of ITestOutputHelper is actually used in the Fact.

The following tests are always green:

public class UnitTest1(Xunit.Abstractions.ITestOutputHelper testOutputHelper)
{
    [FsCheck.Xunit.Property]
    bool Test1()
    {
        return true;
    }

    [Fact]
    void Test2()
    {
        Assert.True(true);
    }
}

(Notice that ITestOutputHelper is injected, but not used).

The following are also green:

public class UnitTest1(Xunit.Abstractions.ITestOutputHelper testOutputHelper)
{
    [FsCheck.Xunit.Property]
    bool Test1()
    {
        testOutputHelper.WriteLine("hey");
        return true;
    }

    [Fact]
    void Test2()
    {
        Assert.True(true);
    }
}

They turn to red as soon as testOutputHelper is used in the xUnit test:

public class UnitTest1(Xunit.Abstractions.ITestOutputHelper testOutputHelper)
{
    [FsCheck.Xunit.Property]
    bool Test1()
    {
        return true;
    }

    [Fact]
    void Test2()
    {
        testOutputHelper.WriteLine("hey");
        Assert.True(true);
    }
}

[arialdomartini]

I am not sure which order xUnit uses (sometimes it seems it sort alphabetically the tests), and anyway Xunit.Extensions.Ordering does not seem to work with FsCheck. But, with the right order it is apparent that xUnit facts start failing after a property executed:

With the following:

public class UnitTest1(Xunit.Abstractions.ITestOutputHelper testOutputHelper)
{
    [Fact]
    void a()
    {
        testOutputHelper.WriteLine("first");
        Assert.True(true);
    }

    [FsCheck.Xunit.Property]
    bool b(int i)
    {
        return true;
    }

    [Fact]
    void c()
    {
        testOutputHelper.WriteLine("last");
        Assert.True(true);
    }
}

a and b pass; c fails.

It looks like the execution of a property leaves something wrong back. I can only imagine of a resource to be disposed of. I'm investigating.

[kurtschelfthout]

The testoutputhelper is suspicious. xunit has a bit of rough edges around handling it - see the comments in the code in the xUnit Property attribute in FsCheck.Xunit. Perhaps it's related to that.