`package:test` should define an exception that is equivalent to skipping the test

[brianquinlan]

I'm asking for the equivalent of the Python SkipTest exception.

Sometime it is difficult to calculate whether the test can be completed successfully outside of the test itself. This provides a mechanism to bail out of the test at any point.

[jakemac53]

What is the expected behavior for setUp and tearDown if you throw this exception? We can't really stop setUp from running in this case since we don't know the test will be skipped until its executing, and that makes me think we should always run tearDown as well since we don't know how much of the setup completed and there may be cleanup work needed. This could result in errors though if late variables haven't yet been assigned or contain a value from a previous test still.

Also, should you be able to throw this exception from within setUp and tearDown?

[brianquinlan]

In Python, raising SkipTest in setUp skips the corresponding test (or group). This use case is explicitly documented as being useful if a needed resource wasn't available during setUp.

[jakemac53]

In Python, raising SkipTest in setUp skips the corresponding test (or group). This use case is explicitly documented as being useful if a needed resource wasn't available during setUp.

Does it still run tearDown though?

[brianquinlan]

In Python, raising SkipTest in setUp skips the corresponding test (or group). This use case is explicitly documented as being useful if a needed resource wasn't available during setUp.

Does it still run tearDown though?

No.

[jakemac53]

Does it still run tearDown though?

No.

Hmm ok, we will need to evaluate whether we think that is the correct behavior. It isn't the behavior I would intuitively expect personally. Just to elaborate, given the following example, the client will never be shut down when it should be:

void main() {
  MyClient? client;
  setUp(() async {
    client = await MyClient.start();
    if (!client.ping()) {
      throw SkipTestException('backend is down, skipping test');
    }
  });

  test('mytest', () async {
     // The actual test, if I did a skip here I would also want tearDown to run though?
  });

  tearDown(() async {
    client?.shutdown();
  });
}

[brianquinlan]

Do we run tearDown now if setUp throws?

[jakemac53]

Do we run tearDown now if setUp throws?

Yes we do

[brianquinlan]

Do we run tearDown now if setUp throws?

Yes we do

Then I'd do the same in this case, i.e., call tearDown if setUp throws.

[jakemac53]

Ok, so then I think the main work to do this would be creating a special kind of exception that indicates a skip, we could either expose this directly or just expose a method called skipTest which throws it (analogous to our current fail function).

When we handle exceptions thrown by tests, instead of always marking the test as failed we would first check if it is one of these special exceptions and if so only mark the test as skipped and not failed.

This would only affect the reporting of the test status and nothing else.

[brianquinlan]

Ok, so then I think the main work to do this would be creating a special kind of exception that indicates a skip, we could either expose this directly or just expose a method called skipTest which throws it (analogous to our current fail function).

Python has the same method with the same spelling ;-)

When we handle exceptions thrown by tests, instead of always marking the test as failed we would first check if it is one of these special exceptions and if so only mark the test as skipped and not failed.

This would only affect the reporting of the test status and nothing else.

That would be amazing!

[jakemac53]

cc @natebosch for thoughts