Enable dart test to shard on files, not individual tests

[christerswahn]

We have a rather large test suite and most tests have a relatively time consuming setup and teardown. We have implemented sharding with dart test, which helps, but not at all to the degree we expected.

It appears that the sharding cuts across the files, i.e. tests from the same test file occur in more than one shard. When splitting the work like this, the same setUpAll and tearDownAll chains need to be run multiple times. This is very inefficient.

We would like a new flag for dart test to make the shard splitter split on files instead of tests.

This has an additional benefit. Often a bug will cause a cluster of test cases to fail, and these are often in the same file, which would mean that they would be reported in the same shard's output and thus easier to get an overview of.

[jakemac53]

The best way to do this today is to organize your files into a structure matching the shards your want and then run those test directories individually, or to use tags in order to tag test suites to run in certain shards and then run those specific tags.

Long term this gives you more control over the shards, so that you can keep them all at roughly the same total running time by watching your CI jobs etc. The built in sharding is random and so individual shard runtime is also random and rarely results in an even distribution of tests (in terms of actual running time).

I do agree that the way things are currently sharded is pretty odd and doesn't work well when combined with setUpAll/tearDownAll in particular. But I also don't know that it is something we would be able to reasonably change at this point.

Splitting based on test suites is even less likely to result in good shards because test suites vary in size a lot.

[christerswahn]

Splitting based on test suites is even less likely to result in good shards because test suites vary in size a lot.

If the number of files is much larger that the number of shards, statistically this will often even out.

Users could shard test manually in test file groups, but this seems a quite reasonable feature to have built-in in the dart test command, especially considering its way-of-working with setup etc.

[jakemac53]

If the number of files is much larger that the number of shards, statistically this will often even out.

I would actually argue this is not true - because what actually matters here is the odds of having even one single slow shard among all your shards. Basically it's a form of the birthday problem. Your total CI time is equal to the slowest shard, so just one bad one can significantly skew the overall time you are waiting for CI.

[jakemac53]

Another point in favor of splitting on test suite boundaries would be that for most unit tests, the loading phase is actually longer than the running phase (takes longer to compile than to run). By splitting at test boundaries we incur the loading phase in multiple shards. This is similar to the long setUpAll/tearDownAll issue.

[jakemac53]

Ultimately I don't think I personally would be opposed to some sort of "sharding strategy" option, which at least allows opting into this form of sharding. But I don't know where it would fit in terms of priorities for this package.