tasty-1.5 release coordination

[Bodigrim]

It's been almost 3 years since tasty-1.4 release, and we accumulated plenty of contributions waiting for a major release. It would be great to release tasty-1.5 soon, but before we proceed with an RC, I'd like to issue the last call for contributions. Ideally once tasty-1.5 is out, we'll refrain from releasing breaking changes for another 2-3 years, so if there is anything you'd like to change let's change it now.

CC @brandonchinn178 @gwils @nomeata @ocharles @kquick @jstolarek @mitchellwrosen @newhoggy @tarleb @barrucadu @414owen @Shimuuar @frasertweedale on behalf of various tasty-* packages.

[brandonchinn178]

I'd like to re-propose two suggestions I made that were put on hold for including breaking changes. I think adding --show-all-options and a TestMeta structure would make tasty nicer/easier to extend. Could we revive the discussion in those issues on the merits of the proposal, now that breaking changes are less of a concern?

#342 #359

EDIT: It would also be great to add a time frame to the initial post, so people have an idea when to submit "last calls" by.

[frasertweedale]

No requests on part of tasty-tmux.

[Bodigrim]

@brandonchinn178 class IsTest remains as is since tasty-0.1, I'm very reluctant to change it. But breaking Ingredients is on the table, we've changed their interface already. We still aim to minimise breakage however.

It would also be great to add a time frame to the initial post, so people have an idea when to submit "last calls" by.

It depends on the activity in this thread. By default I'd wait for responses up to mid August. If someone has an important suggestion, but off for vacation and cannot contribute immediately, we can wait a little longer.

In the meantime you can test your packages against tasty HEAD with this cabal.project:

packages: .

source-repository-package
  type: git
  location: https://github.com/UnkindPartition/tasty
  subdir: core

source-repository-package
  type: git
  location: https://github.com/UnkindPartition/tasty
  subdir: hunit

source-repository-package
  type: git
  location: https://github.com/UnkindPartition/tasty
  subdir: quickcheck

source-repository-package
  type: git
  location: https://github.com/UnkindPartition/tasty
  subdir: smallcheck

source-repository-package
  type: git
  location: https://github.com/Bodigrim/tasty-bench

source-repository-package
  type: git
  location: https://github.com/ocharles/tasty-rerun

[barrucadu]

No special requests from me

[ocharles]

I don't believe any of my libraries need anything, so no requests here either

[Shimuuar]

There's no way to add arbitrary data to Result. tasty-bench and tasty-papi resort to hacks like hiding data in resultDescription and manipulating it. I propose adding field resultExtraData :: [Dynamic]. This way downstream libraries could easily add any data they like to test result.

It is breaking change but it seems that impact should be limited since preferred way of construction Result is testPassed and similar function.

[Bodigrim]

@Shimuuar I like the idea, but I wonder whether extending tasty for the purposes of tasty-bench / tasty-papi is a right thing to do. One might argue that the latter are abusing tasty. OTOH I recall @Kleidukos asking for the same functionality, an ability to store data after test execution, but I don't recall for what purpose.

[Kleidukos]

The idea was to store data produced by the test in order to render it, for later exploration.
So, each test would get a TVar to a data structure that would get enhanced with the results of each test, and this data structure would be rendered in a nice format like HTML so that it can be visualised later.

[Shimuuar]

@Bodigrim It could be abuse but that's what happens when library is released to public. User always have ideas. And building tree of things to do executing them, collating reports doesn't seem to be too far off. In any case it would be much better if we could abuse it without horrible hacks. Also such addition seems to be generally useful

[kquick]

Thank you for checking in, @Bodigrim .

The tasty-checklist and tasty-sugar packages are both fine with the new tasty candidate and I don't foresee any benefits for them to changes in tasty itself.

The tasty-expected-failure package also works fine with the new candidate, but it does contain this workaround https://github.com/nomeata/tasty-expected-failure/blob/master/Test/Tasty/ExpectedFailure.hs#L54-L60 (see also the note starting line 66), which has further documentation in nomeata/tasty-expected-failure#24. The summary is that the test might fail, but the subsequent Show of the test results might differently fail (via exception) and that failure occurs outside of the WrappedTest handling. If the Show of the results was forced within tasty prior to exiting the WrappedText context then the workaround noted above would no longer be necessary.

[Bodigrim]

And building tree of things to do executing them, collating reports doesn't seem to be too far off.

@Shimuuar Yeah, I often use tasty as a general task manager. If you are interested enough, feel free to open an issue to discuss design. Not sure it will make it into tasty-1.5, this is pretty big and I'd like a consensus from other maintainers that it's a right thing to do, but at least we get started.

In any case it would be much better if we could abuse it without horrible hacks.

Playing devil's advocate, is Dynamic and fromDynamic massively better than String and readMaybe?..

If the Show of the results was forced within tasty prior to exiting the WrappedText context then the workaround noted above would no longer be necessary.

@kquick I must admit I'm generally illiterate about Haskell exceptions. Could you possibly raise a PR to discuss it further? I imagine the patch is not massive and it could be easier to discuss a patch than its verbal description, but if I'm wrong feel free to raise an issue instead.

[Shimuuar]

Playing devil's advocate, is Dynamic and fromDynamic massively better than String and readMaybe?..

It is! Maybe not massively. 1) it doesn't break existing printers of results. 2) It actually composes so it's possible to mix tests from different libraries which both want to add some data to Result.

I think it's easier to discuss concrete code, so I made draft PR #381

[Bodigrim]

Unless there are specific objections, I'll cut a release candidate next week.

[kquick]

Apologies for not responding to your request for a patch: got swamped and forgot. Looking into this now, it appears that the concern was addressed in fcfccc8 , so that addresses my stated concern and thumbs-up for a release.