tests: Move validation tests to the haskell test suite

[croyzor]

Here's how tests appear when the validator is not installed:

      execution
        test_pba:                               OK
        validate(test_pba):                     SKIPPED
          hugr_validator not installed

they are then treated as failures in the test suite summary, unless the option --ignore-validation is passed. Then, they look the same but aren't marked as failures (and "hugr_validator not installed" is not in red)

[acl-cqc]

Ok I love what you have achieved here but I'm not quite sure I agree with the means to the end ;-). Is there not a simpler way to achieve this without the bits you have marked BAD, near-repetition of HugrTest, etc., roughly as follows:

Define a new ValidateTest - like your HugrTest, but not containing a TestTree; instead containing an IO ByteString, a FilePath, and a TestName. This is instance IsTest ValidateTest so that (just like your instance IsTests) we can get at the options (you've done this neatly, let's keep that). This implementation "run"s a newly-constructed Test with that TestName that...

• invokes the IO ByteString to get a Hugr as bytes
• Writes that to the FilePath
• Invokes the validator on that FilePath

If you really want, it could "run" a sequentialTest that makes the Hugr and writes it as first step ("compile"), then sequences that with a "validate" step, although I note sequentialTest wasn't really doing anything for me...

if invoking the validator works then we don't care about the options, but if validation fails then we can check the options and only fail if the options say validation required (which is default).

This does avoid checking the validator is installed; if it's not, and you run the testsuite without disabling validation, you'll still get lots of failures (as now right?), the difference is it'll try (and fail) to run the validator 100 times rather than failing to determine its version only once. So less efficient, but in that scenario you should be Ctrl-C-ing out of that and rerunning with --ignore-validation in any case. (IOW: I think checkValidatorInPath adds more complexity than it's worth.)

I can have a go at this if you like....

[acl-cqc]

This seems fairly clear (True = ignore validation, false = validate) but maybe an enum would be even clearer as that is a bit of a double negative

[acl-cqc]

just repeating that with some inlining, that's:

if s == "ignore-validation"
then Just (IgnoreValidation False)
else Nothing

never mind the "else Nothing", but...are you sure??

[croyzor]

🙃

[acl-cqc]

Suggested change

	if exitCode == ExitSuccess
	pure (exitCode == ExitSuccess and "hugr_validator 0." `isPrefixOf` output)

[acl-cqc]

this "interpreterInPath" is really, "validatorAvailable" right...

[croyzor]

(IOW: I think checkValidatorInPath adds more complexity than it's worth.)

It might change your opinion to know that the animus for this change was that I want an easier way to transition from json validation to hugr model validation. I intend for the next version of hugr-validator to consume the model, and the test suite to check that we have >= 0.5.0 and complain if the validator is the wrong version. In principle this could apply to any breaking change to the output format/validator