Making hegel-java usable as a library

[aphyr]

I'd like to write a Clojure client that uses this library, but there are some parts that make that hard.

1. Hegel used to return a map that told you if the test passed, how many cases were executed, number of cases, how many were interesting, whether health checks failed, etc. I can't figure out how to get that information any more. Could we have a structure returned by Hegel.test()? Bonus points if it includes (e.g.) a map of labels to generated values for the minimal failing case(s).

2. It's not clear what the Consumer passed to test() should do. I'm guessing that any non-exceptional return value is a valid test case. Is it OK to throw any exception? Any throwable (e.g. InterruptedException?) Do the exceptions specifically need to be JUnit ones? Is there special JUnit state that Hegel relies on, or is it OK to construct and throw the exceptions ourselves?

3. Does Hegel do anything with the fields of the assertions thrown, or is it just a pass/fail signal?

4. Hegel takes over printing to stderr/stdout, which fights with other logging/printing and does not play well with concurrency. Having a return value for TestCase might fix this, but you might also consider having a Reporter interface with callbacks for various Hegel events.

5. It would be very helpful to have something like testCase.isFinal(), so you could do your own debugging/logging on the final cases.

6. How should we signal spans?

[DRMacIver]

Hi @aphyr! Sorry for the delay in getting back to you, was on holiday and somewhat overwhelmed with notifications when I got back and missed this until just now.

Definitely agreed that hegel-java should be usable as a library. I think probably rather than trying to make the current hegel-java frontend usable in that way, we want a dev.hegel.lowlevel or something like that which exposes something closer to the libhegel C API (with whatever Java niceties are required) for people to use from other JVM libraries, as I think there's significant tension between providing a good frontend and providing a good library for people to build frontends. This should resolve (1), (4), (5), (6) for you.

2. Yes right now any throwable is marked as interesting. There's nothing JUnit specific here, there's some optional JUnit integration, but Hegel works without it. Possibly we should be a bit more fine grained about this, but I'm not 100% sure what the right semantics are here.
3. It looks at type and line number thrown by to distinguish exceptions that probably come from different types of bug.

[aphyr]

Oh that's tremendously helpful, thank you. And no rush, this is on my back burner too.

I don't know if you need to drop super far down--the generator library felt very nicely designed and was straightforward to adapt, (1) and (5) would be API compatible with the current Hegel.test(), and (4) could be provided as an option. I assume there's more cool stuff down in libhegel that might be nice, but those three things are, I think, the only blockers for hegel-clj. :-)

[Liam-DeVoe]

(added hegel-clj to the website! hegeldev/website#94)

[aphyr]

Er, just to be clear, hegel-clj doesn't really work right now--I'll need at least 1 and 4 to get that working. Might want to hold off on linking to it.

[Liam-DeVoe]

ah ok! I'll remove it, happy to add it back whenever you feel it's ready

[aphyr]

Yeah, just let me know when y'all have return values and an option to control the logging, and I'll get back to work on it. I think those are the big blockers.