Test "framework"

[ellnix]

Each part of the app has a ton of implicit dependencies on other parts of the app. Testing any particular module is a nightmare of stubbing and misusing and implicit dependencies on the state of the file system.

I propose we build a testing system that exposes sample testing data with a convenient interface and fast implementation. Here are some ideas:

• A high level src/test_helper.rs module (only compiled with cfg(test) that includes functions for:
  • Fetching test clashes
  • Making custom process::Commands for the aforementioned test clashes
• A collection of clash JSON files in the repo
• A tests/common/cli_helper.rs module that includes functions to set up the file system environment in a certain way to run integration tests directly on the CLI

[Andriamanitra]

Each part of the app has a ton of implicit dependencies on other parts of the app. Testing any particular module is a nightmare of stubbing and misusing and implicit dependencies on the state of the file system.

I think the first part of the solution is minimizing the number of these dependencies. If we have to mock everything the tests won't be super useful.

Fetching test clashes

Doing network requests in tests seems like a really bad idea. The tests need to be deterministic to be reliable (and they should work offline too).

A collection of clash JSON files in the repo

This is what I would prefer. A couple of clashes that are crafted to exercise as much functionality as possible for end-to-end tests should be good enough.

A tests/common/cli_helper.rs module that includes functions to set up the file system environment in a certain way to run integration tests directly on the CLI

Something like this would be useful for testing things like reading the configuration, but it needs to be containerized somehow – we don't want tests poking around on the actual file system when running cargo test on a dev machine. We could probably set up something in Github Actions?

[ellnix]

I think the first part of the solution is minimizing the number of these dependencies. If we have to mock everything the tests won't be super useful.

A lot of these dependencies are just the nature of the problem to be honest. The project itself has a lot of moving parts which cannot really be isolated from each other. Like the stub depending on clash and on the language configuration.

Doing network requests in tests seems like a really bad idea. The tests need to be deterministic to be reliable (and they should work offline too).

Oh absolutely, that was just bad wording from me, I meant fetching them from the repo itself and parsing them. The idea being you'd just have a test_helper::easy_clash(), test_helper::reverse_clash(), and other such methods to use specific fixtures.

Something like this would be useful for testing things like reading the configuration, but it needs to be containerized somehow – we don't want tests poking around on the actual file system when running cargo test on a dev machine. We could probably set up something in Github Actions?

I've not really used containers very much, but I can look into that. I think of all the points in this proposal this is the lowest priority though.