Problem
The only way to test for bugs or invalid behavior in comparison with upstream is by manually testing and listening to MIDI files.
Description
I already suggested it in #1 but i'd like to explain it further here.
So, essentially the idea is similar to what i suggested over at fluidsynth:
FluidSynth/fluidsynth#1747
And it was implemented here:
FluidSynth/fluidsynth#1761
The idea is to use GH actions to compare every commit on master to behavior with upstream.
For now I mean only comparing the synth as it's the most important component of the library, but it could be expanded with things like analyze_midi_data and comparing the text output.
Steps for the workflow would be like this:
- Init both sharp and upstream libraries
- get the test MIDIs
- get the test sound bank (I would suggest either GM.DLS or GeneralUserGS and given that the latter is more complex i think it's better for automated tests)
- For all test MIDIS:
- Render
examples/midi_to_wav_node on upstream
- Render WAV on sharp
- Compare the output (fluidsynth uses
sox util, so that can be used). Exact byte match probably won't be achieved so maybe -80dB of difference?
- If the files are different enough, add to fail counter and upload the WAV renderings as artifacts (for manual inspection)
- If any fails, fail the workflow (you get a warning that a workflow failed). And all files that didn't match are uploaded so you can test them manually!
The biggest advantage here is that since it's GH actions, it runs on the runner for as long as it needs to, so the testing suite can be quite large. Essentially push and forget. If it fails, it'll let you know.
My recommended MIDIs:
Overall, i think that this is a very nice and hassle-free solution for ensuring that SpessaSharp's synth matches upstream in pretty much all cases.
What do you think?
Alternatives
Alternative way of checking the difference would be to invert one wav rendering and mix them both down. Then check if the loudest sample is above some threshold (like the previously mentioned -80dB). This solution doesn't require sox so it can be programmed in manually.
Additional info
Sorry for the wall of text :-]
Problem
The only way to test for bugs or invalid behavior in comparison with upstream is by manually testing and listening to MIDI files.
Description
I already suggested it in #1 but i'd like to explain it further here.
So, essentially the idea is similar to what i suggested over at fluidsynth:
FluidSynth/fluidsynth#1747
And it was implemented here:
FluidSynth/fluidsynth#1761
The idea is to use GH actions to compare every commit on master to behavior with upstream.
For now I mean only comparing the synth as it's the most important component of the library, but it could be expanded with things like analyze_midi_data and comparing the text output.
Steps for the workflow would be like this:
examples/midi_to_wav_nodeon upstreamsoxutil, so that can be used). Exact byte match probably won't be achieved so maybe -80dB of difference?The biggest advantage here is that since it's GH actions, it runs on the runner for as long as it needs to, so the testing suite can be quite large. Essentially push and forget. If it fails, it'll let you know.
My recommended MIDIs:
npm run test:midi(tests/midi_file/generated)th07_19_user.midcomes from there (it's not named like that). But it includes VERY complex MIDI files.Overall, i think that this is a very nice and hassle-free solution for ensuring that SpessaSharp's synth matches upstream in pretty much all cases.
What do you think?
Alternatives
Alternative way of checking the difference would be to invert one wav rendering and mix them both down. Then check if the loudest sample is above some threshold (like the previously mentioned -80dB). This solution doesn't require
soxso it can be programmed in manually.Additional info
Sorry for the wall of text :-]