Document how to add a new evaluation test type

[mcartwright]

Right now we have MUSHRA and pairwise tests, but we may want more test types. We need to explain how to do this in the documentation.

[jfsantos]

I am trying to add an intelligibility test right now and hitting a lot of walls. Is there any chance this documentation could be written, even if it is very coarse-grained? Thanks!

[mcartwright]

Hey,

I'm a bit swamped right now and can't write it out in detail, but the basic steps are:

1. Create a new HTML template for the task in src/caqe/templates/
2. In views.evaluation(), add the logic to render that template and receive/process its post data
3. Add a new task class to caqe.js that inherits from EvaluationTask and modify it as necessary
4. Create a new configuration for your task to src/test_configurations/

For more info, look at b1afcc3 for an example (though it made some other changes as well).

Let me know if you have other questions. Thanks!

[jfsantos]

Thanks, this helped a lot. I think I managed to do it. Are speech intelligibility tests (basically transcription of understood words in a sentence) something you would be interested in adding to CAQE? I can clean up my code and make a pull request later.

[mcartwright]

Definitely. That would be great. Thanks!

[jfsantos]

I have just another quick question: how is one supposed to create a test configuration so that participants will listen to all audio files associated with a given condition within the same HIT?

[mcartwright]

All the data within a condition group should be heard in a single HIT. In the general pairwise configuration, this is separated into multiple conditions with different pairs. In the general mushra configuration, there is just one condition in the condition group.