multithreading problem

[gpertea]

Zachary Zwiesler reported a possible issue with the multi-threading code:

I have been attempting to run fqtrim with multithreading. These are my command line arguments:

fqtrim -V -y 3 -p 8 -o $out $1,$2

The program executes as expected, but I haven't observed any increase in speed with more threads specified. % CPU usage appeared maxed out at up to four cores, but elapsed time remained similar, and assignment of greater than four cores showed % CPU usage around 400 and no decrease in elapsed time. In fact when running ~500 MB files, I observed an increase in elapsed time.

[gpertea]

Very likely this is a problem with my multi-threading code (inefficient producer-consumer implementation, with unnecessary locks & wait cycles etc.), similar to the issues that plagued the initial versions of StringTie's multi-threading code..
As an aside, fqtrim is largely I/O bound (i.e. mostly limited by disk/network streaming speed) so we shouldn't expect a significant decrease of wall time with multiple CPUs (definitely NOT linear), but same/increase of wall time is clearly a problem in the current implementation.