Add bitonic sort implementation

[xaellison]

Implementation of sortperm! and sortperm using a bitonic sorter. The underlying bitonic sorter is general enough for any length array, and can do simple sorting as well as sortperm.

I believe it's faster for many scenarios than the existing quicksort, so maybe sort!{::AnyCuVector} should also use it. The complexity is worse though: for large enough arrays quicksort will outperform this, but that seems to be around lengths of ~16 million. Lengths like 2^n+1 perform similarly to 2^(n+1), hence the jagged performance curve. This data is from rtx 2070.

[codecov]

Codecov Report

Merging #1217 (991f3ae) into master (89d2e82) will decrease coverage by 0.75%.
The diff coverage is 27.31%.

❗ Current head 991f3ae differs from pull request most recent head de44670. Consider uploading reports for the commit de44670 to get more accurate results

@@            Coverage Diff             @@
##           master    #1217      +/-   ##
==========================================
- Coverage   80.09%   79.34%   -0.76%     
==========================================
  Files         119      119              
  Lines        8407     8594     +187     
==========================================
+ Hits         6734     6819      +85     
- Misses       1673     1775     +102     

Impacted Files	Coverage Δ
src/sorting.jl	23.91% <27.31%> (+1.64%)	⬆️
lib/cusparse/interfaces.jl	81.81% <0.00%> (-3.12%)	⬇️
src/accumulate.jl	97.05% <0.00%> (-2.95%)	⬇️
lib/cublas/wrappers.jl	92.56% <0.00%> (+0.22%)	⬆️
lib/cutensor/wrappers.jl	95.47% <0.00%> (+0.84%)	⬆️
src/linalg.jl	86.36% <0.00%> (+6.36%)	⬆️
lib/cudadrv/CUDAdrv.jl	80.00% <0.00%> (+32.00%)	⬆️
lib/cudnn/CUDNN.jl	75.80% <0.00%> (+37.71%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 89d2e82...de44670. Read the comment docs.

[maleadt]

Looks great! Good to have an implementation that doesn't rely on dynamic parallelism, which doesn't always perform well.

It'll take a while before I can review properly, but I added some quick thoughts already.

I believe it's faster for many scenarios than the existing quicksort, so maybe sort!{::AnyCuVector} should also use it. The complexity is worse though: for large enough arrays quicksort will outperform this, but that seems to be around lengths of ~16 million.

Maybe select the best algorithm at run time then? With possible customizability using an alg::Algorithm keyword like Base.

[xaellison]

Thanks for preliminary feedback @maleadt
Two quick thoughts for you:

1. Based on Allow sorting of tuples of numbers #1196 it looks like there is interest in sortperm(a; dims) - do you have an opinion about implementing more features than what exists in Base.sortperm (which doesn't do dims)?
2. I like the idea of selecting an algorithm based on input size but it may be tricky to consistently make a good choice across all hardware (happy to try)

[maleadt]

1. More features is OK. I wonder if we should use a different method then, like how GPUArrays.mapreducedim! supports an init keyword while Base.mapreducedim! doesn't.
2. As long as there's a way to select the algorithm the default selection doesn't need to be particularly robust. Or maybe it's easier to just default to bitonic and document the quicksort option.

[maleadt]

Totally forgot about this, sorry! I've squashed and rebased the commits, and did some minor NFC changes (e.g. rename CuBitonicSort algorithm selector to the unexported CUDA.BitonicSort).

[xaellison]

Cool thanks! I just started a new job so this was off my radar too :)