I would like to better understand how threads are handled by bcl-convert (v3.8.2-12) but I don't find any clear information how to properly tune the parameters.
There are 4 options:
--bcl-num-decompression-threads --bcl-conversion-threads --bcl-num-compression-threads --bcl-num-parallel-tiles
If I have 16 threads, should I set all this option to 16? If each step is independant I guess setting them to 16 would be fine, but I guess a least a part of the steps are overalping e.g ther is decompression, conversion, and compression on the fly.... So I should put 4 threads to each of the 4 steps?
bcl-convert -h says:
--bcl-num-parallel-tiles arg # of tiles to process in parallel (default 1) --bcl-num-conversion-threads arg # of threads for conversion (per tile, default # cpu threads) --bcl-num-compression-threads arg # of threads for fastq.gz output compression (per tile, default # cpu threads, or HW+12) --bcl-num-decompression-threads arg # of threads for bcl/cbcl input decompression (per tile, default half # cpu threads, or HW+8. Only applies when preloading files)
while the online doc says that by default the threads are
I have tried and it is true it is dinamically determined, but this is a huge problem in my nextflow pipeline because it uses all the CPUs while I need some for other purposes. So I definilty want to set the threads to use myself.
P.S: Acutally I had the same thread tuning problem with bcl2fastq, does someone has a good tunnning with the 3 parameters from bcl2fastq that I can get inspired from? I can apply the same to bcl-convert
decompression (bcl-num-decompression-thread) seems to be less demanding
default half # cpu threads
Do a bit of local testing since much of this will depend on speed/efficacy of your storage.