bwa aln writing speed became very slow after being very fast. What is the pattern of output size?

Question

[Jerry]

My lab mate has been using bwa aln to align a 21G fastq.gz file to a dog reference genome using 31 threads. At first the generation speed of .sai file was fast, writing 6GB a day and we estimate in 5 days it will complete. However, after writing the first 23 GB at a fast speed, it only write exactly 1 MB (1048576 bytes) every ten or twenty minutes, though CPU usage is still near 3100%.

Is there anything wrong? What is the structure of .sai file? How long will it complete?

I am new to biostars but I have got help from the site many times before. Thanks everyone for sharing knowledge!

Answer 1

[d-cameron]

using bwa aln

bwa aln is designed for very short (<=50bp) reads with bwa mem preferable for longer reads. Why bwa aln?

However, after writing the first 23 GB at a fast speed, it only write exactly 1 MB (1048576 bytes) every ten or twenty minutes, though CPU usage is still near 3100%.

Is this on a dedicated server? This could be due to a number of factors and I recommend looking at the performance metrics on your server to check if anything is amiss. For example, have you (or someone else) started another program on the same computer that uses a lot of memory? If so, you may be out of memory and bwa may be experiencing heavy swapping which will reduce bwa throughput 10-100x. Other possible causes are, VM memory overcommit, VM CPU overcommit (running a 2 CPU VM and a 31 CPU VM on a 32 core physical machine can't run at the same time), throttling programs (e.g. if you're running on a HPC cluster and only asked for 1 CPU then you may be getting your CPU usage artificially limited to 1 CPU), network throughput/throttling (e.g. if your fastq or output file is on a remote server), or other programs in a pipe with bwa bottlenecking it.

Given your CPU usage appears to be 100%, my guess is that you're using too much memory and bwa is thrashing.