Can BWA restart a calculation after a break?
1
0
Entering edit mode
3.8 years ago
DNAngel ▴ 240

I'm trying to streamline my analyses using bwa and samtools on different servers but I am only allowed to run big processes for 24 hours before it stops processes. The way around this is to have scripts/programs pick up from where it left off (like rsync can do). Is there a way to make bwa and samtools restart calculations without having to completely redo a whole session?

Thanks!

bwa samtools • 1.7k views
ADD COMMENT
1
Entering edit mode

That is such a strange policy! I've heard of prohibiting long-running processes on login nodes, but clusters are meant to handle jobs that take a long time to get to completion. Maybe try talking to your sysadmin to check why processes are being restricted to 24h? Maybe they need special approval for longer processes.

ADD REPLY
1
Entering edit mode

same problem here :-) (see dariober's solution )

ADD REPLY
1
Entering edit mode

Sadly many (?) NGS programs are not check-point capable (some exceptions, e.g. most 10x genomics software). So if a job is interrupted you will have to start over.

  • While not ideal you could try breaking your input files into multiple chunks and start multiple processes in parallel. Even if there is a limit on that it is hopefully reasonable like 10 or 20.
  • You are also using multiple cores (both bwa and samtools will support it) for your jobs, right? That generally speeds things up.
ADD REPLY
0
Entering edit mode

If you share your code we can see if there are ways to optimize it in order to increase the speed. 24 hours is pretty long for alignment unless you are doing really large WGS or Hi-C datasets.

ADD REPLY
0
Entering edit mode

BWA for one species generally isn't that long, it's that my script will automatically run it for 30+ species. The samtools sort however can take a freakin long time and I have no idea how to accommodate that.

ADD REPLY
0
Entering edit mode

You can use multiple cores for samtools sort as well. You have not given us any information about how you are running these jobs.

ADD REPLY
2
Entering edit mode
3.8 years ago

A dumb solution may be to split the input fastq files in chunks small enough to fit the 24h limit, align each file-chunk and then merge. For file splitting, unix comes with the handy split command that you can use e.g. as:

zcat reads.R1.fq.gz | split -l 40000000 - reads.R1.fq.split
zcat reads.R2.fq.gz | split -l 40000000 - reads.R2.fq.split

this will split each input file in chunks of 40M lines (10M reads).

ADD COMMENT
0
Entering edit mode

The alignment itself isn't long when I run it for one species - it's the samtools sort that takes a while too. But, my bash script will run the alignment for 30+ species which in total can take 2 days almost! I think I have to just add some pauses in the script which will somehow allow me to continue. But the dang samtools sort is just so slow!

ADD REPLY
0
Entering edit mode

But, my bash script will run the alignment for 30+ species which in total can take 2 days almost!

use a workflow manager (make, snakemake, etc..) . If the workflow manager itself takes more than 24H00, break your workflow into parts, the last action of workflow01 will call your cluster manager for workflow02 for the next 24H00, the last action of workflow02 will call your cluster manager for workflow03 for the next 24H00, etc...

ADD REPLY

Login before adding your answer.

Traffic: 1065 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6