Question: Bfast And Mpileup - How Time Consuming?
1
gravatar for Travis
5.9 years ago by
Travis2.7k
USA
Travis2.7k wrote:

Hi all,

I need to make an approximate prediction (in terms of days) on how long it would take to:

Align 150 samples each consisting of 10 million 100bp Illumina paired end reads to the human genome and run mpileup for each.

The machine I have at my disposal has 256 GB of RAM and 4x6 core 3500MHz processors for a total of 24 cores. The storage is a locally attached RAID5 SCSI.

Apologies if any pertinent details are missing - I am having to calculate these things in the absence of any practical experience!

Thanks in advance.

samtools snp hardware • 2.0k views
ADD COMMENTlink modified 5.9 years ago • written 5.9 years ago by Travis2.7k
2

Any chance you can run 1 sample to use as a baseline? The good news is you have plenty of memory and a scsi storage. A couple of tips that can boost up the speed: 1. Load your indexes and reference in memory. 2. create a filesystem with raid0 for bfast to perform scratches.

ADD REPLYlink written 5.9 years ago by Drio890
1

are those 150 barcoded, or do you have to process them 1 by 1? I won't be able to give you timings, but sure it will help anyone able to do so. also, knowing the processors' speed and the type of storage you have will definitely be of great help too.

ADD REPLYlink written 5.9 years ago by Jorge Amigo9.5k
1

My suggestion would be to run 1 million reads of one of your samples through your pipeline, then multiple by 1500 to get a ballpark estimate. In addition to getting timing estimates, I find this helps with debugging any issues to help the actual run be hands off.

ADD REPLYlink written 5.9 years ago by Brad Chapman9.1k

Thanks guys. I have added the further detail to the question also.

ADD REPLYlink written 5.9 years ago by Travis2.7k

Also - I cannot perform any tests at the minute as I am awaiting admin permission to deploy new software on our server!

ADD REPLYlink written 5.9 years ago by Travis2.7k

Thanks for the tips but RE the testing one sample, the answer is above your question :)

ADD REPLYlink written 5.9 years ago by Travis2.7k

are you decided on bfast or might you be using another aligner? The thing is that bfast can work in different modes of precision, eg. depending on the number of genome indices used. This would make much difference in terms of memory used and therefore in the number of processed that can be run in parallel on your machine.

ADD REPLYlink written 5.9 years ago by Sophia300

I had basically decided to go with BFast for this one. The purpose is SNP/Indel discovery so fair accuracy is required.

ADD REPLYlink written 5.9 years ago by Travis2.7k
5
gravatar for Sophia
5.9 years ago by
Sophia300
Barcelona
Sophia300 wrote:

I ran several samples of 16 Mio reads each (SOLiD single end) using bfast with one index of hg19:

Using 4 processors on a machine with 24 GB RAM, each sample took aprox. 5 hours from reads to sam/bam. I was using GATK for the variant calls, though. Mpileup takes less than half an hour for one of these samples. As far as I experienced it, the limiting factor in bfast is RAM. Please take into consideration that for each instance of alignment that is run using bfast and its indices, at least 12 GB of RAM should be made available.

I ran one sample using 4 indices (1 primary and 3 secondaries), which took aprox. 8-9 hours on the same machine. Using more than one index does not proportionally increase processing time, since secondary indices only are used to align reads that did not align to the primary index, and it does not use more RAM since indices are used sequentially.

ADD COMMENTlink written 5.9 years ago by Sophia300

I was planning to use the 10 indexes recommended for the human genome in the supporting documentation. Having x10 the RAM you used and x 6 the processors should hopefully make this achievable in reasonable time!

ADD REPLYlink written 5.9 years ago by Travis2.7k

one more thing: make sure to use the -U option in the bfast postprocess step. Otherwise this step will run for days.

ADD REPLYlink written 5.9 years ago by Sophia300
4
gravatar for Nilshomer
5.9 years ago by
Nilshomer100
Nilshomer100 wrote:

I would also suggest that since you have so much RAM, you could use the "-l" option with "bfast match" to load in all the indices into memory at once, instead of having to process each serially and merge temporary files. Post back you results!

ADD COMMENTlink written 5.9 years ago by Nilshomer100

Will do Nils. Are there any other settings you would recommend for a study of this type?

ADD REPLYlink written 5.9 years ago by Travis2.7k
1
gravatar for Istvan Albert
5.9 years ago by
Istvan Albert ♦♦ 70k
University Park, USA
Istvan Albert ♦♦ 70k wrote:

Well here is a guesstimate: 1 million reads over a single core might take about between 1 to 3 hours to process.

ADD COMMENTlink written 5.9 years ago by Istvan Albert ♦♦ 70k

So assuming the upper limit of 3 hours and dividing by 24 cores means approx 8 days.

ADD REPLYlink written 5.9 years ago by Travis2.7k

I am actually curious to see how this turns out in practice - guesstimating can be notoriously off.

ADD REPLYlink written 5.9 years ago by Istvan Albert ♦♦ 70k
1
gravatar for Travis
5.9 years ago by
Travis2.7k
USA
Travis2.7k wrote:

To update on this benchmark:

I generated one million 100bp paired end reads with bgeneratereads and aligned them to the human genome with bfast. I used all ten recommended indexes, each stored in RAM. I ran the test once with 16 cores and once 24 cores. There was no other server load when tests were run.

With 16 cores the test took 80 minutes. This included approx 23 minutes to read the indexes into memory.

WIth 24 cores it actually took longer. This took 95 minutes (including the same 23 minutes to read the indexes into RAM).

The command I used was

    bfast match -f hg19.fa -r ../TestReads/bgen.reads -l -k 22 -K 100 -M 500 -n 16 > aligned.bmf
ADD COMMENTlink written 5.9 years ago by Travis2.7k
0
gravatar for Travis
5.9 years ago by
Travis2.7k
USA
Travis2.7k wrote:

Nils - I will soon be running the test (indexes should be generated tomorrow).

I just wanted to re-check if there are any other alignment paramters you recommend?

Also, I just want to confirm that with my 24 core machine, the most cores I can utilize is 16 (4 to the power of 2)?

ADD COMMENTlink modified 5.9 years ago • written 5.9 years ago by Travis2.7k

only for the bfast index command, the number of threads must be a power of 2. For other commands, you can use eg. 20 or 24.

ADD REPLYlink written 5.9 years ago by Sophia300

Thanks Sophia - I found this out myself a few minutes ago when I ran with 24. Unfortunately it actually seems to run slower with 24 cores than it did with 16!!!

ADD REPLYlink written 5.9 years ago by Travis2.7k
0
gravatar for Travis
5.9 years ago by
Travis2.7k
USA
Travis2.7k wrote:

BUMP

Does anyone have any suggestions as to why the benchmark might be slower with more threads?

ADD COMMENTlink written 5.9 years ago by Travis2.7k
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 693 users visited in the last hour