Question: How does bcftools index work and how to optimize?
1
gravatar for brian.leon6
3.5 years ago by
brian.leon610
United States
brian.leon610 wrote:

Background:

I've been using samtools and bcftools to call variants from 16 human whole genome runs. They were obtained from ncbi in.sra format and were extracted to .bam format. Each of the 16 Bams were sorted, indexed and I used samtools mpileup to generate the .bcf files.

To accomplish this in a reasonable amount of time I split the data into 1 megabase windows (e.g. chr1:1-1000000, chr1:1000001-2000000) which translate to 3,114 windows. I ran them in a cluster accesible to my lab, running 800 jobs at a time. It finished in a reasonable amount of times and I proceeded to generate index files for eac bam using (this will be a simplified version):

#BATCH --array-1-3114%800    

bcftools index <fileWIndow,bcf>

Question:

I assumed that it would take equal or less amount of resources than generating the .bcf files, it didn't. For some reason it substantially slowed the cluster (It's been 0 days since I last brought the cluster to it's knees) After talking with the cluster support team we didn't really reach any solid conclusion on what was going on, I'm guessing if I understood better what the indexing algorithm is doing I might figure out what's going on.

Could anyone roughly explain why the marked differenc ein performance between mpileup and bcftools index? I'm aware the algorithms are different but I would assume making and index is faster than generating the bcf

Thanks in advance if someone decides to tackle this with me, I'll be happy to provide as much info as possible

tl;dr:

How does bcftools index algorithm generate the indexes? What resources would it be using most? Why would it be more resource demanding than creating bcf files?

ADD COMMENTlink written 3.5 years ago by brian.leon610
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 733 users visited in the last hour