Question: How Can I Merge A Large Amount Of Vcf Files?

7.4 years ago by

Matt W • 240

Matt W • 240 wrote:

I have about one thousand vcf files, and this is a ridiculous amount to type by hand into vcf-merge. Is there a way to merge all of these files without typing each name?

I tried putting all of the files into a text file and using stdin, but vcf-merge doesn't support this, i.e.

vcf-merge < list.txt > merge.vcf

Maybe this problem stems from creating this many vcf files to begin with. When using vcf-tools, I need to select multiple locations (i.e.--from-bp 100 --to-bp 200) which means I need to run vcf-tools for each location. Is there a better way to do this?

Thanks in advance for any help!

vcf merge vcftools snp • 34k views

ADD COMMENT • link •

modified 17 months ago by ATpoint ♦ 26k • written 7.4 years ago by Matt W • 240

7.4 years ago by

Chris Miller ♦ 21k

Washington University in St. Louis, MO

Chris Miller ♦ 21k wrote:

If the problem is merely getting the list into vcf-merge, it's as easy as:

vcf-merge $(ls -1 *.vcf | perl -pe 's/\n/ /g') > merge.vcf

(assumes all your vcfs to be merged are in the same directory)

ADD COMMENT • link modified 12 months ago by RamRS ♦ 25k • written 7.4 years ago by Chris Miller ♦ 21k

This is definitely along the track of what I'm trying to achieve. However, I keep getting an error: http://pastebin.com/H5AQZEtp Any ideas?

ADD REPLY • link written 7.4 years ago by Matt W • 240

A quick guess: the argument to ls is the number one (1), not the letter L (l).

ADD REPLY • link modified 7.4 years ago • written 7.4 years ago by matted ♦ 7.2k

That did the trick! Thanks for your help!

ADD REPLY • link written 7.4 years ago by Matt W • 240

7.4 years ago by

Pierre Lindenbaum ♦ 124k

France/Nantes/Institut du Thorax - INSERM UMR1087

Pierre Lindenbaum ♦ 124k wrote:

deprecated : do not use this

I wrote a tool named scanvcf : it concatenates the VCF, adds an extra column with the sample name.

$ head -n3 input.txt

#Sample VCF
Sample1 data/sample1.vcf.gz
Sample2 data/sample1.vcf.gz
Sample2 data/sample1.vcf.gz


$ cat input.txt |scanvcf 
#CHROM POS ID REF ALT QUAL FILTER . FORMAT Call SAMPLE
1 879317 rs7523549 C T 71 0 . GT:GQ:DP:FLT 0/1:34:8:0 Sample1
1 880238 rs3748592 A G 51 0 . GT:GQ:DP:FLT 1/1:51:8:0 Sample1
1 880390 rs3748593 C A 99 0 . GT:GQ:DP:FLT 1/0:99:30:0 Sample1
1 881627 rs2272757 G A 99 0 . GT:GQ:DP:FLT 1/0:59:20:0 Sample1
(...)
Y 13524507 . C T 99 0 . GT:GQ:DP:FLT 1/1:99:233:0 Sample20
Y 21154323 rs10465459 G A 99 0 . GT:GQ:DP:FLT 1/1:99:215:0 Sample20
Y 21154426 rs52812045 G A 99 0 . GT:GQ:DP:FLT 1/0:99:143:0 Sample20
Y 21154466 rs10465460 T A 99 0 . GT:GQ:DP:FLT 1/1:99:134:0 Sample20
Y 21154529 . G A 51 0 . GT:GQ:DP:FLT 1/1:51:8:0 Sample20

You can later filter the result using samplespersnp.

  $ cat list.tsv | scanvcf  |\
  sort -t'  ' -k1,1 -k2,2n -k4,4 -k5,5 -k11,11 |\
  samplespersnp --sample 11 | awk '($8=".")'
1 753269 rs61770172 C G 99 0 . GT:GQ:DP:FLT 1/1:99:116:0 Sample16 1
1 753405 rs61770173 C A 99 0 . GT:GQ:DP:FLT 1/1:63:31:0 Sample10 7
1 753405 rs61770173 C A 81 0 . GT:GQ:DP:FLT 1/1:51:19:0 Sample12 7
1 753405 rs61770173 C A 35 0 . GT:GQ:DP:FLT 1/0:35:66:0 Sample19 7
1 753405 rs61770173 C A 99 0 . GT:GQ:DP:FLT 1/1:99:35:0 Sample3 7
1 753405 rs61770173 C A 90 0 . GT:GQ:DP:FLT 1/1:90:21:0 Sample5 7
1 753405 rs61770173 C A 99 0 . GT:GQ:DP:FLT 1/1:99:36:0 Sample6 7
1 753405 rs61770173 C A 90 0 . GT:GQ:DP:FLT 1/1:90:21:0 Sample9 7
1 876499 rs4372192 A G 39 0 . GT:GQ:DP:FLT 1/1:39:4:0 Sample12 6
1 876499 rs4372192 A G 42 0 . GT:GQ:DP:FLT 1/1:42:5:0 Sample16 6
1 876499 rs4372192 A G 39 0 . GT:GQ:DP:FLT 1/1:39:4:0 Sample17 6
1 876499 rs4372192 A G 45 0 . GT:GQ:DP:FLT 1/1:45:6:0 Sample18 6
1 876499 rs4372192 A G 45 0 . GT:GQ:DP:FLT 1/1:45:6:0 Sample4 6
1 876499 rs4372192 A G 42 0 . GT:GQ:DP:FLT 1/1:42:5:0 Sample6 6
1 877831 rs6672356 T C 42 0 . GT:GQ:DP:FLT 1/1:42:5:0 Sample14 2
1 877831 rs6672356 T C 39 0 . GT:GQ:DP:FLT 1/1:39:4:0 Sample4 2
1 878601 . C T 98 0 . GT:GQ:DP:FLT 0/1:50:11:0 Sample14 1

This tool also takes an option -e for a query over the samples. e.g: "variation must contains Sample11, Sample12 BUT NOT Sample1 to Sample5:"

-e '(Sample11  && Sample12 && (!(Sample1 || Sample2 || Sample3 || Sample4 || Sample5)))'

ADD COMMENT • link modified 12 months ago by RamRS ♦ 25k • written 7.4 years ago by Pierre Lindenbaum ♦ 124k

2.5 years later: my solution is deprecated. This is not what you're looking for.

ADD REPLY • link written 4.8 years ago by Pierre Lindenbaum ♦ 124k

what do you use now?

ADD REPLY • link written 4.6 years ago by Ming Tang • 2.5k

vcffilterjs https://github.com/lindenb/jvarkit/wiki/VCFFilterJS

ADD REPLY • link written 4.6 years ago by Pierre Lindenbaum ♦ 124k

7.4 years ago by

matted ♦ 7.2k

Boston, United States

matted ♦ 7.2k wrote:

If I understand your task correctly, xargs would be a quick way to go (similar to the perl example above):

ls *vcf | xargs vcf-merge > merged.vcf

Or if it's a bit more complicated and you want to use the file list as input:

cat files.txt | xargs vcf-merge > merged.vcf

ADD COMMENT • link modified 12 months ago by RamRS ♦ 25k • written 7.4 years ago by matted ♦ 7.2k

Definitely what I am looking for, but each time I try there is some error. When I had plain vcf files it asked for the tabix index. So I used bgzip | tabix and tried this again doing ls *.vcf.gz and another error reading columns comes up: http://pastebin.com/aDpgkqjw Any ideas? Thanks for your help! EDIT: I also know vcf-merge works correctly because I have used it successfully on a small number of files.

ADD REPLY • link modified 7.4 years ago • written 7.4 years ago by Matt W • 240

Resolved. My blank output was being picked up by the *vcf and throwing an error. Thanks for the help!

ADD REPLY • link written 7.4 years ago by Matt W • 240

6.5 years ago by

Malachi Griffith ♦ 18k

Washington University School of Medicine, St. Louis, USA

Malachi Griffith ♦ 18k wrote:

Another option is: joinx

joinx vcf-merge [OPTIONS] file1.vcf file2.vcf [file3.vcf ...]

ADD COMMENT • link modified 12 months ago by RamRS ♦ 25k • written 6.5 years ago by Malachi Griffith ♦ 18k

7.4 years ago by

Vivek • 2.3k

Denmark

Vivek • 2.3k wrote:

I think GATK CombineVariants walker would support a list of files as input.

ADD COMMENT • link written 7.4 years ago by Vivek • 2.3k

7.4 years ago by

Sukhdeep Singh ♦ 9.9k

Netherlands

Sukhdeep Singh ♦ 9.9k wrote:

What about, time while read line; do vcf-merge file1.vcf $line >> merge.cvf; done < list.txt and your list.txt should have filenames except the first one.

Cheers

ADD COMMENT • link written 7.4 years ago by Sukhdeep Singh ♦ 9.9k

Hello,

The script help to merge all vcf files, but how can you make it possible to add columns with sample name.?

ADD REPLY • link written 4.2 years ago by a.james • 210

17 months ago by

ATpoint ♦ 26k

Germany

ATpoint ♦ 26k wrote:

Years later, only using Unix tool, producing a coord-sorted and bgzipped VCF:

ls *.vcf | head -n 1 | xargs grep '#' > ${BASENAME}_header.txt
cat *.vcf | mawk '$1 ~ /^#/ {next} {print $0 | "sort -k1,1 -k2,2n --parallel=8"}' | \
  cat ${BASENAME}_header.txt /dev/stdin | bgzip -@ 8 > out.catted.vcf.gz

ADD COMMENT • link written 17 months ago by ATpoint ♦ 26k

Please log in to add an answer.

deprecated : do not use this

Similar posts • Search »