VarScan warning and exit after Invalid format for pileup at line 1

Question

[n.tear]

I'm trying to variant call using VarScan but after sorting the BAM files and creating mpileup files using samtools I get this error using VarScan

Im a novice so any help would be greatly appreciated

java -jar ~/workspace/bioinformaticstools/varscan/varscan-master/VarScan.v2.4.3.jar mpileup2snp /mnt/z/Nathan/WESsamples/unsolved/X366.mpileup >X366.VarScan.snp

Only SNPs will be reported
Warning: No p-value threshold provided, so p-values will not be calculated
Min coverage:   8
Min reads2:     2
Min var freq:   0.2
Min avg qual:   15
P-value thresh: 0.01

Reading input from /mnt/z/Nathan/WESsamples/unsolved/X366.mpileup

Warning: Line ignored: Invalid format for pileup at line 1

╝     �═ BC╗ �╔�B�BCF╝�╗  1 10 11 12 13 14 15 16 17 18 19 2 20 21 22 3 4 5 6 7 8 9 MT X Y GL000192.1 GL000225.1 GL000194.1 GL000193.1 GL000200.1 GL000222.1 GL000212.1 GL000195.1 GL000223.1 GL000224. GL000219.1 GL000205.1 GL000215.1 GL000216.1 GL000217.1 GL000199.1 GL000211.1 GL000213.1 GL000220.1 GL000218.1 GL
000209.1 GL000221.1 GL000214.1 GL000228.1 GL000227.1 GL000191.1 GL000208.1 GL000198.1 GL000204.1 GL000233.1 GL000237.1 GL000230.1 GL000242.1 GL000243.1 GL000241.1 GL000236.1 GL000240.1 GL000206.1 GL000232.1 GL000234.1 GL000202.1 GL000238.1 GL000244.1 GL000248.1 GL000196.1 GL000249.1 GL000246.1 GL000203.1 GL000197.1 GL000245.1 GL000247.1 GL000201.1 GL000235.1 GL000239.1 GL000210.1 GL000231. GL000229.1 GL000226.1 GL000207.1 A   /mnt/z/NathanHaffordTear/WESsamples/PPCDunsolved/X366.sorted.ba �
GL000219.1 GL000205.1 GL000215.1 GL000216.1 GL000217.1 GL000199.1 GL000211.1 GL000213.1 GL000220.1 GL                                                                                  ##samtoolsVersion=0.1.19-96b5f2294a

[ATpoint]

The pileup is corrupted. Please post the entire command that was used to generate it. Did you use nohup?

[n.tear]

Hi Thanks for the response. This was the command I used to generate the mpile up file:

samtools mpileup -E -uf /mnt/z/Nathan/ReferenceGenome/Homo_sapiens.GRCh37.dna.primary_assembly.fa /mnt/z/NathanHaffordTear/WESsamples/unsolved/X366.sorted.bam > X366.mpileup

I was following instruction from this paper: https://www.biorxiv.org/content/biorxiv/early/2017/10/11/201145.full.pdf

[ATpoint]

Is that the full command? No nohup or anything, or the command embedded in a wrapper function?

[n.tear]

Yes thats the full command. Sorry im not familiar with nohup or embedding in a wrapper

[ATpoint]

The thing is that your pileup looks like it was contaminated by some binary information, maybe a mix of stderr and stdout, that is why I was asking. Can you post the output of

samtools mpileup -E -uf /mnt/z/Nathan/ReferenceGenome/Homo_sapiens.GRCh37.dna.primary_assembly.fa /mnt/z/NathanHaffordTear/WESsamples/unsolved/X366.sorted.bam | head -n 20

[n.tear]

     head -20 X366.mpileup
╝     ═ BC╗ ╔BBCF╝╗  1 10 11 12 13 14 15 16 17 18 19 2 20 21 22 3 4 5 6 7 8 9 MT X Y GL000192.1 GL000225.1 GL000194.1 GL000193.1 GL000200.1 GL000222.1 GL000212.1 GL000195.1 GL000223.1 GL000224.1 GL000219.1 GL000205.1 GL000215.1 GL000216.1 GL000217.1 GL000199.1 GL000211.1 GL000213.1 GL000220.1 GL000218.1 GL000209.1 GL000221.1 GL000214.1 GL000228.1 GL000227.1 GL000191.1 GL000208.1 GL000198.1 GL000204.1 GL000233.1 GL000237.1 GL000230.1 GL000242.1 GL000243.1 GL000241.1 GL000236.1 GL000240.1 GL000206.1 GL000232.1 GL000234.1 GL000202.1 GL000238.1 GL000244.1 GL000248.1 GL000196.1 GL000249.1 GL000246.1 GL000203.1 GL000197.1 GL000245.1 GL000247.1 GL000201.1 GL000235.1 GL000239.1 GL000210.1 GL000231.1 GL000229.1 GL000226.1 GL000207.1 A   /mnt/z/Nathan/WESsamples/unsolved/X366.sorted.bam                                                                                                                                                                                   ##samtoolsVersion=0.1.19-96b5f2294a##reference=file:///mnt/z/NathanHaffordTear/ReferenceGenome/Homo_sapiens.GRCh37.dna.primary_assembly.fa##contig=<ID=1,length=249250621>##contig=<ID=10,length=135534747>
    ##contig=<ID=11,length=135006516>
    ##contig=<ID=12,length=133851895>
    ##contig=<ID=13,length=115169878>
    ##contig=<ID=14,length=107349540>
    ##contig=<ID=15,length=102531392>
    ##contig=<ID=16,length=90354753>
    ##contig=<ID=17,length=81195210>
    ##contig=<ID=18,length=78077248>
    ##contig=<ID=19,length=59128983>
    ##contig=<ID=2,length=243199373>
    ##contig=<ID=20,length=63025520>
    ##contig=<ID=21,length=48129895>
    ##contig=<ID=22,length=51304566>
    ##contig=<ID=3,length=198022430>
    ##contig=<ID=4,length=191154276>
    ##contig=<ID=5,length=180915260>

I also get this warning when running mpileup with samtools. should I be using bcftools?

[warning] samtools mpileup option `u` is functional, but deprecated. Please switch to using bcftools mpileup in future.

[ATpoint]

I do not know what this is. A mpileup file looks like this:

chr1    3000238 a   1   ^K. A
chr1    3000239 t   1   .   A
chr1    3000240 t   1   .   A
chr1    3000241 t   1   .   A
chr1    3000242 c   1   .   A
chr1    3000243 a   1   .   E
chr1    3000244 t   1   .   E
chr1    3000245 c   1   .   E
chr1    3000246 c   1   .   E
chr1    3000247 a   1   .   E

Do you run this directly via the command line?

[n.tear]

okay thanks yes I am running from a ubuntu bash shell on windows. perhaps I need to QC my bam files to check they are correct

[Ram]

Looks like a partial VCF file. Maybe a BCF file?

[Ram]

You should also add the samtools command that you used to generate the pileup, because from the error message it seems like the problem lies there.

[Ram]

Please use the formatting bar (especially the code option) to present your post better. You can use backticks for inline code (`text` becomes text), or select a chunk of text and use the highlighted button to format it as a code block. I've done it for you this time.

[n.tear]

Hi RamRS. Sorry, will do thanks

[ATpoint]

I made the changes already :)