Question: bbmap.sh: line 352 /random number/ Killed
0
gravatar for mutantjoo0
3 months ago by
mutantjoo00
East Lansing, MI, US
mutantjoo00 wrote:

Hello Jedai Masters of Biostars,

I keep facing crash while mapping contig files with BBMap (v.38.76). I am running command "bbmap.sh in=sample_r1.fastq in2=sample_r2.fastq out=./sample_bbmapped.sam bamscript=bs.sh" and process stops on "Started 40 mapping threads" as highlighted in quote below. I can only see "bbmap.sh:line 352: /random number/ Killed". I supposed it could be due to low memory, but using maximum memory set (377G is max on dev-node of my institute) cannot help. I wonder if anyone have had similar issue or know how to fix this. Thank you for your time and consideration!

May the force be with you, Joo-Young

======================

java -da -Xmx377g -Xms377g -cp /mnt/home/leejooy5/Downloads/bbmap/current/ align2.BBMap build=1 overwrite=true fastareadlen=500 -da -Xmx377g in=Std_host_removed_R1.fastq in2=Std_host_removed_R2.fastq out=./sorted_indexed20200219/Std_megahit_BBmapped.sam bamscript=bs.sh
Executing align2.BBMap [build=1, overwrite=true, fastareadlen=500, -da, -Xmx377g, in=Std_host_removed_R1.fastq, in2=Std_host_removed_R2.fastq, out=./sorted_indexed20200219/Std_megahit_BBmapped.sam, bamscript=bs.sh]
Version 38.76

Retaining first best site only for ambiguous mappings.
Found samtools 1.9
Set genome to 1

Loaded Reference: 0.509 seconds.
Loading index for chunk 1-1, build 1
Generated Index: 1.078 seconds.
Analyzed Index: 2.608 seconds.
Started output stream: 0.075 seconds.
Cleared Memory: 0.389 seconds.
Processing reads in paired-ended mode.
Started read stream.
**Started 40 mapping threads.**
/mnt/home/leejooy5/Downloads/bbmap/**bbmap.sh: line 352: 10803 Killed** java -da -Xmx377g -Xms377g -cp /mnt/home/leejooy5/Downloads/bbmap/current/ align2.BBMap build=1 overwrite=true fastareadlen=500 -da -Xmx377g in=Std_host_removed_R1.fastq in2=Std_host_removed_R2.fastq out=./sorted_indexed20200219/Std_megahit_BBmapped.sam bamscript=bs.sh
Note: This script is designed to run with the amount of memory detected by BBMap.
If Samtools crashes, please ensure you are running on the same platform as BBMap,
or reduce Samtools' memory setting (the -m flag).
Note: Please ignore any warnings about 'EOF marker is absent'; this is a bug in samtools that occurs when using piped input.
[bam_sort_core] merging from 0 files and 3 in-memory blocks...
ADD COMMENTlink modified 3 months ago by genomax84k • written 3 months ago by mutantjoo00
1

Have you pre-made the indexes and are they in the same folder as where your reads are? You should never use ALL of the available memory for BBMap jobs. 80% of Max is a safe number. That said unless you have a humongous genome you should not need that much memory. Human genome takes about 30G.

Can you try following and let me know if this works? Adjust the 40g as needed up or down but that should be enough.

bbmap.sh -Xmx40g  threads=4 in=sample_r1.fastq in2=sample_r2.fastq out=./sample_bbmapped.sam bamscript=bs.sh

As long as you have samtools available in your $PATH you can just do :

bbmap.sh -Xmx40g  threads=4 in=sample_r1.fastq in2=sample_r2.fastq out=./sample_bbmapped.bam
ADD REPLYlink modified 3 months ago • written 3 months ago by genomax84k

Hi Genomax,

Thank you for your comment. Followings are commands I originally used to index and map.

# to index:  /mnt/home/leejooy5/Downloads/bbmap/bbmap.sh -Xmx20g ref=../megahit_assembly/SAMPLE/SAMPLE.final.contigs.fa nodisk 

# to map:  /mnt/home/leejooy5/Downloads/bbmap/bbmap.sh -Xmx20g in=SAMPLE_host_removed_R1.fastq in2=SAMPLE_host_removed_R2.fastq out=./sorted_indexed20200219/SAMPLE_megahit_BBmapped.sam bamscript=bs.sh; sh bs.sh

I tried both of your suggestion before I increased -Xmx set. Also I tried following your suggestion, and it was still killed with or without bamscript option.

I will try after re-assembly and let you know.

ADD REPLYlink modified 3 months ago • written 3 months ago by mutantjoo00

Does your account have limits on how much memory and/or CPU you are allowed to use? What is the size of SAMPLE.final.contigs.fa?

ADD REPLYlink written 3 months ago by genomax84k

Here is my resource in hpc:

(base) -bash-4.2$ free -mh
              total        used        free      shared  buff/cache   available
Mem:           377G         35G        289G        4.5G         52G        335G
Swap:            0B          0B          0B

I tried with higher memories than 20G, but those are still not working. And size of the trouble maker is 66M (Std and Std_re, re-assembly below). Other files with similar and even larger contigs files made no trouble.

(base) -bash-4.2$ ls -alh .//final.contigs.fa

-rw-r----- 1 leejooy5 mmg 2.4M Oct 11 16:22 ./DEB/DEB.final.contigs.fa
-rw-r----- 1 leejooy5 mmg 6.0M Oct 11 16:25 ./DES/DES.final.contigs.fa
-rw-r----- 1 leejooy5 mmg 2.4M Oct 11 16:26 ./eNAT/eNAT.final.contigs.fa
-rw-r----- 1 leejooy5 mmg  56M Oct 11 16:42 ./GR25/GR25.final.contigs.fa
-rw-r----- 1 leejooy5 mmg  26M Oct 11 16:43 ./GR26/GR26.final.contigs.fa
-rw-r----- 1 leejooy5 mmg 3.9M Oct 11 17:51 ./GR27/GR27.final.contigs.fa
-rw-r----- 1 leejooy5 mmg 7.4M Oct 11 17:52 ./GR28/GR28.final.contigs.fa
-rw-r----- 1 leejooy5 mmg  31M Oct 11 17:56 ./GR29/GR29.final.contigs.fa
-rw-r----- 1 leejooy5 mmg  41M Oct 11 18:34 ./GR30/GR30.final.contigs.fa
-rw-r----- 1 leejooy5 mmg  70M Oct 11 18:42 ./GR31/GR31.final.contigs.fa
-rw-r----- 1 leejooy5 mmg  88M Oct 11 18:48 ./GR32/GR32.final.contigs.fa
-rw-r----- 1 leejooy5 mmg  32M Oct 11 18:43 ./GR33/GR33.final.contigs.fa
-rw-r----- 1 leejooy5 mmg 3.5M Oct 11 20:13 ./GR34/GR34.final.contigs.fa
-rw-r----- 1 leejooy5 mmg 7.7M Oct 11 20:17 ./GR35/GR35.final.contigs.fa
-rw-r----- 1 leejooy5 mmg 5.4M Oct 11 20:18 ./GR36/GR36.final.contigs.fa
-rw-r----- 1 leejooy5 mmg  78M Oct 11 20:50 ./GR37/GR37.final.contigs.fa
-rw-r----- 1 leejooy5 mmg  37M Oct 11 20:29 ./GR38/GR38.final.contigs.fa
-rw-r----- 1 leejooy5 mmg  24M Oct 11 20:37 ./GR39/GR39.final.contigs.fa
-rw-r----- 1 leejooy5 mmg  49M Oct 11 20:43 ./GR40/GR40.final.contigs.fa
-rw-r----- 1 leejooy5 mmg 129M Oct 11 22:38 ./GR41_1/GR41_1.final.contigs.fa
-rw-r----- 1 leejooy5 mmg 109M Oct 11 23:09 ./GR41_2/GR41_2.final.contigs.fa
-rw-r----- 1 leejooy5 mmg 301K Oct 11 23:16 ./GR42/GR42.final.contigs.fa
-rw-r----- 1 leejooy5 mmg 7.0M Oct 11 23:19 ./GR43/GR43.final.contigs.fa
-rw-r----- 1 leejooy5 mmg  45M Oct 11 23:31 ./GR44/GR44.final.contigs.fa
-rw-r----- 1 leejooy5 mmg  772 Oct 11 16:28 ./NFW/NFW.final.contigs.fa
-rw-r----- 1 leejooy5 mmg  66M Feb 23 15:29 ./**Std_re**/final.contigs.fa
-rw-r----- 1 leejooy5 mmg  66M Oct 11 16:52 ./**Std**/Std.final.contigs.fa
ADD REPLYlink written 3 months ago by mutantjoo00
1

Do you actually have * in the directory names above? Wonder if those are causing issues.

bbmap.sh -Xmx40g  threads=4 in1=sample_r1.fastq in2=sample_r2.fastq out=sample_bbmapped.bam ref=your_reference.fa
ADD REPLYlink modified 3 months ago • written 3 months ago by genomax84k

Sorry for the inconvenience. I made mistakes misleading you. There are no file contains * in its name. I used symbol * NAME * to highlight NAME in bold.

ADD REPLYlink written 3 months ago by mutantjoo00

Did you try the command above (which will create a reference on the fly and then map)?

ADD REPLYlink written 3 months ago by genomax84k

Yes, I tried and it did not work. Followings are two trials I did with difference in the order of BBMap flag and their output. As you can see, quality of outputs are not bad although those two have difference in number of Total Sequences.

trial 1:

$ /mnt/home/leejooy5/Downloads/bbmap/bbmap.sh -Xmx40g threads=4 in1=Std_host_removed_R1.fastq in2=Std_host_removed_R2.fastq out=./Std_bbmapped/Std_bbmapped.bam ref=../megahit_assembly/Std/Std.final.contigs.fa nodisk
java -ea -Xmx40g -Xms40g -cp /mnt/home/leejooy5/Downloads/bbmap/current/ align2.BBMap build=1 overwrite=true fastareadlen=500 -Xmx40g threads=4 in1=Std_host_removed_R1.fastq in2=Std_host_removed_R2.fastq out=./Std_bbmapped/Std_bbmapped.bam ref=../megahit_assembly/Std/Std.final.contigs.fa nodisk
Executing align2.BBMap [build=1, overwrite=true, fastareadlen=500, -Xmx40g, threads=4, in1=Std_host_removed_R1.fastq, in2=Std_host_removed_R2.fastq, out=./Std_bbmapped/Std_bbmapped.bam, ref=../megahit_assembly/Std/Std.final.contigs.fa, nodisk]
Version 38.76

Set threads to 4
Retaining first best site only for ambiguous mappings.
Executing dna.FastaToChromArrays2 [../megahit_assembly/Std/Std.final.contigs.fa, 1, writeinthread=false, genscaffoldinfo=true, retain, waitforwriting=false, gz=true, maxlen=536670912, writechroms=false, minscaf=1, midpad=300, startpad=8000, stoppad=8000, nodisk=true]

Set genScaffoldInfo=true
Set genome to 1

Loaded Reference:       0.030 seconds.
Loading index for chunk 1-1, build 1
Indexing threads started for block 0-1
Indexing threads finished for block 0-1
Generated Index:        7.241 seconds.
Analyzed Index:         2.610 seconds.
Found samtools 1.9
Started output stream:  0.258 seconds.
Cleared Memory:         0.441 seconds.
Processing reads in paired-ended mode.
Started read stream.
Started 4 mapping threads. [15:06~
/mnt/home/leejooy5/Downloads/bbmap/bbmap.sh: line 352: 17703 Killed                  java -ea -Xmx40g -Xms40g -cp /mnt/home/leejooy5/Downloads/bbmap/current/ align2.BBMap build=1 overwrite=true fastareadlen=500 -Xmx40g threads=4 in1=Std_host_removed_R1.fastq in2=Std_host_removed_R2.fastq out=./Std_bbmapped/Std_bbmapped.bam ref=../megahit_assembly/Std/Std.final.contigs.fa

trial2:

$ /mnt/home/leejooy5/Downloads/bbmap/bbmap.sh -Xmx40g -threads=4 ref=../megahit_assembly/Std/Std.final.contigs.fa nodisk in1=Std_host_removed_R1.fastq in2=Std_host_removed_R2.fastq out=./Std_bbmapped/Std2_bbmapped.bam
java -ea -Xmx40g -Xms40g -cp /mnt/home/leejooy5/Downloads/bbmap/current/ align2.BBMap build=1 overwrite=true fastareadlen=500 -Xmx40g -threads=4 ref=../megahit_assembly/Std/Std.final.contigs.fa nodisk in1=Std_host_removed_R1.fastq in2=Std_host_removed_R2.fastq out=./Std_bbmapped/Std2_bbmapped.bam
Executing align2.BBMap [build=1, overwrite=true, fastareadlen=500, -Xmx40g, threads=4, ref=../megahit_assembly/Std/Std.final.contigs.fa, nodisk, in1=Std_host_removed_R1.fastq, in2=Std_host_removed_R2.fastq, out=./Std_bbmapped/Std2_bbmapped.bam]
Version 38.76

Set threads to 4
Retaining first best site only for ambiguous mappings.
Executing dna.FastaToChromArrays2 [../megahit_assembly/Std/Std.final.contigs.fa, 1, writeinthread=false, genscaffoldinfo=true, retain, waitforwriting=false, gz=true, maxlen=536670912, writechroms=false, minscaf=1, midpad=300, startpad=8000, stoppad=8000, nodisk=true]

Set genScaffoldInfo=true
Set genome to 1

Loaded Reference:       0.035 seconds.
Loading index for chunk 1-1, build 1
Indexing threads started for block 0-1
Indexing threads finished for block 0-1
Generated Index:        6.390 seconds.
Analyzed Index:         2.782 seconds.
Found samtools 1.9
Started output stream:  0.097 seconds.
Cleared Memory:         0.382 seconds.
Processing reads in paired-ended mode.
Started read stream.
Started 4 mapping threads.
/mnt/home/leejooy5/Downloads/bbmap/bbmap.sh: line 352: 31551 Killed                  java -ea -Xmx40g -Xms40g -cp /mnt/home/leejooy5/Downloads/bbmap/current/ align2.BBMap build=1 overwrite=true fastareadlen=500 -Xmx40g -threads=4 ref=../megahit_assembly/Std/Std.final.contigs.fa nodisk in1=Std_host_removed_R1.fastq in2=Std_host_removed_R2.fastq out=./Std_bbmapped/Std2_bbmapped.bam

output:

$ ls -lha
total 11G
drwxr-x--- 4 leejooy5 mmg 8.0K Feb 26 17:08 .
drwxr-x--- 5 leejooy5 mmg 8.0K Feb 26 18:02 ..
-rw-r----- 1 leejooy5 mmg 5.1G Feb 26 16:20 Std2_bbmapped.bam
-rw-r----- 1 leejooy5 mmg 5.8G Feb 26 15:34 Std_bbmapped.bam
ADD REPLYlink written 3 months ago by mutantjoo00

(continued reply)

fastqc results:

Measure Value

Filename Std_bbmapped.bam

File type Conventional base calls

Encoding Sanger / Illumina 1.9

Total Sequences 66328781

Sequences flagged as poor quality 0

Sequence length 24-150

%GC 47

Measure Value

Filename Std2_bbmapped.bam

File type Conventional base calls

Encoding Sanger / Illumina 1.9

Total Sequences 58126312

Sequences flagged as poor quality 0

Sequence length 24-150

%GC 47

ADD REPLYlink modified 3 months ago • written 3 months ago by mutantjoo00
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 1809 users visited in the last hour