Entering edit mode
4.2 years ago
mutantjoo0
•
0
Hello Jedai Masters of Biostars,
I keep facing crash while mapping contig files with BBMap (v.38.76). I am running command "bbmap.sh in=sample_r1.fastq in2=sample_r2.fastq out=./sample_bbmapped.sam bamscript=bs.sh" and process stops on "Started 40 mapping threads" as highlighted in quote below. I can only see "bbmap.sh:line 352: /random number/ Killed". I supposed it could be due to low memory, but using maximum memory set (377G is max on dev-node of my institute) cannot help. I wonder if anyone have had similar issue or know how to fix this. Thank you for your time and consideration!
May the force be with you, Joo-Young
======================
java -da -Xmx377g -Xms377g -cp /mnt/home/leejooy5/Downloads/bbmap/current/ align2.BBMap build=1 overwrite=true fastareadlen=500 -da -Xmx377g in=Std_host_removed_R1.fastq in2=Std_host_removed_R2.fastq out=./sorted_indexed20200219/Std_megahit_BBmapped.sam bamscript=bs.sh
Executing align2.BBMap [build=1, overwrite=true, fastareadlen=500, -da, -Xmx377g, in=Std_host_removed_R1.fastq, in2=Std_host_removed_R2.fastq, out=./sorted_indexed20200219/Std_megahit_BBmapped.sam, bamscript=bs.sh]
Version 38.76
Retaining first best site only for ambiguous mappings.
Found samtools 1.9
Set genome to 1
Loaded Reference: 0.509 seconds.
Loading index for chunk 1-1, build 1
Generated Index: 1.078 seconds.
Analyzed Index: 2.608 seconds.
Started output stream: 0.075 seconds.
Cleared Memory: 0.389 seconds.
Processing reads in paired-ended mode.
Started read stream.
**Started 40 mapping threads.**
/mnt/home/leejooy5/Downloads/bbmap/**bbmap.sh: line 352: 10803 Killed** java -da -Xmx377g -Xms377g -cp /mnt/home/leejooy5/Downloads/bbmap/current/ align2.BBMap build=1 overwrite=true fastareadlen=500 -da -Xmx377g in=Std_host_removed_R1.fastq in2=Std_host_removed_R2.fastq out=./sorted_indexed20200219/Std_megahit_BBmapped.sam bamscript=bs.sh
Note: This script is designed to run with the amount of memory detected by BBMap.
If Samtools crashes, please ensure you are running on the same platform as BBMap,
or reduce Samtools' memory setting (the -m flag).
Note: Please ignore any warnings about 'EOF marker is absent'; this is a bug in samtools that occurs when using piped input.
[bam_sort_core] merging from 0 files and 3 in-memory blocks...
Have you pre-made the indexes and are they in the same folder as where your reads are? You should never use ALL of the available memory for BBMap jobs. 80% of Max is a safe number. That said unless you have a humongous genome you should not need that much memory. Human genome takes about 30G.
Can you try following and let me know if this works? Adjust the
40gas needed up or down but that should be enough.As long as you have
samtoolsavailable in your$PATHyou can just do :Hi Genomax,
Thank you for your comment. Followings are commands I originally used to index and map.
I tried both of your suggestion before I increased -Xmx set. Also I tried following your suggestion, and it was still killed with or without bamscript option.
I will try after re-assembly and let you know.
Does your account have limits on how much memory and/or CPU you are allowed to use? What is the size of
SAMPLE.final.contigs.fa?Here is my resource in hpc:
I tried with higher memories than 20G, but those are still not working. And size of the trouble maker is 66M (Std and Std_re, re-assembly below). Other files with similar and even larger contigs files made no trouble.
(base) -bash-4.2$ ls -alh .//final.contigs.fa
Do you actually have
*in the directory names above? Wonder if those are causing issues.Sorry for the inconvenience. I made mistakes misleading you. There are no file contains * in its name. I used symbol * NAME * to highlight NAME in bold.
Did you try the command above (which will create a reference on the fly and then map)?
Yes, I tried and it did not work. Followings are two trials I did with difference in the order of BBMap flag and their output. As you can see, quality of outputs are not bad although those two have difference in number of Total Sequences.
trial 1:
trial2:
output:
(continued reply)
fastqc results:
Measure Value
Filename Std_bbmapped.bam
File type Conventional base calls
Encoding Sanger / Illumina 1.9
Total Sequences 66328781
Sequences flagged as poor quality 0
Sequence length 24-150
%GC 47
Measure Value
Filename Std2_bbmapped.bam
File type Conventional base calls
Encoding Sanger / Illumina 1.9
Total Sequences 58126312
Sequences flagged as poor quality 0
Sequence length 24-150
%GC 47
I am trying to run BBmap with this command but I am getting this error can anyone help??
Did you create indexes for your reference? If you did you are not providing those to the command. Trying to create a SAM file from a fasta format file will result in fake Q-scores as long as you keep that in mind.
If you only have a reference (not indexes) then try:
If bbmap indexes are pre-made then: