Question: bwa mem mapping multi-thread/single thread
0
gravatar for whaiyu06
2.8 years ago by
whaiyu0670
whaiyu0670 wrote:

Hi everyone :

I used the software cutadapt to remove the adapter after I received the raw data from sequencing company.(The command is:

cutadapt \
            -a AGATCGGAAGAGCACACGTCTGAACTCCAGTCAC \
            -A AGATCGGAAGAGCGTCGTGTAGGGAAAGAGTGTAGATCTCGGTGGTCGCCGTATCATT \
            -o A01_R1.clean.fastq.gz -p A01_R2.clean.fastq.gz \
        A01_R1.fastq.gz  A01_R2.fastq.gz )

After I gained the clean data,I mapped short reads of WGS to the hg19 reference using BWA mem.But but the sam file of the output didn't update after run more than 24h or more longer even the Linux Server showed program still running in the background,when I used multi-thread to run the command

nohup bwa mem -t 4 -M -R "@RG\tID:A01\tSM:A01\tPL:illumina\tLB:lib1\tPU:unit1" /public1/data/reference/public/Homo_sapiens_assembly19.fasta /public1/barcode/WGS_SC/PF_data/SingleCellAnal/A01_R1.clean.fastq.gz /public1/barcode/WGS_SC/PF_data/SingleCellAnal/A01_R2.clean.fastq.gz 1>A01.sam 2>stderr/A01.bwa.stderr &.

The log of A01.bwa.stderr using the multi-thread  as follows: 
[M::mem_pestat] mean and std.dev: (1405.67, 1625.87)
[M::mem_pestat] low and high boundaries for proper pairs: (1, 10748)
[M::mem_pestat] analyzing insert size distribution for orientation RR...
[M::mem_pestat] (25, 50, 75) percentile: (1291, 3007, 3959)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 9295)
[M::mem_pestat] mean and std.dev: (3058.62, 2334.28)
[M::mem_pestat] low and high boundaries for proper pairs: (1, 12396)
[M::mem_pestat] skip orientation FF
[M::mem_pestat] skip orientation RF
[M::mem_pestat] skip orientation RR

Strangely,when I changed single-thread, the output was fine.It looks like OK,because the log as follows:

[M::mem_pestat] (25, 50, 75) percentile: (303, 344, 394)
[M::mem_pestat] low and high boundaries for computing mean and std.dev: (121, 576)
[M::mem_pestat] mean and std.dev: (349.67, 66.67)
[M::mem_pestat] low and high boundaries for proper pairs: (30, 667)
[M::mem_pestat] skip orientation RF as there are not enough pairs
[M::mem_pestat] skip orientation RR as there are not enough pairs
[M::mem_process_seqs] Processed 14216 reads in 6.793 CPU sec, 6.725 real sec
[main] Version: 0.7.17-r1188
[main] CMD: bwa mem -M -R @RG\tID:A01\tSM:A01\tPL:illumina\tLB:lib1\tPU:NONE 
/public1/data/reference/public/Homo_sapiens_assembly19.fasta /public1/barcode/WGS_SC/PF_data/SingleCellAnal/A01_R1.clean.fastq.gz /public1/barcode/WGS_SC/PF_data/SingleCellAnal/A01_R2.clean.fastq.gz
[main] Real time: 123044.077 sec; CPU: 123276.915 sec

So,I'm confused very much if the operation of cutadapt makes the pair-end fq files incomplete.And why the single-thread seem to make no mistakes?Whether the output after run single thread command is believable? Too many questions and thanks for your time. Best

sequencing alignment • 2.4k views
ADD COMMENTlink modified 10 months ago by ATpoint42k • written 2.8 years ago by whaiyu0670

Actually you need to paste the command line you used, otherwise we can not analyze the problems. Thanks.

ADD REPLYlink written 2.8 years ago by jinxinhao198860

the command line I used is:

nohup bwa mem -t 4 -M -R "@RG\tID:A01-2m-B8\tSM:A01-2m-B8\tPL:illumina\tLB:lib1\tPU:unit1" /public1/data/reference/public/Homo_sapiens_assembly19.fasta /public1/barcode/WGS_SC/PF_data/SingleCellAnal/trimmed.A01-2m-B8_R1.fastq.gz /public1/barcode/WGS_SC/PF_data/SingleCellAnal/trimmed.A01-2m-B8_R2.fastq.gz 1>A01-2m-B8.sam 2>stderr/A01-2m-B8.bwa.stderr &
ADD REPLYlink modified 10 months ago by _r_am31k • written 2.8 years ago by whaiyu0670

Do you check the nohup.out file and stderr file to see if there is any error message?

ADD REPLYlink written 2.8 years ago by jinxinhao198860

Yes,I have checked them ,but there's nothing wrong have been reported.

ADD REPLYlink written 2.8 years ago by whaiyu0670

It is possible that a lot of stuff is happening in memory, but 24h is suspicious. What's the config on the machine you're running bwa on?

ADD REPLYlink written 2.8 years ago by _r_am31k

I used 32 threads ,8 cpu cores of the Linux server to map the reads.I don't know the other config of the machine ,but if it is necessary I'll check them.Thank you!

ADD REPLYlink written 2.8 years ago by whaiyu0670
0
gravatar for ATpoint
10 months ago by
ATpoint42k
Germany
ATpoint42k wrote:

Just to answer this after 23 month: Most likely (matching my own experience) your job got killed due to memory shortage. You can always check with top -u your_user_name if your job is still running.

ADD COMMENTlink written 10 months ago by ATpoint42k
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 1521 users visited in the last hour