Parallelizing Read Trimming/Filtering And Genome Assembly!!
2
0
Entering edit mode
10.9 years ago
Rahul Sharma ▴ 660

Hi all,

I want to make a parallel shell script, which could run the trimming processes parallel on different processors and later do the genome assembly. I am trying the following script:

Before doing this, one could also split the whole read file into small chunks (using split command) and use for loop to do the trimming on multiple processors.

#!/usr/bin/bash

STEP1: Library1 read Filtering/Trimming & # Use any software of your choice

STEP2: Library2 read Filtering/Trimming &

STEP3: Library3 read Filtering/Trimming &

wait

velveth demo 65 -short -fastq Library1 -short2 -fastq Library2 -short3 -fastq Library3

velvetg demo -exp_cov auto

Dose anybody have better way to write this? Which would also take care of successful completion of first 3 steps. using their process Ids.

I would really appreciate your inputs. Thanks in advance!

Best regards,

Rahul

parallel trimming assembly • 2.8k views
ADD COMMENT
2
Entering edit mode
10.9 years ago

Your 2 inquiries are quite different:

1 - the trimming step is very easily parallelizable as you don't need the information in the other sequences to process one, I think the best way to do it in bash would be by using GNU parallel (take a look here: GNU Parallel - parallelize serial command line programs without changing them ) . However, I've been processing quite a lot of big datasets but I've always figured it was fast enough using a single processor and I'm wondering if this worths doing it in parallel.

2 - the assembly step is much more complex and I think you cannot parallelize an assembler which has not been designed for. Until recently I did not know about any assembler capable to do it, but now there is Ray and maybe (but not sure it'd fit your needs) Minia, but I personally don't have any experience with any of those.

ADD COMMENT
1
Entering edit mode
10.9 years ago

use a makefile with option -j (num parallel jobs)

VELVET2: VELVET1
    velvetg demo -exp_cov auto

VELVET1: Library1 Library2 Library3
    velveth demo 65 -short -fastq Library1 -short2 -fastq Library2 -short3 -fastq Library3


Library1:
       statements1

Library2:
       statements2

Library3:
       statements3
ADD COMMENT

Login before adding your answer.

Traffic: 1764 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6