Question: abyss-pe and abyss-map
1
gravatar for malabady
3.3 years ago by
malabady30
United States
malabady30 wrote:

Hi,

In my abyss-pe command line, I use "np=10" since I am running abyss-mpi. At the mapping stag in which abyss-pe calls abyss-map, it work only on on thread!

Here is the leading part of the abyss-pe command:

$  /usr/local/abyss/1.9.0/bin/abyss-pe  np=10  k=64 name=AbC  -d \

Here is the abyss-map command that abyss-pe executes:

$ abyss-map   -j1 -l64    P_R2_001_val_2.fq.gz P_R1_001_val_1.fq.gz Ab10_10-3.fa \
        |abyss-fixmate   -l64  -h pe150_1-3.hist \
        |sort -snk3 -k4 \
        |DistanceEst   -j1 -k64 -l64 -s200 -n10   -o pe150_1-3.dist pe150_1-3.hist

Is there any way to make abyss-pe run abyss-map multi-threaded, i.e j10 for instance? I thought that abyss-pe will always make j=np, but this is not true. I am using abyss v1.9.

Thanks.

 

 

 

 

ADD COMMENTlink modified 3.3 years ago by benv710 • written 3.3 years ago by malabady30
1

my wild guess is that abyss-pe is a wrapper to abyss-map, hence the abyss-pe exacutes 10 abyss-map commands in parallel. Can you check with 'top' and see how many programs are being executed ? If you are using np=10, either you should see 10 abyss-map processes or 1 abyss-pe process with 1000% CPU usage. If you just see one process of abyss-map with 100% CPU usage, you need to install the mpirun (openMPI) program properly. If openMPI is not configured properly, some programs silently skips the parallel running and uses single core.
 

ADD REPLYlink modified 3.3 years ago • written 3.3 years ago by geek_y9.3k

It is on abyss-map with 100% CPU usage!

But if openMPI isn't installed properly, shouldn't I have had the same problem in the earlier stages in the assembly? All steps before abyss-map were parallelized on the 10 nodes and worked just fine!  
What do you think about running " abyss-pe  np=10 j=10 ...... " ? I tried it in a dry run and abyss-map commands has j10. still didn't test this in  a real run.  


 

ADD REPLYlink written 3.3 years ago by malabady30

np=10 should ideally use 10 cores. But in past I saw few programs that silently run on single CPU, without any errors or warnings if mpi is not installed properly. I am not sure about the previous steps, but can you verify that your openmpi is installed properly ?

ADD REPLYlink modified 3.3 years ago • written 3.3 years ago by geek_y9.3k
3
gravatar for benv
3.3 years ago by
benv710
Canada
benv710 wrote:

Hi malabady,

Use `j=10` in addition to `np=10` in your abyss-pe command line.

`j` specifies the number of threads to use, for programs in the ABySS pipeline that are multithreaded (e.g. `abyss-map`).  `np` specifies the number of MPI processes (a.k.a. "MPI ranks") to `ABYSS-P`, which is the first stage of the ABySS pipeline.  `ABYSS-P` is the only program in the assembly pipeline that uses MPI and it is not multithreaded.  The remainder of the pipeline runs on only a single machine.

Edit: fix typo and clarify that `ABYSS-P` is not multithreaded.

Edit 2: Correction: Use `j=10` *in addition to* `np=10`.  Mention that the later stages of the pipeline only run on a single cluster node.

ADD COMMENTlink modified 3.3 years ago • written 3.3 years ago by benv710

I see. But if I use j=10 and not np=10, would't this affect how ABYSS-P is run at early stages of the assembly? how about passing both flags, i.e. np=10 j=10 to abyss-pe?

 

ADD REPLYlink written 3.3 years ago by malabady30

Yep, my mistake. In your case, you should use both `np=10` *and* `j=10`.  (I've corrected my answer.) The `j=10` will have no effect on `ABYSS-P` since it is not a multithreaded program.

ADD REPLYlink written 3.3 years ago by benv710
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 775 users visited in the last hour