blast with multithreads is much slower
0
0
Entering edit mode
4 weeks ago
biobiu ▴ 130

Hello! I'm running blastn (megablast) using two jobs. One without multithreading (-num_threads 1) and the one with 30 threads (-num_threads 30). Unexpectedly the run with the multithreads seems to be much much slower and finished to align about 10% of the records that the one thread run has finished. The command and the input fasta and DB are the same for both runs. The two jobs were given same resources expect of the number of cores. It's very weird, I suspect it might be related to the ram allocated per thread. Any other ideas? BTW, I plan to split the fasta query and send differnt jobs with lower threads to make it work faster, but just curious from this unexpected behaviour

blast • 264 views
ADD COMMENT
0
Entering edit mode

some random experiences:

  • only few steps of the blast algorithm is actually multi-threaded (the majority is single-threaded )
  • there is a plateau on the number of cores (I think it was something like 4 or 8, above it very little to no speed gain is expected)
  • it can be that you created way to much overhead when running it on 30 cores (compared to the one core)
ADD REPLY
0
Entering edit mode

Thanks. Yes I plan to split and to use max 8 threads in each job. Anyway it is weird that 30 threads are 90% slower

ADD REPLY
0
Entering edit mode

that is indeed considerably slower, in the sense that I expect some other bottleneck as well. Perhaps there is some network latency accessing the (central) DB?

ADD REPLY
0
Entering edit mode

Simply throwing multiple threads at something does not always make process go fast. Besides reasons noted by lieven.sterck you may have I/O bottlenecks on your machine where the cores may be waiting to get data from disk/memory. Starting multiple blast processes may make things worse since now more than one job will compete for the available bandwidth/throughput.

NCBI has extremely beefy architecture behind web blast and that is one reason they can run things quickly.

ADD REPLY
0
Entering edit mode

Of course. Things went slow with one thread so I hoped to speed things up a little bit with multithreading. The surprise was not from not getting much faster but getting much slower. It's indeed might be a IO bottleneck. Tries to understand why it happen to get better decisions. Thanks

ADD REPLY

Login before adding your answer.

Traffic: 1656 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6