How to allocate memory and CPU without SLURM in a linux cluster?
2
0
Entering edit mode
16 months ago

Hello,

I am running the following script in a Linux-based cluster.

for FILE in $FASTQ_DIR/*.fastq
do
    BASENAME=$(basename $FILE .fastq)
    bwa mem -t 4 -M $REFERENCE $FILE > $BAM_DIR/$BASENAME.sam
done

When I check the status after running this code, it shows that I am using only 1 CPU and 0.4% of the memory. The cluster has 256 CPUs and 512 Gb RAM.

Is there a way to specify more number of CPUs and RAM to be used for specific codes? I do not have the SLURM system installed in the cluster.

Previously I used the following code, but it did not run or do anything.

systemd-run --scope -p CPUQuota=60% -p MemoryMax=300M --user ./test.bash

Thank you

linux HPC • 1.8k views
ADD COMMENT
1
Entering edit mode

The cluster has 256 CPUs and 512 Mb RAM

You must mean 512 GB of RAM.

If you are working on a cluster then presumably it has some kind of job scheduling system. This question is best addressed to your local systems tech support.

ADD REPLY
0
Entering edit mode

Ah sorry. Yes, it is 512 GB of RAM. I addressed this and got a reply that they are working on building a job scheduling system. So at the moment, there is no such scheduling option on this cluster.

ADD REPLY
1
Entering edit mode

Looks like you already know how to limit system resources (systemd-run) based on your original post.

Please keep in mind that aligners may not use all cores/threads at all times. As long as you are specifying the right options things should work.

ADD REPLY
0
Entering edit mode
ADD REPLY
0
Entering edit mode

There is no SLURM in the cluster I am working on.

ADD REPLY
0
Entering edit mode

Oops, my bad: I saw the keyword slurm on OP's title and added it as a tag. OP mentions "without SLURM"

ADD REPLY
0
Entering edit mode

You deleted a bunch of posts that had all received feedback. Please do not do this again, or your account will be suspended.

ADD REPLY
2
Entering edit mode
16 months ago

By specifying -t 4 in the bwa command line, you are telling bwa to use 4 threads. Now, each of those threads will not neccessary max out a CPU, because at any one time, each thread might be waiting for data from disk, or waiting for another thread to finish what it is doing to provide it with the information it needs.

If you want bwa to use more CPU at once, you need to increase the number of threads it is allocated by using a higher number than 4 in your -t option. However, be aware that generally there is a point of dimishing returns on allocating more an more threads to a single task, where adding more threads doesn't help becuase you are bottlenecked by something else.

bwa will use the memory it uses - in the case of batch linux apps like this, allocating more or less memory makes no difference to speed as long as memory is sufficient that there is no swapping to disk (large linux servers usually have disk swapping disabled anyway).

However, the best way to speed this up would be to run multiple instances of bwa at once. At the moment you are running on one fastq, waiting for that to finish, and then running on the next. You would improve things by running 4 or 5 instances at once. You can do this with the parallel tool https://www.gnu.org/software/parallel/.

ADD COMMENT
0
Entering edit mode

Thank you for the reply.

ADD REPLY
0
Entering edit mode
16 months ago
Mensur Dlakic ★ 28k

Presumably your system has 512 Gb of memory, because 512 Mb would be tiny. So if you are specifying 300 Mb on command line, that would be about 0.4% of memory.

ADD COMMENT
0
Entering edit mode

No, sorry. It is 512 Gb of memory.

ADD REPLY
0
Entering edit mode

GB, not Gb. They mean different things.

ADD REPLY
0
Entering edit mode

Mensur, it should be GB, not Gb. Though Gb is not exactly fitting to this context, it is still relevant and we should be clear when correcting others.

ADD REPLY
0
Entering edit mode

Sure thing - I appreciate the pedantry as much as the next person.

Speaking of things we should be doing, I suggest that we should be professional to strangers by not telling them they are unprofessional.

ADD REPLY
0
Entering edit mode

I appreciate the feedback. I'll keep that in mind.

ADD REPLY

Login before adding your answer.

Traffic: 1877 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6