Question: Job management programs
1
gravatar for evcon
4 days ago by
evcon10
evcon10 wrote:

I’m moving from a HPC that had slurm for job scheduling to one with no job management program. Is there a program that can sorta manage lots of parallel jobs kinda like personal slurm? I’m using screen but it’s kinda annoying, I’d like to start a lot of jobs and log off and come back later.

linux • 101 views
ADD COMMENTlink modified 3 days ago by dariober10k • written 4 days ago by evcon10

You can setup slurm on a single node if you'd like.

ADD REPLYlink written 4 days ago by Devon Ryan92k

Hi Devon, would you mind expanding on that? I mean, I know you can install slurm on a single node, but is it something reasonably doable by someone without admin rights and without investing a fair bit of effort maintaining slurm over time?

ADD REPLYlink written 3 days ago by dariober10k

No, you'd need admin rights :(

ADD REPLYlink written 2 days ago by Devon Ryan92k

to one with no job management program

Do you mean to say that it is a "free for all" situation i.e. you could occupy the entire cluster with your jobs without consideration to others? Surely some form of control over user processes is in place e.g. via limits per user account /etc/security/limits.conf?

ADD REPLYlink written 3 days ago by genomax74k
1
gravatar for Jean-Karim Heriche
4 days ago by
EMBL Heidelberg, Germany
Jean-Karim Heriche21k wrote:

Look into scientific workflow management systems.

ADD COMMENTlink written 4 days ago by Jean-Karim Heriche21k
1
gravatar for dariober
3 days ago by
dariober10k
WCIP | Glasgow | UK
dariober10k wrote:

Depending on how much effort you want to invest, you could run your program(s) via snakemake and let snakemake handle parallelism and job dependencies. There will be a learning curve but in my opionion is totally worth it.

Alternatively, with xargs (available on *nix systems) you can run programs in parallel. For example, run the list of bash scripts captured by ls, run up to 8 in parallel:

ls job_number.*.sh | xargs -P 8 -n 1 bash

parallel is an alternative to xargs, arguably more powerful than xargs.

ADD COMMENTlink written 3 days ago by dariober10k
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 1221 users visited in the last hour