Question: Job management programs
1
gravatar for evcon
10 months ago by
evcon10
evcon10 wrote:

I’m moving from a HPC that had slurm for job scheduling to one with no job management program. Is there a program that can sorta manage lots of parallel jobs kinda like personal slurm? I’m using screen but it’s kinda annoying, I’d like to start a lot of jobs and log off and come back later.

linux • 308 views
ADD COMMENTlink modified 8 months ago by Malte Petersen10 • written 10 months ago by evcon10
1

to one with no job management program

Do you mean to say that it is a "free for all" situation i.e. you could occupy the entire cluster with your jobs without consideration to others? Surely some form of control over user processes is in place e.g. via limits per user account /etc/security/limits.conf?

ADD REPLYlink written 10 months ago by genomax90k

You can setup slurm on a single node if you'd like.

ADD REPLYlink written 10 months ago by Devon Ryan96k

Hi Devon, would you mind expanding on that? I mean, I know you can install slurm on a single node, but is it something reasonably doable by someone without admin rights and without investing a fair bit of effort maintaining slurm over time?

ADD REPLYlink written 10 months ago by dariober11k

No, you'd need admin rights :(

ADD REPLYlink written 10 months ago by Devon Ryan96k
1
gravatar for Jean-Karim Heriche
10 months ago by
EMBL Heidelberg, Germany
Jean-Karim Heriche23k wrote:

Look into scientific workflow management systems.

ADD COMMENTlink written 10 months ago by Jean-Karim Heriche23k
1
gravatar for dariober
10 months ago by
dariober11k
WCIP | Glasgow | UK
dariober11k wrote:

Depending on how much effort you want to invest, you could run your program(s) via snakemake and let snakemake handle parallelism and job dependencies. There will be a learning curve but in my opionion is totally worth it.

Alternatively, with xargs (available on *nix systems) you can run programs in parallel. For example, run the list of bash scripts captured by ls, run up to 8 in parallel:

ls job_number.*.sh | xargs -P 8 -n 1 bash

parallel is an alternative to xargs, arguably more powerful than xargs.

ADD COMMENTlink written 10 months ago by dariober11k
1
gravatar for Malte Petersen
8 months ago by
Freiburg, Germany
Malte Petersen10 wrote:

You want the task spooler, a "personal batch system" that doesn't require root privileges. You define how many threads your system can run in parallel and then just submit jobs to the queue. The spooler starts the jobs as soon as enough cores are free.

It's deliberately simple and only considers parallel threads, not memory or other resources. It doesn't send emails by default, but you can script what should happen when a task finishes (including sending an email). As a personal "submit and forget" replacement for a full-fledged job manager, I liked it a lot.

ADD COMMENTlink written 8 months ago by Malte Petersen10
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 1703 users visited in the last hour