Tutorial:Installing multiprocessor-capable MEME to work with a Slurm job scheduler environment
0
5
Entering edit mode
4.2 years ago

There was some local interest in running a parallelized version of the meme binary in the MEME suite (http://meme-suite.org/). Many moons ago, I had to do this for an environment that ran under a Sun Grid Engine and posted a question here about it.

MEME has undergone some revisions and we've changed job schedulers, away from SGE and to Slurm. Recently, I put together a makefile that demonstrates how to build an OpenMPI-capable meme binary (one that can run meme -p <num_processors>) and includes some targets that test this binary locally and on a cluster that uses a Slurm job scheduler.

This assumes that the end user has a cluster that is managed with a Slurm job scheduler, and that the Slurm nodes use the modules system to manage software packages.

Starting with a {data,results,src}-structured project folder, one could put the following makefile in the src folder and run targets.

One can make the download, extract, and build targets to compile MEME 4.12.0 with OpenMPI support.

For my setup, it is necessary to hardcode paths to Ghostscript gs and ImageMagick convert, as the module system puts these binaries somewhere other than /usr/bin, which is where meme would otherwise look. Depending on your module setup, this may not be required.

Changing the cluster partition on which jobs will run will also almost certainly be a necessity.

Review the test_local and test_slurm targets to see how you can run meme -p "locally" (on a host that also has modules support and multiple cores/processors) and on a Slurm-based cluster.

The Slurm-focused test_slurm target creates ../results/slurm_32 and ../results/slurm_64 directories. The stderr.txt file in each directory returns their runtimes.

It may be useful to test adjustment of Slurm nodes, ntasks-per-node and cpus parameters to match the -p <N> argument and review overall performance. This will be specific to the local environment.

Hope this is useful to others!

meme slurm parallelization parallel Tutorial • 1.6k views