Configuring MAKER in SLURM HPC: WARNING: Could not get initialization lock. Trying Again...

Question

[memy]

MAKER is failing run in MPI mode and terminates with the following blocks of error messages.

WARNING: Could not get initialization lock. Trying Again...
WARNING: Could not get initialization lock. Trying Again...
WARNING: Could not get initialization lock. Trying Again...
.
.
.

It is installed properly as indicated in the tools installation document and works well when running it in non-MPI mode.

However, the non-MPI implementation takes a considerable time when dealing with annotating chromosome-scale genome assemblies (plant genomes in my case). I have tried the non-MPI implementation running each chromosome (60Mb - 80Mb) separately and it's taking forever to finish the first round.

I really appreciate suggestion and workaround on this issue.

[GenoMax]

What kind of hardware are you running this on? You need to have the right kind of hardware (multiple CPU's not just cores), MPI libraries and an OS that will support this.

[memy]

It is on a HPC cluster (SLURM-based) and MAKER is installed centrally from easybuild and configured with OpenMPI/4.1.4. I also installed it locally in my HPC space similarly using the latest version obtained from the tool's lab page and I'm getting exactly the same error message.

Here' how I launch MAKER after loading necessary modules and providing input files in the configuration file (maker_opts.clt) and making sure that all supporting tools installed and captured in the config maker_exe.ctl file.

MAKERDIR="sample_00"
mpiexec -n 'tot_cpus_specified_across_mutiple_cores' maker -qq -base ${MAKERDIR} -fix_nucleotides

So I am submitting this to multiple nodes (--nodes SLURM) and specify the total number of cpus across the nodes used in -n flag above.

As I mentioned in initial post, it works well when submitting an instance of MAKER job to a single node (say 48 cpus in it). I will then run another instance of MAKER job after the first job was started and similarly run multiple MAKER jobs each to a single node. In this case what seems to be happening is each sequences of the genome (say 12 chromosomes) is run separately each on a single core and will utilize the cpus available in it (indicated in cpu param of maker_opts.ctl config file). But I found that it's not helping much.

I am not sure how the MPI implementation of MAKER works in simplistic term assuming that we have only one sequences (one chromosome). Whether it spits the job to multiple cores and combine it eventually (which is what I am expecting maker to do haha) or it assigns it to a single node and when it detect multiple sequences then submit each to a single node?