Nextflow permission error in Singularity Cronjob

Question

[son_of_7less]

I am able to run a nextflow script from the terminal but it fails with the below error when I run it as a cron job.

Terminal command

singularity run variant.call.sif nextflow run ./pf.resistance.nf --sampleid ERR8704637 --outDir ./output/ --reads "./input/ERR8704637_{1,2}.fastq.gz" -resume

Cron job command

/usr/bin/singularity run $HOME/pfal_variant_calling/variant.sif nextflow run $HOME/pfal_variant_calling/pf.resistance.nf --sampleid ERR8704637 --outDir $HOME/pfal_variant_calling/output/ --reads "$HOME/pfal_variant_calling/input/ERR8704637_\{1,2\}.fastq.gz" -resume

Error log

    mkdir: cannot create directory ‘/home/mambauser/.nextflow’: Read-only file system
Warning: Couldn't create cached classpath folder: /home/mambauser/.nextflow/tmp/launcher/next
flow-one_22.10.1/pc_name -- Maybe NXF_HOME is not writable?
Exception in thread "main" java.lang.ExceptionInInitializerError
        at nextflow.cli.Launcher.init(Launcher.groovy:123)
        at nextflow.cli.Launcher.<init>(Launcher.groovy:84)
        at nextflow.cli.Launcher.main(Launcher.groovy:646)
Caused by: java.lang.IllegalStateException: Cannot create path '/home/mambauser/.nextflow' --
 check file system access permission
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native M
ethod)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeCon
structorAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Deleg
atingConstructorAccessorImpl.java:45)
        at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499
)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
        at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:72)
        at org.codehaus.groovy.reflection.CachedConstructor.doConstructorInvoke(CachedConstru
ctor.java:59)
        at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrap.callC
onstructor(ConstructorSite.java:84)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSite
Array.java:59)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCall
Site.java:263)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCall
Site.java:277)
        at nextflow.Const.getHomeDir(Const.groovy:111)
        at nextflow.Const.<clinit>(Const.groovy:51)
        ... 3 more

Yes, the error log makes sense because that directory is owned by root.

Nextflow creates the .nextflow directory in the directory in which it is launched which is my home directory which is mounted in the singularity container so why does it try to create it in the /home/mambauser directory of the container?

I have verified that the working directory when the container launches is my system's home directory.

Also, why does it not throw this error when launched from the command line? Perhaps there's some property of cron that I do not know.

Answer 1

[ATpoint]

mkdir: cannot create directory ‘/home/mambauser/.nextflow’: Read-only file system

This path is not writable. The recommended way of running Singularity/Apptainer with Nextflow is to actually let Nextflow handle the Singularity execution, see https://www.nextflow.io/docs/latest/singularity.html. Maybe this will solve this problem, maybe it's due to some internal singularity "thing", afair the home directory is "special" in singularity, as there is a dedicated --no-home parameter, and by default you don't have to mount (--bind) $HOME. You can test by just touching a file in this directory and see if it's possible. If so then you have to find a location that you have write permission to.

[son_of_7less]

Yes, the error log makes sense because that directory is owned by root.

Nextflow creates the .nextflow directory in the directory in which it is launched which is my home directory which is mounted in the singularity container so why does it try to create it in the /home/mambauser directory of the container?

I have verified that the working directory when the container launches is my system's home directory.

Also, why does it not throw this error when launched from the command line? Perhaps there's some property of cron that I do not know.

I know about the Nextflow -with-singularity option but I'm running on multiple files on a server and I"ll much rather have cron handle it so I don't have keep the connection to the server running the whole time.

[ATpoint]

As said, run Singularity via Nextflow as it is recommended on the documentation instead of launching Singularity manually. This should take care of it. The idea is that Nextflow itself runs outside of the container, and it only uses the container to launch the jobs (=the tools you use).

[son_of_7less]

I know about the Nextflow -with-singularity option but I'm running on multiple files on a server and I"ll much rather have cron handle it so I don't have to keep the connection to the server running the whole time. Also, I can't update the Java on the server so my best option is to run Nextflow from the container.

[ATpoint]

You do not need a running connection. NExtflow can be put to background with -bg or by submitting the nextflow job to a scheduler, or by using screen. Your choice what you do, but your non-standard way of doing this is likely cause of the issue.

[son_of_7less]

Thank you screen was what I needed.