Nextflow: No space left on device, process using output directory as input?
2
0
Entering edit mode
10 days ago
Eliveri ▴ 340

The last process of my Nextflow workflow failed with the following job.log. The workflow is run with a singularity container.

cp: error writing 'results/FILE_ST01_WGS_S1/S3TIMULATE_ST01_WGS_S1.sam': No space left on device
cp: cannot create regular file 'results/FILE_ST01_WGS_S1/FILE_ST01_WGS_S1.clean.bam': No space left on device
cp: cannot create regular file 'results/FILEST01_WGS_S1/FILE_ST01_WGS_S1.sorted.bam': No space left on device
cp: cannot create directory 'results/FILE_ST08_WGS_S8': No space left on device
...


It is a bit strange, since looking in the results directory, I could find all the directories and files which were created in previous processes. And all the previous Nextflow processes were completed without error (no errors in job.log(s)). The last process is also not very memory intensive. It is an rscript with takes some of the .txt and .tsv results from the outdir and makes a report.

There seems to be adequate space on the drive (it is a scratch drive).

The last process is to run a report which has little to do with the files and directories mentioned in the job.log errors.

The last process:

process run_report {

tag "Run quality report"

publishDir params.outdir, mode:'copy'

input:
tuple path(file_1), path(file_2)
path outdir
path rscript

output:
file('run_quality_report.html')

script:
"""
Rscript -e 'rmarkdown::render(input = "$rscript", output_dir = getwd(), params = list(directory = "$outdir"))'
"""
}


After inspecting the .command.run. It seems the process might be trying to copy ALL of the files in the outdir because outdir is an input in the 'nxf_stage()could this be the issue? And should I avoid using theoutdir as an input?

nxf_stage() {
true
# stage input files
rm -f Bam_stats_pf_Final.tsv
rm -f Bam_stats_hs_Final.tsv
rm -f results_victoria_all
rm -f run_quality_report.Rmd
cp -fRL /wynton/scratch/finterly/new_workflow/work/fe/ec06795d4f7811a7385efbc69cf4fa/Bam_stats_pf_Final.tsv Bam_stats_pf_Final.tsv
cp -fRL /wynton/scratch/finterly/new_workflow/work/ce/9c20df52f840bb21f13c48dc4df75d/Bam_stats_hs_Final.tsv Bam_stats_hs_Final.tsv
cp -fRL /wynton/scratch/finterly/results_victoria_all results_victoria_all
cp -fRL /wynton/scratch/finterly/new_workflow/run_quality_report.Rmd run_quality_report.Rmd
}

nxf_unstage() {
true
cp .command.out /wynton/scratch/finterly/new_workflow/work/f5/85eaf31ed30707d936a7fe8a8c9841/.command.out || true
cp .command.err /wynton/scratch/finterly/new_workflow/work/f5/85eaf31ed30707d936a7fe8a8c9841/.command.err || true
cp .command.trace /wynton/scratch/finterly/new_workflow/work/f5/85eaf31ed30707d936a7fe8a8c9841/.command.trace || true
[[ ${nxf_main_ret:=0} != 0 ]] && return IFS=$'\n'
for name in $(eval "ls -1d run_quality_report.html" | sort | uniq); do nxf_fs_move "$name" /wynton/scratch/finterly/new_workflow/work/f5/85eaf31ed30707d936a7fe8a8c9841 || true
done
unset IFS
}

nextflow • 496 views
1
Entering edit mode

Yes, don't use the outdir as an input. You might get infinite loops.

3
Entering edit mode
10 days ago
ATpoint 70k

That's not a nextflow error. Check whether there are per-user quotas that you might exceed on scratch. Total scratch size is often not a good metric. Check with the admins and documentation of that server.

0
Entering edit mode

2
Entering edit mode
10 days ago
Mensur Dlakic ★ 23k

I often tell people that computers have no sense of humor. The error message you got is not a prank - that is what the computer sees. So either you are wrong in what space is available (or available to you, to be more precise) in the current directory, or the program is using a temporary directory somewhere else that has ran out of space.

0
Entering edit mode

Thank you! You are correct, I took a closer look at the .command.sh in particular. It seems that by using the outdir` as input, I may be causing nextflow to copy everything?

1
Entering edit mode

Nextflow should only link inputs (unless stated otherwise), so even if you use outdir as input it should not be much of an issue, but still I'd recommend not to do so, and properly select the files or folder from the previous process that you need for an input as the next process