slurm Exceeded step memory limit: Interleaving FastQ files

Question

[Biogeek]

Dear all,

I am interleaving FastQ files (total combined 32GB (16GB each)). I've provided the script with 1 node and 26GB RAM for processing. Basic command as below:

paste $1 $2 | paste - - - - | awk -v OFS="\n" -v FS="\t" '{print($1,$3,$5,$7,$2,$4,$6,$8)}'

No matter how much mem I give the task I get the error:

slurmstepd: error: Exceeded step memory limit at some point.

The file size seems complete (adding size of the two input .fastq files together gives resulting size of interleaved .fastq file' with an exit code of 0 (sacct) command; however, the error leaves me doubting if there will be errors in it.

I'm unsure wether this job has ran without any hiccups, or if I can rectify to reproduce no error. Any guidance?

Many thanks!

[GenoMax]

Not answering your question but suggesting an alternative. Use reformat.sh from BBMap suite. No awk/paste needed.

reformat.sh in1=R1.fq.gz in2=R2.fq.gz out=interleaved.fq.gz

[Biogeek]

Always saving the day Genomax. Thank you very much. I've used BBDUK. I'll run that now and report back!