run "sra-human-scrubber" with SLURM

Question

[emmanouil.a]

Hi, about sra-human-scrubber:

if I run in a terminal

./scrub.sh test

it works and I have as output

2022-07-30 15:55:18     aligns_to version 0.60
2022-07-30 15:55:18     hardware threads: 80, omp threads: 80
2022-07-30 15:55:19     loading time (sec) 0
2022-07-30 15:55:19     /tmp/tmp.q0E2fhyiOM/temp.fasta
2022-07-30 15:55:19     100% processed
2022-07-30 15:55:19     total spot count: 2
2022-07-30 15:55:19     total read count: 2
2022-07-30 15:55:19     total time (sec) 0

but when I run the same in a SLURM submitted job

#!/usr/bin/env bash
#SBATCH --cpus-per-task=6
#SBATCH --mem=25G
#SBATCH --time=02:00:00

./scrub.sh test

I have the following output

2022-07-30 15:55:35     aligns_to version 0.60
2022-07-30 15:55:35     hardware threads: 80, omp threads: 6
2022-07-30 15:55:35     loading time (sec) 0
2022-07-30 15:55:35     /tmp/tmp.7PZUfM7UT2/temp.fasta
2022-07-30 15:55:35     0% processed
2022-07-30 15:55:35     100% processed
2022-07-30 15:55:35     total spot count: 0
2022-07-30 15:55:35     total read count: 0
2022-07-30 15:55:35     total time (sec) 0
0  spot(s) removed.

I do not have any error. How can I make it works?

[GenoMax]

I am going to guess that running test copies some file from the software bundle and then runs it. Depending on how your slurm cluster is setup the folder with the software may not be reachable from worker node where your job runs.

If you provide a different test file that you know has data then does scrub.sh work under SLURM. If it is does then above is your answer.

[emmanouil.a]

Hi, here testing the code using real samples (the tool's files have 770 permissions):

1) run directly in the terminal: **OK**
2) create a file locally "touch", add the code and run through it in the terminal: **OK**
3) create a file locally "touch", add the code and SLURM options, submit to SLURM: ***problem as in my original code here post***
4) open a SLURM interactive job and run the script as in 1) and 2): **OK**
5) run directly in the terminal many fastq files with a loop "for": **OK**
5) run directly in the terminal many fastq files with a loop "while": **problem as in my original code here post***

[GenoMax]

If you have SLURM on your cluster you should be using that no matter. Usefor or while to submit SLURM jobs but not for actually running analyses. Error with number 6 may simply be some logic issue with your while statement.

Thank you for trying various combinations. Number 3 above tells me that your cluster is setup in some specific way and worker nodes are still not able to reach the software folder. It may be best to contact your local system admins with the above set of data.

[emmanouil.a]

Thank you GenoMax.The Admin is on holidays and I'm at junior level for those thinks. I will try with "GNU parallel" but I will expect problems with the multithreading. Seems that the code "align_to" is using multithreading (whole system cpus without limitations) and it is out of my control.