Question: samtools sort: failed to create temporary file: No space left on device
0
gravatar for lizhichao
13 days ago by
lizhichao50
lizhichao50 wrote:

Hi,I run the cwl workflow in the local machine. I write the cwl and run in [/Bigdata], but the running work seems to fill the /dev and reports that No space left on device. I want to konw why? my fastq is about 70G

   Filesystem      Size  Used Avail Use% Mounted on
udev             32G  4.0K   32G   1% /dev
tmpfs           6.3G  1.1M  6.3G   1% /run
/dev/vda1        99G   94G     0 100% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
none            5.0M     0  5.0M   0% /run/lock
none             32G     0   32G   0% /run/shm
none            100M     0  100M   0% /run/user
/dev/vdc1       1.4T 1007G  302G  77% /var/lib/docker
/dev/vdb1       296G   49G  232G  18% /home
overlay         1.4T 1007G  302G  77% 
Filesystem      Size  Used Avail Use% Mounted on
/dev/vdd1       2.0T  220G  1.7T  12% /Bigdata
cwl samtools • 172 views
ADD COMMENTlink modified 10 days ago by ATpoint19k • written 13 days ago by lizhichao50
2
gravatar for WouterDeCoster
13 days ago by
Belgium
WouterDeCoster40k wrote:

Probably because of a bunch of temporary files going to /tmp/. Samtools has a -T argument to specify where temporary files should be made.

ADD COMMENTlink written 13 days ago by WouterDeCoster40k

Wouter is absolutely right. Just for you know what to look for next time, this most likely where your /temp is:

Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        99G   94G     0 100% /

You're analysis shows you it's completely filled up. Something like this is always a warning sign and is very likely to cause system instability.

ADD REPLYlink modified 12 days ago • written 12 days ago by Carambakaracho1.4k

Thank you. my error is : samtools sort: failed to create temporary file "/BOredq/NA12878.sorted.bam.tmp.0000.bam": No space left on device And samtools sort module's "-T" option just specify the prefix instead of output dir. $samtools sort Usage: samtools sort [options...] [in.bam] Options: -l INT Set compression level, from 0 (uncompressed) to 9 (best) -m INT Set maximum memory per thread; suffix K/M/G recognized [768M] -n Sort by read name -t TAG Sort by value of TAG. Uses position as secondary index (or read name if -n is set) -o FILE Write final output to FILE rather than standard output -T PREFIX Write temporary files to PREFIX.nnnn.bam --input-fmt-option OPT[=VAL] Specify a single input file format option in the form of OPTION or OPTION=VALUE -O, --output-fmt FORMAT[,OPT[=VAL]]... Specify output format (SAM, BAM, CRAM) --output-fmt-option OPT[=VAL] Specify a single output file format option in the form of OPTION or OPTION=VALUE --reference FILE Reference sequence FASTA FILE [null] -@, --threads INT Number of additional threads to use [0]

can we specify all of cwl output dir including tmp files to running dir such as [ /Bigdata] ?

ADD REPLYlink written 12 days ago by lizhichao50

And samtools sort module's "-T" option just specify the prefix instead of output dir.

And what if you would use, as a "prefix" /Bigdata/tmp/mytemp?

ADD REPLYlink written 12 days ago by WouterDeCoster40k
1

I dont think it could work,because the running environment is virtual after staging files. If i try to specify the external path such as /Bigdata/... in command,I believe it will failed because of not finding the path.

ADD REPLYlink written 11 days ago by lizhichao50
2
gravatar for Tom
10 days ago by
Tom340
Bielefeld University, CeBiTec, Germany
Tom340 wrote:

Your cwl-runner is writing everything to a temporary directory before collecting the final output and placing it in /Bigdata.

The solution depends on which cwl-runner you are using. For cwltool, you could use the command line parameters --tmpdir-prefix and --tmp-outdir-prefix and specify a device with lots of room.

Depending on your os you could probably also bind the tmp directory to another drive with a command like mount -B /tmp /mnt/tmp.

ADD COMMENTlink modified 10 days ago • written 10 days ago by Tom340

Tank you very much! I believe this can solve my problem. I will try it as soon as poosible

ADD REPLYlink written 5 days ago by lizhichao50
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 1036 users visited in the last hour