how to unzip the files in batch?
7
2
Entering edit mode
9.5 years ago

Hi guys, I know this question is dumb. How to convert the .fastq.gz into fastq in batches? For example, if we have 100 .fastq.gz, how can we use the command to let them to be .fastq then? Thank you

next-gen RNA-Seq SNP alignment • 55k views
ADD COMMENT
3
Entering edit mode

Why do you need to unzip the fastq files? In most cases it is better to keep them compressed. Most NGS tools can handle compressed files directly, and it is generally faster to read a compressed file than an uncompressed one.

ADD REPLY
1
Entering edit mode

Look at a few "xargs" usage on internet or simply try gunzip *.fastq.gz

ADD REPLY
0
Entering edit mode

If you have LSF (replace bsub with qsub -cwd for SGE):

ls *.fastq.gz | xargs -i echo bsub gzip -d {} | sh
ADD REPLY
9
Entering edit mode
9.5 years ago
arnstrm ★ 1.9k

GNU Parallel, FTW!

If you have access to a HPC cluster, open a interactive job session. In my case I have a 64 core node, so I do it as:

qsub -I -l nodes=1:ppn=64 -l walltime=1:00:00

then run gunzip on all 64 processors

parallel -j64 "gunzip {}" ::: *.fastq.gz

I'll be done in no time!

ADD COMMENT
7
Entering edit mode
9.5 years ago
venu 7.1k
$ gunzip *.gz

or

$ for f in *.gz; do gunzip $f; done
ADD COMMENT
2
Entering edit mode

For parallel processing (assuming more cores than jobs)

for f in *.gz; do gunzip $f & done

ADD REPLY
1
Entering edit mode

This would start as many jobs as the number of files, which is not an elegant way. Best way to do it is to use GNU Parallel as told by arnstm.

parallel --jobs <int cores> gunzip {} ::: *.fastq.gz
ADD REPLY
0
Entering edit mode

Awesome! Worked great on my 480 files

ADD REPLY
0
Entering edit mode

Isn't this the easiest way? =)

ADD REPLY
1
Entering edit mode
9.5 years ago
tomc ▴ 90

Or don't.

pass them compressed if the tool handles it ,or decompress on the fly i.e.

zcat file.gz | whatever_tool

There are a bunch of utilities for processing gzip without explicitly writing out the unzipped file

http://www.nongnu.org/zutils/manual/zutils_manual.html

ADD COMMENT
1
Entering edit mode
9.5 years ago
Vivek Todur ▴ 60

Hi,

PIGZ could be potential solution, its parallel version of GZip and uses the multi threading out of the box with so many other parameters to tweak the performance. You can find one here: http://zlib.net/pigz/

You can simply run pigz -d *.gz to extract the .gz files

Thanks

ADD COMMENT
0
Entering edit mode

pigz is indeed a very simple way to achieve such a task.

ADD REPLY
0
Entering edit mode
ADD COMMENT
0
Entering edit mode
9.5 years ago
ravi.uhdnis ▴ 220

It might do the job

gunzip '*.gz'
ADD COMMENT
0
Entering edit mode
9.5 years ago
cvu ▴ 180

gunzip *.fastq.gz will do the job!

ADD COMMENT

Login before adding your answer.

Traffic: 1865 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6