Quickest way to extract subset of reads from huge fastq file

Question

[Prakki Rama]

Hi all,

Could I please know if there is quickest way to extract reads from a huge fastq file to another. I already tried the following.

grep -A3 '1:N:0:' ORGAN1.fastq >ORGAN1.cleaned.fastq

but grep takes too long. Any oneliners from you are very much appreciated.

Thank you

Prakki Rama.

Answer 1

[Philipp Bayer]

A faster way is to do this:

LC_ALL=C fgrep -A3 '1:N:0:' ORGAN1.fastq >ORGAN1.cleaned.fastq

Here is an explanation of why this is so much faster.

[Prakki Rama]

Wow!! Normal grep on a sample file took 17 sec, whereas LC_ALL=C just took only 4 sec. Wonderful! Thank you very much.

[Prakki Rama]

For finding the fixed string using LC_ALL=C fgrep is very fast. But when it comes to finding regex, it is slower (although slightly faster than normal grep).

[Philipp Bayer]

fgrep doesn't work with regexes (that's why it's faster), could it be that it switches to egrep or grep -g for you?

[Prakki Rama]

Yes. You are true. It does not work for regex. I was only looking only at the time of execution. My mistake.

Answer 2

[Alex Reynolds]

If you are repeatedly querying this file, try splitting it into smaller units (say, with UNIX split), and then search through the smaller files in parallel. You could do this with, say, jobs scheduled on an SGE grid, or with GNU Parallel.

[Prakki Rama]

Thank you Alex. But, I might not need repeatedly query the file. Grep is taking long time. sed's situation is also more or less seems same.

Answer 3

[da44da]

Thanks for the information shared. It was looking on the internet.

[Alex Reynolds]

Try to avoid adding an answer if you're not answering the question. You can always use the "Add Comment" button below the question or below another answer if you want to.