Can clumpify.sh be used in a streaming fashion downstream of bcl-convert

Question

[tamu.anand]

Hi

I wanted to know if it is possible to use clumpify.sh in a streaming fashion - like one would do with bwa mem .... | samtools

I would like to demultiplex with bcl-convert and on the fly, I want to use clumpify.sh.

Thanks in advance.

[GenoMax]

tagging: Brian Bushnell

Answer 1

[GenoMax]

I don't think you can do that though @Brian will perhaps provide an official word. Limitation may be with bcl-convert. bcl-convert writes output to files (not sure if you can make to write to STDOUT). Then there will be the issue of paired-end data. clumpify.sh will read from STDIN (which I tested) but the data may need to be interleaved when streaming since there is only one STDIN.

[Brian Bushnell]

This is correct. Clumpify will accept stdin and work fine as long as all the data fits in memory, but it doesn't fully stream since it needs to load every single read before it can start deduplicating or sorting.

For example:

bcl-convert | clumpify.sh in=stdin.fq interleaved out=clumped.fq.gz

...that assumes input is interleaved and I don't know whether bcl-convert does that, but it might, since I think all of our files come out of demultiplexing already interleaved...

There's also the issue of multiple output streams... I don't personally run the bcl tools but if you had multiple output streams at once, I don't know how you'd pipe them all. One at a time is fine though.

[GenoMax]

since I think all of our files come out of demultiplexing already interleaved...

Only if JGI uses non-Illumina software to convert basecalls.

[Brian Bushnell]

Nope, we use Illumina software, so I guess there's a downstream process for interleaving them before they get archived. It makes tracking a lot easier when one library yields one file.

[tamu.anand]

Thanks a lot Brian and Genomax.

[tamu.anand]

Thanks @genomax. I will await Brian's guidance on this.