Can clumpify.sh be used in a streaming fashion downstream of bcl-convert
1
0
Entering edit mode
3 months ago
tamu.anand ▴ 30

Hi

I wanted to know if it is possible to use clumpify.sh in a streaming fashion - like one would do with bwa mem .... | samtools

I would like to demultiplex with bcl-convert and on the fly, I want to use clumpify.sh.

Thanks in advance.

bbmap clumpify bclconvert • 766 views
ADD COMMENT
0
Entering edit mode

tagging: Brian Bushnell

ADD REPLY
2
Entering edit mode
3 months ago
GenoMax 139k

I don't think you can do that though @Brian will perhaps provide an official word. Limitation may be with bcl-convert. bcl-convert writes output to files (not sure if you can make to write to STDOUT). Then there will be the issue of paired-end data. clumpify.sh will read from STDIN (which I tested) but the data may need to be interleaved when streaming since there is only one STDIN.

ADD COMMENT
1
Entering edit mode

This is correct. Clumpify will accept stdin and work fine as long as all the data fits in memory, but it doesn't fully stream since it needs to load every single read before it can start deduplicating or sorting.

For example:

bcl-convert | clumpify.sh in=stdin.fq interleaved out=clumped.fq.gz

...that assumes input is interleaved and I don't know whether bcl-convert does that, but it might, since I think all of our files come out of demultiplexing already interleaved...

There's also the issue of multiple output streams... I don't personally run the bcl tools but if you had multiple output streams at once, I don't know how you'd pipe them all. One at a time is fine though.

ADD REPLY
1
Entering edit mode

since I think all of our files come out of demultiplexing already interleaved...

Only if JGI uses non-Illumina software to convert basecalls.

ADD REPLY
0
Entering edit mode

Nope, we use Illumina software, so I guess there's a downstream process for interleaving them before they get archived. It makes tracking a lot easier when one library yields one file.

ADD REPLY
0
Entering edit mode

Thanks a lot Brian and Genomax.

ADD REPLY
0
Entering edit mode

Thanks @genomax. I will await Brian's guidance on this.

ADD REPLY

Login before adding your answer.

Traffic: 1097 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6