BBduk reading fastq from S3 directly - Is it possibile?
0
0
Entering edit mode
2.3 years ago
Marko ▴ 20

Hello to all,

I am not from Bioinf field but there is no issue for me running bbduk trimming command :)

I was wondering is it possibile to load paired fastq reads directly form S3 bucket?

Even better, load input from S3 and stream the output to S3 directly as well = ]

I find many tools are "missing" this feature now that the cloud is "default" go to. Especially for storage.

BBduk BBTools BBMap • 946 views
ADD COMMENT
0
Entering edit mode

I find many tools are "missing" this feature now that the cloud is "default" go to. Especially for storage.

I think this is because people prioritize solving problems that are objectively more difficult. Storage is not an issue for most people in this field, as one can buy a 4-10 Tb hard disk for well under $500.

ADD REPLY
0
Entering edit mode

Yes ofcorse. I did not mean to criticize. I am aware that my correspondence is blunt sometimes though :).

ADD REPLY
0
Entering edit mode

You can use a workflow orchestrator like nextflow - make sure to configure it with your aws stuff and then use nextflow's s3 storage support to write your workflow(s) - this way you can stream data right from S3 and write back to s3.

IMHO, using the fusion filesystem with Nextflow will be best as shown here

ADD REPLY
0
Entering edit mode

Hi, thanks for taking time to share. Since I don't come from bioinf side my question probably lacks context. Question is more of a "low level" nature. Lie techincal without using "third party" tools. Rather I asked it for educational purpose.

I will ask similar in a new thread hoping to make myself more clear.

ADD REPLY

Login before adding your answer.

Traffic: 2044 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6