Setting up our own server for Genomic Analysis

Question

[Maverick]

Hello Everyone,

We are planning to build our very own server for genomic analysis in our university. We currently are renting from another university's facility for our computing and storage needs where we just upload our data run our pipeline and download the output.

We use 200TB of raw data storage (will require 300TB for the new one we would be building by ourselves, also planning to keep a replica file storage as back up - an additional 300 TB). We have a data pipeline that we run in an environment of 50 TB space with the sample input data being 0.5 to 1TB. The software and tools we use are - Annovar, BWA(Burrows-Wheeler Aligner), GATK, R, Picard, SPiCE, Samtools.

Generally our pipeline takes a day, sometimes 2 to complete. From my understanding we need 600 TB in total including the replica for archival storage. What would be the decent RAM required to push and pull data (a couple of TBs for processing) from this storage?

What are the server configurations that would be ideal for this? What are the questions that need to be answered for me to start building this server? What would be the problems i need to address? Any pointers, checklists or a question to start with is much appreciated?

Thank you in Advance!

Answer 1

[GenoMax]

There are two different questions here.

There is compute resources and the storage. When you are looking at over 0.5 PB of storage you need to be careful about planning for a robust system. While cost is always a prime concern going cheap can bite you big time in future (if the data is critical). Setting this up in house is going to steal time from a productive scientist unless you are going to have access to a systems administrator.

If you are already doing this on existing hardware then you minimally are going to need whatever you are using now so start there and plan for future additions.

[Maverick]

Thank you so much for your response. FYI: I made an error when mentioning about the storage details for our pipeline. But your reply still holds true. Thank you once again. I will start weighing the crtical nature of our data against our budget.