Forum:Recommendation required for solution of computation requirements in Genomics field
2
0
Entering edit mode
3 months ago

I'm working in a Genomics services company. I want to ask for recommendations on how can we fulfil our computational requirements. We got a lot of genomics data for analysis purpose. for which we need computational power. So I wanna ask what is the best possible solution? Cloud services like Azure/ AWS or in house servers and workstation considering the limitation that we came across power breakage issue on daily basis..

DataAnalysis NGS Genomics • 1.1k views
ADD COMMENT
2
Entering edit mode

Use of cloud resources is a decision that needs to be made at the highest level especially if you work for a company. It has all sorts of legal and financial implications that need to be considered/managed by people who have the necessary authority to make these decisions. You may be one of those people but you will also have to talk with other groups (management, IT, legal), if you are serious about exploring this option. This process may take months if legal agreements are needed.

ADD REPLY
0
Entering edit mode

Use laptops and connect to AWS with Mosh. Further still run tmux on the AWS end. This is my WFH setup (except I'm using in-house servers). I don't get power blackouts but my VPN cuts off every 3 days and there's no way to re-establish the connection before it has cut itself off (what a brilliant policy!).

ADD REPLY
0
Entering edit mode

Ok thanx. will the VPN cutoff terminate the running command as well? Genome Annotation software like MAKER runs for weeks on plant data can't afford to lose connection as well.

ADD REPLY
0
Entering edit mode

No. Your programs keep running on the cloud. You just reattach/reconnect to the terminal session as needed.

ADD REPLY
0
Entering edit mode

Mosh looks interesting, as suggested above. Screen (older) and tmux (newer) are essential session tools. Nohup & is an older, and worse, suggestion.

If I had power cut issues I definitely would go to the cloud. Even with big USVs in house you're going to have massive problems with local storage and servers. Often USVs only allow controlled shutdown.

Just run through pricing scenarios first for the cloud with your bosses. Everything costs big time up there. I hope you have a good data manager to maintain important data online and sync off the rest.

ADD REPLY
0
Entering edit mode

I was just about to ask about the pricing..!! I think pricing is a major issue, especially where $ price is too high.!

ADD REPLY
0
Entering edit mode

ok that will serve the purpose then.

ADD REPLY
0
Entering edit mode

Also if you use cloud9 in AWS, your session is always there.

ADD REPLY
0
Entering edit mode

If your company has a steady stream of consistent, reliable funding, then cloud can definitely be an option (after taking into consideration the things mentioned by GenoMax ). If you are relying on grants, or have unsteady revenue, you might consider having your own data centers. After all, you dont want to lose your entire compute infrastructure after an important client contract falls through and your company can suddenly not afford AWS fees. You might also consider a hybrid approach, with some dedicated company servers plus expandable cloud space. But having your own servers or data center is not the end of it; who is going to setup and maintain them? You will need a team of people to run the data center, and there will likely be regular maintenance costs involved there as well. The "best" solution is going to be heavily tailored to your exact circumstances. If you are just starting out, it might be easier to test-drive cloud services, if only to get an estimate of the total compute power you need. From there you can decide to invest in either dedicated hardware or more cloud services for production use. In my experience, most private companies end up using a lot of cloud, while many universities and research institutes tend to use more dedicated HPC servers possibly with some cloud resources to supplement it.

ADD REPLY
0
Entering edit mode

I think the hybrid model suits us..we do have a server in place but the power cut issue is there. So, commands running for few days should be shifted on cloud and for the rest in-house servers are sufficient.

ADD REPLY
0
Entering edit mode
3 months ago
colindaven ★ 3.0k

For cloud costs, this might be helpful

https://www.sciencedirect.com/science/article/pii/S2352551719301052

However, its a shame that they concentrated on the big main platforms, which, to my knowledge are by far the most expensive.

ADD COMMENT
0
Entering edit mode
3 months ago

We have a big computataion server on site, with an uninterruptible power supply. If the power outages don't last long and you don't need internet all the time, it could be a good solution.

ADD COMMENT
0
Entering edit mode

This makes no sense. How would you connect to a single server remotely when the network is down? UPS can help with frequent power interruptions, not with outages.

ADD REPLY

Login before adding your answer.

Traffic: 1694 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6