Entering edit mode
3.0 years ago
hafiz.talhamalik
▴
350
I'm working in a Genomics services company. I want to ask for recommendations on how can we fulfil our computational requirements. We got a lot of genomics data for analysis purpose. for which we need computational power. So I wanna ask what is the best possible solution? Cloud services like Azure/ AWS or in house servers and workstation considering the limitation that we came across power breakage issue on daily basis..
Use of cloud resources is a decision that needs to be made at the highest level especially if you work for a company. It has all sorts of legal and financial implications that need to be considered/managed by people who have the necessary authority to make these decisions. You may be one of those people but you will also have to talk with other groups (management, IT, legal), if you are serious about exploring this option. This process may take months if legal agreements are needed.
Use laptops and connect to AWS with Mosh. Further still run tmux on the AWS end. This is my WFH setup (except I'm using in-house servers). I don't get power blackouts but my VPN cuts off every 3 days and there's no way to re-establish the connection before it has cut itself off (what a brilliant policy!).
Ok thanx. will the VPN cutoff terminate the running command as well? Genome Annotation software like MAKER runs for weeks on plant data can't afford to lose connection as well.
No. Your programs keep running on the cloud. You just reattach/reconnect to the terminal session as needed.
Mosh looks interesting, as suggested above. Screen (older) and tmux (newer) are essential session tools. Nohup & is an older, and worse, suggestion.
If I had power cut issues I definitely would go to the cloud. Even with big USVs in house you're going to have massive problems with local storage and servers. Often USVs only allow controlled shutdown.
Just run through pricing scenarios first for the cloud with your bosses. Everything costs big time up there. I hope you have a good data manager to maintain important data online and sync off the rest.
I was just about to ask about the pricing..!! I think pricing is a major issue, especially where $ price is too high.!
ok that will serve the purpose then.
Also if you use cloud9 in AWS, your session is always there.
If your company has a steady stream of consistent, reliable funding, then cloud can definitely be an option (after taking into consideration the things mentioned by GenoMax ). If you are relying on grants, or have unsteady revenue, you might consider having your own data centers. After all, you dont want to lose your entire compute infrastructure after an important client contract falls through and your company can suddenly not afford AWS fees. You might also consider a hybrid approach, with some dedicated company servers plus expandable cloud space. But having your own servers or data center is not the end of it; who is going to setup and maintain them? You will need a team of people to run the data center, and there will likely be regular maintenance costs involved there as well. The "best" solution is going to be heavily tailored to your exact circumstances. If you are just starting out, it might be easier to test-drive cloud services, if only to get an estimate of the total compute power you need. From there you can decide to invest in either dedicated hardware or more cloud services for production use. In my experience, most private companies end up using a lot of cloud, while many universities and research institutes tend to use more dedicated HPC servers possibly with some cloud resources to supplement it.
I think the hybrid model suits us..we do have a server in place but the power cut issue is there. So, commands running for few days should be shifted on cloud and for the rest in-house servers are sufficient.