Question: (Closed) Building large RAM Computing Server with a limited budget.
1
gravatar for futurethinkers
17 months ago by
futurethinkers10 wrote:

In my institution, there is a lot of talent available, but for computing we only have a handful of rather old servers available. My expectation is that many institutions do have the budget to buy 128GB(+) RAM servers priced far north of the 10K pricing mark. Servers that are likely using the Xeon Intel technology, which might be very good processors, but perhaps not optimal, when one aims to squeeze the most out of the budget (bang for the buck).

The aim is to build a 128(+)GB RAM Computing server for less then 10K. I was wondering if anyone has any experience/tips regarding buying/building such a 'budget' server. Also, I can say that I am open to any 'hack' or 'trick' that might get the job done. Examples of this could be:

1.) Use AMD processors, instead of Intel processors. I find it hard to find out how much this can be a (cost saving) option. Pricing for servers is not very transparent.

2.) Use consumer hardware instead of server hardware. In the link, it is shown how one can get 128GB of RAM onto a normal PC motherboard: http://www.pcworld.com/article/2938855/hardcore-hardware-we-stuffed-this-pc-with-128gb-of-cutting-edge-ddr4-ram.html .. As far as I can see, using the new DDR4 RAM 8 slots motherboards plus 8x16 ECC RAM kits, this machine could be well under 3K and perhaps it is even possible to put 8 * 32GB RAM kits into such a motherboard and have a 256GB RAM Computing Server for ~3K (2 x http://tinyurl.com/h3e8om5). I also heard that Xeon processors have some advantages (e.g. larger cache), but is this really worth the extra money in our case? Also, I generally find that the servers are not running on full speed because there are too many large jobs using only one core at certain times, which might influence the overall picture. Assuming it is even a workable solution: ~3K is quite a different number then the 20K+ one might need to pay for a Xeon computing server.

3.) Cloud services: Although some of the data we would like to analyse might perhaps be best kept behind a firewall, it seems that there is a trend of using such services. It seems that people are even analyzing dbGAP data on AWS, these days, but perhaps these approaches are immature/expensive (if you take: equipment depreciation/power usage, etc. into consideration).

Also, one of this issues is that most vendors that sell server hardware do not put prices on their websites making it difficult to estimate the cost. I understand that these systems might require some service, but since all these things have to be included in the price, this will likely make it more expensive.

Thanks!

snp rna-seq next-gen genome • 978 views
ADD COMMENTlink modified 17 months ago by John11k • written 17 months ago by futurethinkers10
1

Hello futurethinkers!

We believe that this post does not fit the main topic of this site.

I do not see the connection to bioinformatics here, anyway this might be better to ask on server fault

For this reason we have closed your question. This allows us to keep the site focused on the topics that the community can help with.

If you disagree please tell us why in a reply below, we'll be happy to talk about it.

Cheers!

ADD REPLYlink written 17 months ago by Michael Dondrup43k

You don't say what kind of an institute you are at but most will have some sort of contract pricing available from major hardware vendors. You would sometimes be surprised at the discounts you can get. There are several second tier vendors that get mentioned here that can offer attractive pricing on hardware.

If your strict focus is on price then AMD chips would fit. Though to be fair I have not seen a recent review of opteron servers chips in the last couple of years. They just don't seem to be as popular. Your choice of motherboards would also be limited since not many manufacturers are making them. You have not talked about storage but I assume that has to fit within the $10K budget as well. Do you have access to server room space (since you would likely not want a dual opteron machine under your desk) and systems administration expertise?

Cloud services can be useful for someone like you since they offer the advantage of not having to maintain any local infrastructure and the elasticity of dialing up just the right amount of hardware you may need for a given job. Security for google cloud (and likely for Amazon AWS) are top notch and may be as good as what you can provide locally. Google is HIPAA (some parts are FISMA compliant) certified and they are willing to sign business associate agreements with institutions which covers your data with a legal framework. If you are in the US this may be a consideration.
Just because something can be done does not mean it is going to serve your purpose well. Don't stuff a regular PC motherboard with 128G of RAM and expect it to work great.

ADD REPLYlink modified 17 months ago • written 17 months ago by genomax33k
1
gravatar for John
17 months ago by
John11k
Germany
John11k wrote:

As soon as you want something to fit in a 1 or 2U space, the cost of everything doubles and the quality doesn't improve a whole bunch more. Yes "server-grade" components tend to be better, but is it worth it? No, not at all. The hard drives fail, capacitors pop, and RAM suddenly stops working with more-or-less the same frequency as consumer hardware. A lot of what you pay for with server-grade hardware is the peace of mind that comes with a warranty - everyone knows that "brush-less titanium-coated metorite graphite" or whatever the sales pitch is, is not going to make a more durable hard drive. The only reason these overly-priced products exist is that some contractors will get paid a % of the cost of the machine they build, and other people have sunk-cost-fallacy issues.

If you're paying for high-performance rather than high-durability, then this is even more obvious -- although to Intel/AMD's credit they're a lot more up-front about it. The difference between a 2.4Ghz octacore chip and a 2.6Ghz octacore chip is a 50% price jump. So as long as you aim for the middle-ground and aren't tempted to "upgrade", you should get a fair deal.

At the end of last year I built myself a compute PC, mainly because I was really struggling to get the software I needed installed on my institute's servers (this was before Devon and a new OPs guy joined my institute - now stuff gets updated before I even know its out of date!). Before then however, the institute servers ran a really ancient version of Red Hat Enterprise Linux with a kernel so ancient I couldn't install some modern tools like Kallisto. Other userland software that couldn't be installed as a user was 8 years + out of date, and the guy responsible for maintaining the servers at the time was very uncommunicative. It was also riddled with security holes and other uncomfortable issues. So I decided to build my own, and this is what I went for:

Motherboard: ASUS Z10PE-D8

CPUs: 2x Xeon E5-2640 v3 Octa-core @2.60 GHz (in hindsight, i wish i got the 2.4Ghz)

RAM: 8x 16Gb Samsung M393A2G40DB0-CPB

Scratch space: A few Samsung SSD 850 PRO 256GB in RAID0 (I would not buy these now, new SSD prices are crashing).

Long-term storage: x3 4Tb cheap external USB drives from the local supermarket in RAID5

Graphics Card: Some garbage i found at the bottom of a box in my cupboard

Chassis: Cooler Master Cosmos II

The whole thing cost me €3974, has 32 virtual cores @ 2.6Ghz, 128Gb of RAM

My experience so far in using it has been that it's impossible to make a good all-round compute server for Bioinformatics. Far better to make one for YOUR bioinformatics. For example, mapping reads with Bismark was totally memory-bound, using up all 128Gb of RAM, and utilizing only a handful of cores. Meanwhile, bwa-meth used up all my cores, and only 1/2 the RAM. The GATK pipeline however, really pushed that SSD RAID0 array to its limits - I was constantly having to move data around so I didn't run out of space in /tmp.

But then Devon came and the servers got totally upgraded :P hahahaha - Not that i'm complaining! This is how it should be :D

EDIT: Sorry I wrote this in the time it took Micheal to close it -- and he was right to do so in my opinion because server design/maintenance is not a bioinformatician's job.

ADD COMMENTlink modified 17 months ago • written 17 months ago by John11k
Please log in to add an answer.
The thread is closed. No new answers may be added.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 1842 users visited in the last hour