Can I use external hard drive to increase the speed of R processes (increase virtual memory?

Question

[Sara]

Hi everyone, I want to do an analysis of RNAseq data of 150 patients and I have only 8GB RAM. I want to know is it possible to process the raw data using this RAM. I will buy a 5TB HDD. I was thinking to increase the virtual memory but I do not know will it work or not. Any suggestion?

[Michael]

Hi Sara,

Let me first say that this is not a technical but an organizational problem. The core problem is to find available compute resources that are already up and running for free. Most likely, the solution is around the corner. The best way to approach this problem depends on your employer and your position. If you are in academia, a student or Ph.D. student or post-doc in a research group at a University, ask your group leader or supervisor. Often, hospitals have their own computational resources as well.150 (new?) patient samples indicate a larger project which should have funding. Most universities and computational research groups have computers and servers you can use. If students require to use compute resources for their projects, these should be provided (in the same way you are not supposed to build up your own wet-lab).

Then there are national HPC resources researchers may be entitled to use. In Norway, we have Uninett Sigma2, which we can use, and costs are billed to the university partners. I am pretty sure there is something similar or much better in the UK.

Last resort: buy new hardware or use cloud-based solutions (elastic cloud etc.), but that means you will have to pay yourself or from the project. And you may not be allowed to transfer the data that way.

Using your employer's resources will also make sure you comply with the local requirements and regulations for e.g. data management and privacy regulations.

Best of luck with your project.

[GenoMax]

Hate to say it but this is not going to work (with perhaps following caveats). subread may be the only aligner capable of running in 8G of RAM for human size genomes. Aligning to transcriptome (without decoys ~4 G) using salmon would be possible but not with genome decoy (that needs ~20G RAM).

You can only process one patient sample at a time so it will take a while to do all 150. Then you may not be able to analyze the count data that point on anyway.

Find alternate hardware (or use cloud if your local institutional policies allow for it considering this is patient data).