Question: Using Amazon Web Services?
15
gravatar for Peter
8.9 years ago by
Peter170
Peter170 wrote:

Hello,

I am trying to use amazon web services for bioinformatics. I created a linux EC2 instance, but I've found that the space is limited to 10gB. I am attempting to take advantage of the EBS to create a larger hard drive space for me to work with. I am using the AWS console to attach the EBS to EC2 instance, but cannot figure out how to use the attached storage. Can anyone help me with the steps to make this all work?

Thanks!

cloud • 3.3k views
ADD COMMENTlink modified 8.4 years ago by Casbon3.2k • written 8.9 years ago by Peter170
1

10GB space sounds woefully inadequate to any modern computer. Are you sure you have it right? Check here http://aws.amazon.com/ec2/instance-types/

ADD REPLYlink written 8.9 years ago by Istvan Albert ♦♦ 81k

This tutorial post might be useful: Introduction to AWS Cloud Computing

ADD REPLYlink written 4.4 years ago by Malachi Griffith17k
18
gravatar for Alex
8.9 years ago by
Alex1.4k
Theodosius Dobzhansky Center for Genome Bioinformatics
Alex1.4k wrote:

Useful EC2 tutorials for bioinformatics:

  1. Getting started guide: http://docs.amazonwebservices.com/AWSEC2/latest/GettingStartedGuide/
  2. Getting started JAVA: http://aws.amazon.com/articles/Amazon-EC2/3586
  3. Getting started Python: http://aws.amazon.com/articles/Amazon-EC2/3998
  4. EBS (this is MySQL example, but it is very useful and solves your current problems): http://aws.amazon.com/articles/Amazon-EC2/1663
  5. Hadoop tutorial: http://aws.amazon.com/articles/Amazon-EC2/873

To check available space try this:

[root@domU-12-31-39-00-B2-01 ~]# df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1             10321208   1636668   8160252  17% /
/dev/sda2            153899044    192072 145889348   1% /mnt

10Gb is the size of root disk, "other" disk space is in the /mnt directory.

For data analysis I usually use instance disk space ~1Tb. Note that all data will be deleted when you delete instance. And I save all important data to EBS (permanent storage). If you use US zone then network connection with NCBI is fantastic so expensive data keeping in permanent EBS is not necessary. Alternatively you can use public bioinformatics EBS, see full list here http://aws.amazon.com/datasets?_encoding=UTF8&jiveRedirect=1

Available biological public EBS:

  1. 1000 genomes
  2. Human Liver Cohort (Sage Bionetworks)
  3. C57BL/6J by C3H/HeJ Mouse Cross (Sage Bionetworks)
  4. Ensembl Annotated Human Genome Data - for FASTA
  5. Ensembl Annotated Human Genome Data - for MySQL
  6. Illumina - Jay Flatley (CEO of Illumina) Human Genome Data Set
  7. GenBank
  8. YRI Trio Dataset
  9. Ensembl - FASTA Database Files
  10. Unigene
  11. and more
ADD COMMENTlink modified 15 days ago by RamRS24k • written 8.9 years ago by Alex1.4k
1

Just one suggestion: df -h would be more readable :)

ADD REPLYlink modified 15 days ago by RamRS24k • written 8.8 years ago by Michael Schubert6.9k

Additionally, if you boot of an EBS-backed image, you can have up boot volumes up to 10 TB

ADD REPLYlink written 8.9 years ago by Mndoci1.2k

Just realized I wrote 10 TB .. make that 1 TB

ADD REPLYlink written 8.4 years ago by Mndoci1.2k
1
gravatar for Marina Manrique
8.4 years ago by
Marina Manrique1.3k
Granada
Marina Manrique1.3k wrote:

Maybe you just have to mount the EBS volume,

Here you can get very basics instructions to mount an attached volume to the instance.

Once you've attached the volume to the instance you just have to mount it with "mount" command

HTH

Marina

ADD COMMENTlink written 8.4 years ago by Marina Manrique1.3k
1
gravatar for Casbon
8.4 years ago by
Casbon3.2k
Casbon3.2k wrote:

If you use an instance store based volume, rather than a EBS backed one, you will have ~300GB to play with at /mnt. However, it will not be persisted when the instance is deallocated.

Since you have 10GB, you are on EBS. This means you will need to attach a larger EBS at /dev/sdx and mount it as Marina points out.

ADD COMMENTlink written 8.4 years ago by Casbon3.2k
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 2312 users visited in the last hour