Question: So frustrated compiling necessary packages in a linux system
4
gravatar for johnnytam100
8 weeks ago by
johnnytam100100
johnnytam100100 wrote:

I have just moved to a new lab and have to deal with installing all necessary packages for bioinformatics in a linux system again.

As I have experienced in the previous lab, compiling packages is a pain as error always appear and prerequisites seem endless.

My obstacles:

1) I do not have root permission

2) I cannot use sudo or yum or even git etc. (as either I do not have root permission or they were not installed)

I want to learn how to compile everything with ease in a linux system.

May I have not only the 'how-tos' but also some rule of thumbs to follow, starting from scratch?

The linux system in the new lab is a CentOS7 (centos-release-7-5.1804.4.el7.centos.x86_64)

Thanks a lot!

linux build compile • 410 views
ADD COMMENTlink modified 4 days ago by amyloid.beta.martin10 • written 8 weeks ago by johnnytam100100

Between conda, docker and brew, you can install pretty much everything without root permissions.

ADD REPLYlink written 8 weeks ago by jrj.healey9.1k
1

linuxbrew/homebrew at least need git installed

ADD REPLYlink written 8 weeks ago by cmdcolin950

I have installed bioconda right now ;) Do you think I need the other two?

ADD REPLYlink written 7 weeks ago by johnnytam100100

Docker might be useful too. The trend seems to be for bioinformatics packages to be in those two managers lately. Brew is not as popular as it once was.

Conda will have you covered for a significant number of tools alone though.

ADD REPLYlink written 7 weeks ago by jrj.healey9.1k

Alternatively, you may consider installing GVL or BioLinux as your base install image, which comes with a significant amount of software preconfigured.

ADD REPLYlink written 8 weeks ago by jrj.healey9.1k

How do you compare BioLinux with Ubuntu? I am planning to build my own linux machine and now I'm choosing between the two!

ADD REPLYlink written 7 weeks ago by johnnytam100100

BioLinux is Ubuntu, it just ships with a lot of preinstalled bioinformatics packages. I’m not sure whether the project is still ongoing/supported, but there are images available nonetheless.

To my knowledge you shouldn’t lose any of the ‘ubuntu Experience’ on BioLinux

ADD REPLYlink written 7 weeks ago by jrj.healey9.1k

My preferred option as a non root user is to find the programs at github, download, tar unzip and configure, make.

KK

ADD REPLYlink written 8 weeks ago by karthic90

I did that actually... maybe the core of my problem is I got errors saying prerequisites are not installed, which needs root permission.

ADD REPLYlink written 7 weeks ago by johnnytam100100
7
gravatar for shenwei356
8 weeks ago by
shenwei3564.3k
China
shenwei3564.3k wrote:

you need bioconda: https://bioconda.github.io/

ADD COMMENTlink written 8 weeks ago by shenwei3564.3k

+1 for the bioconda suggestion. It really has made my work easier.

ADD REPLYlink written 8 weeks ago by James Ashmore2.5k

Many thanks! I have installed bioconda now! Great package!

ADD REPLYlink written 7 weeks ago by johnnytam100100
5
gravatar for dariober
8 weeks ago by
dariober9.7k
Glasgow - UK
dariober9.7k wrote:

I'm not sure mine is a good advice... I usually install programs under my home directory. For programs whose installation is based on make, usually this works:

./configure --prefix=$HOME
make
make install
..etc

For python packages, usually this works:

pip install --user <my package>

For R packages, install.packages(...) prompts you to install in a user directory if the default one is read-only.

And so on... I installed all sorts of things in my home directory, postgres, git, python etc... But yes, be prepared for some frustration and a lot of scanning of StackOverflow posts (but it's a good learning experience anyway)!

ADD COMMENTlink written 8 weeks ago by dariober9.7k

I guess, when you are not root, you cannot do make install I guess. But still the programs would have been compiled under that user, so you can actually use them by giving the full path of the program.

However if you do not want to run the programs like that, you can export the path.

KK

ADD REPLYlink written 8 weeks ago by karthic90
1

Hi- the point of --prefix=/my/dir is to instruct make install to put files and dependencies under /my/dir, where /my/dir is a directory where you have full permissions, like your home directory. So make install shouldn't complain. If then $HOME/bin is in your PATH (and it should be, if not add it) executables will be available from anywhere you call them.

The disadvantage of installing stuff in your home, in my experience, is that you may end up with a mixture of system-wide and user dependencies which can cause headaches to troubleshoot (I do have some comic-horror stories to tell...) but as I said, it may be a good learning exercise in itself.

ADD REPLYlink written 8 weeks ago by dariober9.7k
1

On most Linux distributions, if you make a bin directory under ~, it will be automatically added to PATH for you ;)

ADD REPLYlink written 8 weeks ago by jrj.healey9.1k

You shouldn’t need root to use make.

ADD REPLYlink written 8 weeks ago by jrj.healey9.1k

Thanks for the comment! I always got errors stating prerequisites are required. Unfortunately, I cannot install those prerequisites without root permission...

ADD REPLYlink written 7 weeks ago by johnnytam100100
3
gravatar for amjadcsu
8 weeks ago by
amjadcsu80
Saudi Arabia
amjadcsu80 wrote:

Do you have your own linux box where you are root.?

If yes I would recommend learning singularity. Build a singularity image on your own linux box and then copy and run it on lab machine . https://www.sylabs.io/singularity

ADD COMMENTlink written 8 weeks ago by amjadcsu80

Second this, and add that with VM you can run Linux OS on any local machine and so have access to root. Then test/build your Singularity recipe there, upload and pull from Singularity-Hub as user. Such a good system!

ADD REPLYlink written 8 weeks ago by bruce.moran420
1

May I know what is special with singularity? Thanks!

ADD REPLYlink written 7 weeks ago by johnnytam100100
1

Singularity is a 'container', which is basically a portable computing environment. I think of it like a virtual machine, but at a software level. Containers are built based on 'recipes' which define the environment, including setup of all libraries, software, tools, reference data etc. just like you are installing to your server as root. Containers can be hosted on Singularity-Hub from where you can 'pull' to your local machine as user (no root required). And once defined, and used on a project, the container is available to anyone so that your workflow is completely reproducible, down to the exact compute environment. That's the really special part IMO.

NB building containers on server requires root (pulling from hub does not). Install of Singularity (available for *nix, OsX, Win) to the server also requires root, but the case for a sysadmin to install is very straightforward as it basically cuts out a lot of work for them! Also NB that you can use (bio)conda in a container recipe.

Once you have pulled/built the container on your server, you have an image that you can specify to execute jobs inside. You can shell into it like it is another server. And so your server doesn't have e.g. aligners like BWA or STAR installed, your container image does, and you therefore align your fastq in the container, with all output going back out of the container to your server.

ADD REPLYlink written 7 weeks ago by bruce.moran420
1
gravatar for amjadcsu
8 weeks ago by
amjadcsu80
Saudi Arabia
amjadcsu80 wrote:

As we speak few days ago, singularity 3.0 came out and it has remote builder option. You do not need to be root or even own a linux box to build singularity image on cloud . https://cloud.sylabs.io/builder Is this the magic wand that just allows bioinformatician to build/test/run their pipelines without having to worry about sys admins :)

ADD COMMENTlink written 8 weeks ago by amjadcsu80

How do you compare singularity with other linux OS e.g. ubuntu and biolinux? I have a spare computer and would really like to build my own linux machine. I am choosing which OS to install.

ADD REPLYlink written 7 weeks ago by johnnytam100100

You can build Singularity containers as Ubuntu, CentOs, BioLinux etc. The underlying OS kernel might have some impact on versions you can run (e.g. don't think you can run CentOs7 container on a CentOs5 machine).

ADD REPLYlink written 7 weeks ago by bruce.moran420
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 1110 users visited in the last hour