How to install pip on HPC server

Question

[Bioinfonext]

I am working on HPC server where already different version of python installed.

I need to use pip to install some bioinformatics software.

But When I am trying to load latest version of python and after that when type python version it always showing old python version so please suggest How I can install pip first and after that how should I use it to install other software:

[3052751@login1(alces) ~]$ python --version
Python 2.6.6
[3052751@login1(alces) ~]$ module load apps/python3
Unable to load apps/python3/3.4.3/gcc-4.4.7 -- alternative module (apps/python3/3.6.4/gcc-4.4.7) already loaded.
[3052771@login1(alces) ~]$ python --version
Python 2.6.6
[3052751@login1(alces) ~]$ pip --version
-bash: pip: command not found

[Devon Ryan]

Your module load command had issues and anyway you should try pip3.

[GenoMax]

What do you get with module avail python? This command should list all python modules available on your cluster.

[Bioinfonext]

[3052751@login1(alces) ~]$ module avail python

---  /opt/gridware/etc/el6/modules  ---

---  /opt/apps/etc/modules  ---

  python/gcc-4.4.7/2.7.9

---  /opt/service/etc/modules  ---

[3052751@login1(alces) ~]$

[GenoMax]

What do you get if you do following by running in your $HOME.

find . -type f -iname "*htseq*"

[Bioinfonext]

Hi genomax:

[3052751@login2(alces) ~]$ find . -type f -iname "*htseq*"

./.python-eggs/HTSeq-0.6.1p1-py2.7-linux-x86_64.egg-tmp/HTSeq/_HTSeq.so

./gridware/share/python/3.4.3/lib/python3.6/site-packages/HTSeq/_HTSeq.cpython-36m-x86_64-linux-gnu.so

./gridware/share/python/3.4.3/lib/python3.6/site-packages/HTSeq/__pycache__/_HTSeq_internal.cpython-36.pyc

./gridware/share/python/3.4.3/lib/python3.6/site-packages/HTSeq/_HTSeq_internal.py

./gridware/share/python/3.4.3/bin/htseq-qa

./gridware/share/python/3.4.3/bin/htseq-count

./.local/lib/python3.6/site-packages/HTSeq/_HTSeq.cpython-36m-x86_64-linux-gnu.so

./.local/lib/python3.6/site-packages/HTSeq/__pycache__/_HTSeq_internal.cpython-36.pyc

./.local/lib/python3.6/site-packages/HTSeq/_HTSeq_internal.py

./.local/bin/htseq-qa

./.local/bin/htseq-count

[GenoMax]

Well the programs you are looking for are right there. In two places.

./.local/bin/htseq-count

and

./gridware/share/python/3.4.3/bin/htseq-count

If your $HOME is not available on all cluster nodes then you should use ./gridware/share/python/3.4.3/bin/htseq-count.

[Bioinfonext]

Hi genomax:

Older version of htseq is already install on HPC server: that I can run by using below command but How should I run recent one that I have installed:

older one:

[3052751@login2(alces) ~]$ module add apps/htseq

[3052751@login2(alces) ~]$ python -m HTSeq.scripts.count

Usage: count.py [options] alignment_file gff_file

This script takes an alignment file in SAM/BAM format and a feature file in
GFF format and calculates for each feature the number of reads mapping to it.
See http://www-huber.embl.de/users/anders/HTSeq/doc/count.html for details.

Options:
  -h, --help            show this help message and exit

  -f SAMTYPE, --format=SAMTYPE

                        type of <alignment_file> data, either 'sam' or 'bam
'
                        (default: sam)

  -r ORDER, --order=ORDER

[Bioinfonext]

latest one:

[3052751@login2(alces) ~]$ python -m /gridware/share/python/3.4.3/bin/htseq-count

/opt/gridware/pkg/el6/apps/python/2.7.8/gcc-4.4.7/bin/python: Import by filename is not supported.

[GenoMax]

module purge apps/htseq

To remove the old module you have already loaded.

Then run (from the directory where you ran the find command)

./.local/bin/htseq-count -h

or

./gridware/share/python/3.4.3/bin/htseq-count -h

You should not need to add python3 at beginning.

[Bioinfonext]

I run find commad from the same location:

[3052751@login2(alces) ~]$ /gridware/share/python/3.4.3/bin/htseq-count -h

-bash: /gridware/share/python/3.4.3/bin/htseq-count: No such file or directory

[3052751@login2(alces) ~]$ ./.local/bin/htseq-count -h

/opt/gridware/pkg/el6/apps/python3/3.6.4/gcc-4.4.7/bin/python3.6: error while loading shared libraries: 

libpython3.6m.so.1.0: cannot open shared object file: No such file or directory

[3052751@login2(alces) ~]$

[GenoMax]

What modules are currently loaded?

module list

[Bioinfonext]

[

3052751@login2(alces) ~]$ module list

No Modulefiles Currently Loaded.

[Bioinfonext]

Do I need to load python3?

[GenoMax]

Yes you should try that.

[Bioinfonext]

[3052771@login2(alces) ~]$ module load apps/python3

[3052751@login2(alces) ~]$ ./gridware/share/python/3.4.3/bin/htseq-count -h

/opt/gridware/pkg/el6/apps/python3/3.6.4/gcc-4.4.7/bin/python3.6: error while lo                                          

ading shared libraries: libpython3.6m.so.1.0: cannot open shared object file: No                                           such file or directory
[3052751@login2(alces) ~]$

[GenoMax]

Something is set up strangely on your cluster. At this point you are going to need some local help to figure out what is going on.

I have run out of things to try :-(

[Bioinfonext]

Hi genomax,

thanks for your time and so much effort, surely I am going ask to admin here and let you know if something comes out.

Answer 1

[GenoMax]

It appears that you already have python v.3.6.4 module loaded. Normally python3 is accessed by doing python3 -V. pip should alreayd be installed there (try pip3 as @Devon suggested). If not use the directions here.

Ideally you should ask the HPC admins to install this so all users can benefit.

[Bioinfonext]

Hi genomax,

Now it is showing python3

[3052771@login1(alces) ~]$ python3 -V

Python 3.6.4

How I can check pip already install or not?

Yes it is showing pip3:

[3052771@login1(alces) ~]$ pip3 --version

pip 9.0.1 from /opt/gridware/pkg/el6/apps/python3/3.6.4/gcc-4.4.7/lib/python3.6/site-packages (python 3.6)

[Bioinfonext]

Now I am trying to install latest version of HTseq but I am not able to found in current directory:where it is install ?

[

3052771@login1(alces) ~]$ pip3 install HTSeq

Collecting HTSeq
  Downloading HTSeq-0.9.1-cp36-cp36m-manylinux1_x86_64.whl (1.1MB)
    100% |################################| 1.1MB 866kB/s
Collecting pysam>=0.9.0 (from HTSeq)
  Downloading pysam-0.13-cp36-cp36m-manylinux1_x86_64.whl (9.3MB)
    100% |################################| 9.3MB 19kB/s
Collecting numpy (from HTSeq)
  Downloading numpy-1.14.0-cp36-cp36m-manylinux1_x86_64.whl (17.2MB)
    100% |################################| 17.2MB 30kB/s
Installing collected packages: pysam, numpy, HTSeq
Successfully installed HTSeq-0.9.1 numpy-1.14.0 pysam-0.13

[3052771@login1(alces) ~]$ ls

clusterware-ssh-keygen.log  gridware  hisat2-2.1.0  ncbi  NGS_Tools

[GenoMax]

I think pip installs into $HOME/.local/bin. Check there.

[Bioinfonext]

I just try this:

[3052771@login1(alces) ~]$ pwd

/users/3052751

[3052771@login1(alces) ~]$ cd /home/local/bin

-bash: cd: /home/local/bin: No such file or directory

[Devon Ryan]

ls ~/.local/bin

[Bioinfonext]

[3052751@login1(alces) ~]$ ls ~/.local/bin

ls: cannot access /users/3052751/.local/bin: No such file or directory

[Bioinfonext]

[3052751@login1(alces) ~]$ ls ~/local/bin

ls: cannot access /users/3052751/local/bin: No such file or directory

[GenoMax]

Check the period before local .local.

[Devon Ryan]

I'm a bit surprised that that worked without --user, though perhaps your admins modified things to make it the default.

[Bioinfonext]

how can I try to install to it again with path location?

[Devon Ryan]

I think there's a --force option, though usually specifying either --user or --prefix should do the trick.

[Bioinfonext]

[3052751@login1(alces) ~]$ pip3 install HTSeq --user /users/3052771/

Directory '/users/3052751/' is not installable. File 'setup.py' not found.

[Devon Ryan]

You can't just randomly try combinations of things. --user takes no parameters, which is why you're getting that results. I'm not going to provide you with specific commands to run, I expect you to be able to use the information provided to construct reasonable commands.

[Bioinfonext]

Hi Devon,

Thanks for your all help: I try this also but not finding the HTseq:

[3052751@login1(alces) ~]$ pip3 install HTSeq --user

Collecting HTSeq
  Using cached HTSeq-0.9.1-cp36-cp36m-manylinux1_x86_64.whl
Collecting numpy (from HTSeq)
  Using cached numpy-1.14.0-cp36-cp36m-manylinux1_x86_64.whl
Collecting pysam>=0.9.0 (from HTSeq)
  Using cached pysam-0.13-cp36-cp36m-manylinux1_x86_64.whl
Installing collected packages: numpy, pysam, HTSeq
Successfully installed HTSeq-0.9.1 numpy-1.14.0 pysam-0.13
[3052751@login1(alces) ~]$ ls
clusterware-ssh-keygen.log  gridware  hisat2-2.1.0  ncbi  NGS_Tools

[GenoMax]

You should ask your admins where the default location for pip installs is set to? Sounds like it is not in $HOME/.local/bin. You could also run find . --name HTseq -print to see if that locates the program.

[Bioinfonext]

Thanks I will ask to admin about this.

Thaks a lot for your time and prompt response.