Cannot install NumPy using pip and import it

Question

[Chvatil]

I'm actually using Matplotlib and Numpy for a Python script on a ssh.

I had to creat a virtual python3 environment on my ssh account.

then, I made: pip3 install numpy

and everything worked well:

Collecting numpy
  Using cached https://files.pythonhosted.org/packages/7b/61/11b05cc37ccdaabad89f04dbdc2a02905cf6de6f9b05816dba843beed328/numpy-1.14.3-cp35-cp35m-manylinux1_x86_64.whl
Installing collected packages: numpy
Successfully installed numpy-1.14.3

pip3 show numpy

Name: numpy
Version: 1.14.3
Summary: NumPy: array processing for numbers, strings, records, and objects.
Home-page: http://www.numpy.org
Author: Travis E. Oliphant et al.
Author-email: None
License: BSD
Location: /panhome/me/.local/lib/python3.5/site-packages

Then, in my script I put the PYTHONPATH such:

export PYTHONPATH=$PYTHONPATH:/panhome/me/.local/lib/python3.5/site-packages

but I actually get the following error:

Traceback (most recent call last):
  File "/pandata/me/LEPIWASP/blast_database/public_scripts-master/Diamond_BLAST_add_taxonomic_info/Diamond_blast_to_taxid.py", line 25, in <module>
    import matplotlib
  File "/panhome/me/.local/lib/python3.5/site-packages/matplotlib/__init__.py", line 127, in <module>
    from . import cbook
  File "/panhome/me/.local/lib/python3.5/site-packages/matplotlib/cbook/__init__.py", line 35, in <module>
    import numpy as np
  File "/panhome/me/.local/lib/python3.5/site-packages/numpy/__init__.py", line 142, in <module>
    from . import add_newdocs
  File "/panhome/me/.local/lib/python3.5/site-packages/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/panhome/me/.local/lib/python3.5/site-packages/numpy/lib/__init__.py", line 8, in <module>
    from .type_check import *
  File "/panhome/me/.local/lib/python3.5/site-packages/numpy/lib/type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "/panhome/me/.local/lib/python3.5/site-packages/numpy/core/__init__.py", line 26, in <module>
    raise ImportError(msg)
ImportError: 
Importing the multiarray numpy extension module failed.  Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try `git clean -xdf` (removes all
files not under version control).  Otherwise reinstall numpy.

Original error was: cannot import name multiarray

I already tried to unistall numpy and reinstate it several times.

Thanks for your help.

Import command line in the script file: 
# imports
from __future__ import print_function
import time
import os
import sys
from optparse import OptionParser  # TODO: update to argparser
import datetime
import logging
import logging.handlers
import matplotlib
# this code added to prevent this error:
# self.tk = _tkinter.create(screenName, baseName,
# className, interactive, wantobjects, useTk, sync, use)
# _tkinter.TclError: no display name and
# no $DISPLAY environment variable
# Force matplotlib to not use any Xwindows backend.
matplotlib.use('Agg')
import matplotlib.pyplot as plt
import pylab

Here is my script to run the process:

#PBS -S /bin/bash
#PBS -l nodes=1:ppn=8
#PBS -e /pandata/me/LEPIWASP/blast_database/tax_name_candidates_sp1.error
#PBS -o /pandata/me/LEPIWASP/blast_database/tax_name_candidates_sp1.out
#PBS -q q1hour
#PBS -N tax_name_candidates_sp1


export PYTHONPATH=$PYTHONPATH:/panhome/me/.local/lib/python3.5/site-packages:/usr/lib/python3/dist-packages/
diamond_tab_output=/pandata/me/LEPIWASP/blast_database/matches.m8
Diamond_blast_to_taxid=/pandata/me/LEPIWASP/blast_database/public_scripts-master/Diamond_BLAST_add_taxonomic_info/Diamond_blast_to_taxid.py

taxid=/pandata/me/LEPIWASP/blast_database/gi_taxid_prot.dmp

categories=/pandata/me/LEPIWASP/blast_database/categories.dmp

names=/pandata/me/LEPIWASP/blast_database/names.dmp

description=/pandata/me/LEPIWASP/blast_database/acc_to_des.tab

$Diamond_blast_to_taxid -i $diamond_tab_output -t $taxid -c $categories -n $names -d $description -o outfile_sp1.tab

[Joe]

Try python -m pip install numpy.

If you’re intending to run of a different python binary, just substitute that in to the command above instead.

[Chvatil]

I did it :

Collecting numpy
  Using cached https://files.pythonhosted.org/packages/7b/61/11b05cc37ccdaabad89f04dbdc2a02905cf6de6f9b05816dba843beed328/numpy-1.14.3-cp35-cp35m-manylinux1_x86_64.whl
Installing collected packages: numpy
Successfully installed numpy-1.14.3

and I stil get the same first error message :/

[Devon Ryan]

Run rm -rf ~/.local/lib/python3.5/site-packages/numpy, then pip install it again. On occasion you get left-over egg files and they cause weird behavior like this.

[Chvatil]

I'm still getting the same message, I tried several times to uninstall numpy without sucess...

[shoujun.gu]

Did you start the virtual environment first?

[Chvatil]

Yep, @shoujun.gu I forgot to do it but I actually got another issue:

    Traceback (most recent call last):
      File "/pandatame/LEPIWASP/blast_database/public_scripts-master/Diamond_BLAST_add_taxonomic_info/Diamond_blast_to_taxid
    .py", line 23, in <module>
        import logging
      File "/usr/lib/python3.5/logging/__init__.py", line 26, in <module>
        import sys, os, time, io, traceback, warnings, weakref, collections
      File "/usr/lib/python3.5/traceback.py", line 5, in <module>
        import linecache
      File "/panhome/me/benjamin-python/lib/python3.5/linecache.py", line 8, in <module>
        import functools
      File "/panhome/me/benjamin-python/lib/python3.5/functools.py", line 23, in <module>
        from weakref import WeakKeyDictionary
      File "/panhome/me/benjamin-python/lib/python3.5/weakref.py", line 12, in <module>
        from _weakref import (
    ImportError: cannot import name '_remove_dead_weakref'

I already tried to recreat a new environment without success.

[Devon Ryan]

Your python environment seems to be broken. You'll need to reinstall python (just download conda instead).

[Chvatil]

How can I do it, I'm on a ssh environment. Can I do it the same way I did with :

virtualenv -p /usr/bin/python3 mypython
source ~/mypythonbin/activate

???

[GenoMax]

Hello Toto26 ,

Don't forget to follow up on your threads.

If an answer was helpful, you should upvote it; if the answer resolved your question, you should mark it as accepted. You can accept more than one if they work.

Please do the same for this and your previous questions as well.

Answer 1

[Eric Lim]

$ wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
$ sh Miniconda3-latest-Linux-x86_64.sh
# follow on-screen steps to finish installation
$ conda install numpy

That should get rid of all the notorious errors from pip.

[Chvatil]

and then, what I write on my script to tell where is the good modul pythonpath? Should I do:

source ~/miniconda3/bin/activate?

[Eric Lim]

Judging from the comments and responses above, I don't think you need to. If you have multiple python binaries installed, just make sure PATH is pointing to conda's bin directory first.

If you need to run things in private environments for best practices, follow https://conda.io/docs/user-guide/tasks/manage-environments.html.

Play around with it. If you still encounter errors, paste the errors here for discussion.

[Chvatil]

OK thank you all for your help :)