unexpected error in conda when installing new packages or environments
1
0
Entering edit mode
3 months ago
gubrins ▴ 260

Heys,

Since some weeks on, I have been having problems with conda when trying to install new packages or creating new environments. Also, when uploading previous environments, I need to add the full path, otherwise it does not recognize it.

This is the code:

conda create -y -n qc3c -c cerebis -c conda-forge -c bioconda qc3C


And this is the error message:

Traceback (most recent call last):

File "/home/panthera/miniconda3/lib/python3.9/site-packages/conda/core/subdir_data.py", line 531, in fetch_repodata_remote_request
resp.raise_for_status()
File "/home/panthera/miniconda3/lib/python3.9/site-packages/requests/models.py", line 960, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://repo.anaconda.com/pkgs/free/noarch/current_repodata.json

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/panthera/miniconda3/lib/python3.9/site-packages/requests/models.py", line 910, in json
File "/home/panthera/miniconda3/lib/python3.9/site-packages/simplejson/__init__.py", line 525, in loads
return _default_decoder.decode(s)
File "/home/panthera/miniconda3/lib/python3.9/site-packages/simplejson/decoder.py", line 370, in decode
obj, end = self.raw_decode(s)
File "/home/panthera/miniconda3/lib/python3.9/site-packages/simplejson/decoder.py", line 400, in raw_decode
return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/panthera/miniconda3/lib/python3.9/site-packages/conda/exceptions.py", line 1129, in __call__
return func(*args, **kwargs)
File "/home/panthera/miniconda3/lib/python3.9/site-packages/conda/cli/main.py", line 86, in main_subshell
exit_code = do_call(args, p)
File "/home/panthera/miniconda3/lib/python3.9/site-packages/conda/cli/conda_argparse.py", line 93, in do_call
return getattr(module, func_name)(args, parser)
File "/home/panthera/miniconda3/lib/python3.9/site-packages/conda/notices/core.py", line 72, in wrapper
return_value = func(*args, **kwargs)
File "/home/panthera/miniconda3/lib/python3.9/site-packages/conda/cli/main_create.py", line 43, in execute
install(args, parser, 'create')
File "/home/panthera/miniconda3/lib/python3.9/site-packages/conda/cli/install.py", line 261, in install
File "/home/panthera/miniconda3/lib/python3.9/site-packages/conda/core/solve.py", line 156, in solve_for_transaction
File "/home/panthera/miniconda3/lib/python3.9/site-packages/conda/core/solve.py", line 199, in solve_for_diff
final_precs = self.solve_final_state(update_modifier, deps_modifier, prune, ignore_pinned,
File "/home/panthera/miniconda3/lib/python3.9/site-packages/conda/core/solve.py", line 304, in solve_final_state
File "/home/panthera/miniconda3/lib/python3.9/site-packages/conda/common/io.py", line 86, in decorated
return f(*args, **kwds)
File "/home/panthera/miniconda3/lib/python3.9/site-packages/conda/core/solve.py", line 467, in _collect_all_metadata
index, r = self._prepare(prepared_specs)
File "/home/panthera/miniconda3/lib/python3.9/site-packages/conda/core/solve.py", line 1061, in _prepare
reduced_index = get_reduced_index(self.prefix, self.channels,
File "/home/panthera/miniconda3/lib/python3.9/site-packages/conda/core/index.py", line 301, in get_reduced_index
new_records = SubdirData.query_all(spec, channels=channels, subdirs=subdirs,
File "/home/panthera/miniconda3/lib/python3.9/site-packages/conda/core/subdir_data.py", line 137, in query_all
result = tuple(concat(executor.map(subdir_query, channel_urls)))
File "/home/panthera/miniconda3/lib/python3.9/concurrent/futures/_base.py", line 609, in result_iterator
yield fs.pop().result()
File "/home/panthera/miniconda3/lib/python3.9/concurrent/futures/_base.py", line 439, in result
return self.__get_result()
File "/home/panthera/miniconda3/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
raise self._exception
File "/home/panthera/miniconda3/lib/python3.9/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/panthera/miniconda3/lib/python3.9/site-packages/conda/core/subdir_data.py", line 129, in <lambda>
subdir_query = lambda url: tuple(SubdirData(Channel(url), repodata_fn=repodata_fn).query(
File "/home/panthera/miniconda3/lib/python3.9/site-packages/conda/core/subdir_data.py", line 142, in query
File "/home/panthera/miniconda3/lib/python3.9/site-packages/conda/core/subdir_data.py", line 207, in load
File "/home/panthera/miniconda3/lib/python3.9/site-packages/conda/core/subdir_data.py", line 277, in _load
raw_repodata_str = fetch_repodata_remote_request(
File "/home/panthera/miniconda3/lib/python3.9/site-packages/conda/core/subdir_data.py", line 587, in fetch_repodata_remote_request
raise UnavailableInvalidChannel(
File "/home/panthera/miniconda3/lib/python3.9/site-packages/conda/exceptions.py", line 464, in __init__
body = response.json()
File "/home/panthera/miniconda3/lib/python3.9/site-packages/requests/models.py", line 917, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: [Errno Expecting value] <html>
<body>
<ul>
<li>Code: NoSuchKey</li>
<li>Message: The specified key does not exist.</li>
<li>Key: pkgs/free/noarch/current_repodata.json</li>
<li>RequestId: YGVRRSETKSRK39TN</li>
<li>HostId: hx5GhLCHGp6XDUtRNaVWGCO5x7AaICOVDjc3+WsdvmgvxdXKgX1Mau68lDFg9EIE5SHxzjpaCPg=</li>
</ul>
<hr/>
</body>
</html>
: 0
$/home/panthera/miniconda3/bin/conda create -y -n qc3c -c cerebis -c conda-forge -c bioconda qc3C environment variables: CIO_TEST= CONDA_DEFAULT_ENV=base CONDA_EXE=/home/panthera/miniconda3/bin/conda CONDA_PREFIX=/home/panthera/miniconda3 CONDA_PROMPT_MODIFIER=(base) CONDA_PYTHON_EXE=/home/panthera/miniconda3/bin/python CONDA_ROOT=/home/panthera/miniconda3 CONDA_SHLVL=1 CURL_CA_BUNDLE= LD_LIBRARY_PATH=/home/panthera/software/beagle-lib/lib:/home/panthera/software/beagle- lib/lib:/home/panthera/software/beagle- lib/lib:/home/panthera/software/beagle- lib/lib:/home/panthera/software/beagle- lib/lib:/home/panthera/software/beagle- lib/lib:/home/panthera/software/beagle-lib/lib: PATH=/home/panthera/.local/bin:/home/panthera/bin:/home/panthera/miniconda3 /bin:/home/panthera/miniconda3/condabin:/usr/local/sbin:/usr/local/bin :/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/ home/panthera/bin:/home/panthera/software/MitoFinder:/mnt/DiscoD/ibero lacerta/analyses/hi-c/tutorial_manual/SALSA2/bin:/home/panthera/bin:/h ome/panthera/software/MitoFinder:/mnt/DiscoD/iberolacerta/analyses/hi- c/tutorial_manual/SALSA2/bin:/home/panthera/bin:/home/panthera/softwar e/MitoFinder:/mnt/DiscoD/iberolacerta/analyses/hi-c/tutorial_manual/SA LSA2/bin:/home/panthera/bin:/home/panthera/software/MitoFinder:/mnt/Di scoD/iberolacerta/analyses/hi-c/tutorial_manual/SALSA2/bin:/home/panth era/bin:/home/panthera/software/MitoFinder:/mnt/DiscoD/iberolacerta/an alyses/hi-c/tutorial_manual/SALSA2/bin:/home/panthera/bin:/home/panthe ra/software/MitoFinder:/mnt/DiscoD/iberolacerta/analyses/hi-c/tutorial _manual/SALSA2/bin:/home/panthera/bin:/home/panthera/software/MitoFind er:/mnt/DiscoD/iberolacerta/analyses/hi-c/tutorial_manual/SALSA2/bin REQUESTS_CA_BUNDLE= SSL_CERT_FILE= active environment : base active env location : /home/panthera/miniconda3 shell level : 1 user config file : /home/panthera/.condarc populated config files : /home/panthera/.condarc conda version : 22.9.0 conda-build version : not installed python version : 3.9.12.final.0 virtual packages : __linux=5.15.0=0 __glibc=2.31=0 __unix=0=0 __archspec=1=x86_64 base environment : /home/panthera/miniconda3 (writable) conda av data dir : /home/panthera/miniconda3/etc/conda conda av metadata url : None channel URLs : https://conda.anaconda.org/cerebis/linux-64 https://conda.anaconda.org/cerebis/noarch https://conda.anaconda.org/conda-forge/linux-64 https://conda.anaconda.org/conda-forge/noarch https://conda.anaconda.org/bioconda/linux-64 https://conda.anaconda.org/bioconda/noarch https://conda.anaconda.org/r/linux-64 https://conda.anaconda.org/r/noarch https://conda.anaconda.org/bioconda/label/cf201901/linux-64 https://conda.anaconda.org/bioconda/label/cf201901/noarch https://conda.anaconda.org/biobuilds/linux-64 https://conda.anaconda.org/biobuilds/noarch https://conda.anaconda.org/yuxiang/linux-64 https://conda.anaconda.org/yuxiang/noarch https://repo.anaconda.com/pkgs/main/linux-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/free/linux-64 https://repo.anaconda.com/pkgs/free/noarch https://repo.anaconda.com/pkgs/r/linux-64 https://repo.anaconda.com/pkgs/r/noarch package cache : /home/panthera/miniconda3/pkgs /home/panthera/.conda/pkgs envs directories : /home/panthera/miniconda3/envs /home/panthera/.conda/envs platform : linux-64 user-agent : conda/22.9.0 requests/2.27.1 CPython/3.9.12 Linux/5.15.0-43-generic ubuntu/20.04.3 glibc/2.31 UID:GID : 1000:1000 netrc file : None offline mode : False An unexpected error has occurred. Conda has prepared the above report. Upload successful.  Any idea of why is this happening? I did not manage to solve it. conda • 1.5k views ADD COMMENT 1 Entering edit mode That's weird. Is it maybe hitting that 404 error on current_repodata.json and then trying to parse JSON from it anyway? For me it fails on that URL, goes to another URL, and then seems OK. I notice I don't have simplejson in my base environment, though. I wonder if it's to blame for some reason, by way of the requests module. Maybe share the output of conda info and conda list -n base --explicit. Those might gives some hints. ADD REPLY 0 Entering edit mode Thanks for your answer!! We are desperates with this... Here it is: conda info  results: active environment : base active env location : /home/panthera/miniconda3 shell level : 1 user config file : /home/panthera/.condarc populated config files : /home/panthera/.condarc conda version : 22.9.0 conda-build version : not installed python version : 3.9.12.final.0 virtual packages : __linux=5.15.0=0 __glibc=2.31=0 __unix=0=0 __archspec=1=x86_64 base environment : /home/panthera/miniconda3 (writable) conda av data dir : /home/panthera/miniconda3/etc/conda conda av metadata url : None channel URLs : https://conda.anaconda.org/conda-forge/linux-64 https://conda.anaconda.org/conda-forge/noarch https://conda.anaconda.org/bioconda/linux-64 https://conda.anaconda.org/bioconda/noarch https://conda.anaconda.org/biobuilds/linux-64 https://conda.anaconda.org/biobuilds/noarch https://repo.anaconda.com/pkgs/main/linux-64 https://repo.anaconda.com/pkgs/main/noarch https://repo.anaconda.com/pkgs/free/linux-64 https://repo.anaconda.com/pkgs/free/noarch https://repo.anaconda.com/pkgs/r/linux-64 https://repo.anaconda.com/pkgs/r/noarch package cache : /home/panthera/miniconda3/pkgs /home/panthera/.conda/pkgs envs directories : /home/panthera/miniconda3/envs /home/panthera/.conda/envs platform : linux-64 user-agent : conda/22.9.0 requests/2.27.1 CPython/3.9.12 Linux/5.15.0-48-generic ubuntu/20.04.3 glibc/2.31 UID:GID : 1000:1000 netrc file : None offline mode : False  And the second one: conda list -n base --explicit  result: # This file may be used to create an environment using: #$ conda create --name <env> --file <this file>
# platform: linux-64
@EXPLICIT
https://repo.anaconda.com/pkgs/main/linux-64/_libgcc_mutex-0.1-main.conda
https://conda.anaconda.org/conda-forge/linux-64/ca-certificates-2022.9.24-ha878542_0.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/ld_impl_linux-64-2.35.1-h7274673_9.conda
https://repo.anaconda.com/pkgs/main/noarch/tzdata-2022a-hda174b7_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/libgomp-11.2.0-h1234567_1.conda
https://repo.anaconda.com/pkgs/main/linux-64/_openmp_mutex-4.5-1_gnu.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/libgcc-ng-11.2.0-h1234567_1.conda
https://repo.anaconda.com/pkgs/main/linux-64/libffi-3.3-he6710b0_2.conda
https://repo.anaconda.com/pkgs/main/linux-64/ncurses-6.3-h7f8727e_2.conda
https://repo.anaconda.com/pkgs/main/linux-64/openssl-1.1.1q-h7f8727e_0.conda
https://conda.anaconda.org/bioconda/linux-64/pretextgraph-0.0.6-h9f5acd7_1.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/xz-5.2.5-h7b6447c_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/yaml-0.2.5-h7b6447c_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/zlib-1.2.12-h7f8727e_1.conda
https://repo.anaconda.com/pkgs/main/linux-64/tk-8.6.11-h1ccaba5_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/sqlite-3.38.2-hc218d9a_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/python-3.9.12-h12debd9_0.conda
https://conda.anaconda.org/conda-forge/noarch/certifi-2022.9.24-pyhd8ed1ab_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/charset-normalizer-2.0.4-pyhd3eb1b0_0.conda
https://repo.anaconda.com/pkgs/main/noarch/colorama-0.4.4-pyhd3eb1b0_0.conda
https://repo.anaconda.com/pkgs/main/noarch/idna-3.3-pyhd3eb1b0_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/pycosat-0.6.3-py39h27cfd23_0.conda
https://repo.anaconda.com/pkgs/main/noarch/pycparser-2.21-pyhd3eb1b0_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/pysocks-1.7.1-py39h06a4308_0.conda
https://conda.anaconda.org/conda-forge/linux-64/python_abi-3.9-2_cp39.tar.bz2
https://repo.anaconda.com/pkgs/main/linux-64/ruamel_yaml-0.15.100-py39h27cfd23_0.conda
https://repo.anaconda.com/pkgs/main/noarch/six-1.16.0-pyhd3eb1b0_1.conda
https://conda.anaconda.org/conda-forge/noarch/toolz-0.12.0-pyhd8ed1ab_0.tar.bz2
https://repo.anaconda.com/pkgs/main/noarch/wheel-0.37.1-pyhd3eb1b0_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/cffi-1.15.0-py39hd667e15_1.conda
https://repo.anaconda.com/pkgs/main/linux-64/setuptools-61.2.0-py39h06a4308_0.conda
https://repo.anaconda.com/pkgs/main/noarch/tqdm-4.63.0-pyhd3eb1b0_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/brotlipy-0.7.0-py39h27cfd23_1003.conda
https://repo.anaconda.com/pkgs/main/linux-64/conda-package-handling-1.8.1-py39h7f8727e_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/cryptography-36.0.0-py39h9ce1e76_0.conda
https://repo.anaconda.com/pkgs/main/linux-64/pip-21.2.4-py39h06a4308_0.conda
https://repo.anaconda.com/pkgs/main/noarch/conda-content-trust-0.1.1-pyhd3eb1b0_0.conda
https://repo.anaconda.com/pkgs/main/noarch/pyopenssl-22.0.0-pyhd3eb1b0_0.conda
https://repo.anaconda.com/pkgs/main/noarch/urllib3-1.26.8-pyhd3eb1b0_0.conda
https://repo.anaconda.com/pkgs/main/noarch/requests-2.27.1-pyhd3eb1b0_0.conda
https://conda.anaconda.org/conda-forge/linux-64/conda-22.9.0-py39hf3d152e_0.tar.bz2

1
Entering edit mode

Odd that there's no simplejson listed there, even though it's definitely installed in the environment. Maybe it came from pip? What does pip list (within the base environment) show you?

My hunch is that something's wrong with your conda setup but it's not clear where. I noticed there are a lot of extra channels and shell environment variables and things defined. Depending on how much pressure you're under to just make it all work, you could try with a totally separate miniconda3 install and shell environment, so you at least have a working case to compare with. If you set up an alternate HOME and PATH in a subshell you can actually get an isolated miniconda setup under the same user account. (I've used that before for troubleshooting tricky conda behavior. You'd just have to be comfortable with some shell scripting to set it up.)

0
Entering edit mode

Thank you very much for your help Jesse, I really appreciate it.

This is the output of pip list

Package                Version
---------------------- ---------
bio                    1.4.0
biopython              1.79
biothings-client       0.2.6
brotlipy               0.7.0
certifi                2022.9.24
cffi                   1.15.0
charset-normalizer     2.0.4
click                  8.1.3
colorama               0.4.4
coloredlogs            15.0.1
colormath              3.0.0
commonmark             0.9.1
conda                  22.9.0
conda-content-trust    0+unknown
conda-package-handling 1.8.1
contourpy              1.0.5
cryptography           36.0.0
cycler                 0.11.0
fonttools              4.37.3
future                 0.18.2
humanfriendly          10.0
idna                   3.3
Jinja2                 3.1.2
kiwisolver             1.4.4
lzstring               1.0.4
Markdown               3.4.1
MarkupSafe             2.1.1
matplotlib             3.6.0
multiqc                1.13
mygene                 3.2.2
networkx               2.8.6
numpy                  1.23.3
packaging              21.3
Pillow                 9.2.0
pip                    21.2.4
pycosat                0.6.3
pycparser              2.21
pyfastaq               3.17.0
Pygments               2.13.0
pyOpenSSL              22.0.0
pyparsing              3.0.9
PySocks                1.7.1
python-dateutil        2.8.2
PyYAML                 6.0
requests               2.27.1
rich                   12.5.1
rich-click             1.5.2
ruamel-yaml-conda      0.15.100
seqtk                  0.5.0
setuptools             61.2.0
simplejson             3.17.6
six                    1.16.0
spectra                0.0.11
toolz                  0.12.0
tqdm                   4.63.0
urllib3                1.26.8
wheel                  0.37.1
zipp                   3.8.1


Do you know any tutorial about how doing what you mentioned? I know how to create a conda environment, but not a shell environment. I was looking for it and I only found how to create environmental variables in shell...

1
Entering edit mode

By shell environment I just mean the variables defined in your current command line (all the beagle stuff in LD_LIBRARY_PATH, many things in PATH, etc.) Nothing jumps out at me specifically. It's just stuff you could simplify for troubleshooting. A couple years back I made a little bundle of scripts for troublehsooting our own conda-based software, so I just generalized that same method a bit now for arbitrary conda-related commands. Give this a try:

git clone https://gitlab.com/ressy/conda-debug.git
cd conda-debug
env -i bash --noprofile --norc
./conda_wrapper.sh conda_setup.sh
./conda_wrapper.sh conda create -y -n qc3c -c cerebis -c conda-forge -c bioconda qc3C
./conda_wrapper.sh bash
conda activate qc3c


That env command starts up a very limited shell without any of your usual settings, and the next ones set up conda inside the current directory only, with no outside references. If that works, your problem is somewhere in your existing conda files or shell configuration (bash, I'm guessing?) and it's a matter of figuring our what's different (or just reinstalling if you want to just make it work).

If the above doesn't work, I'm stumped!

0
Entering edit mode

Again, thank you very much Jesse for all the effort. I've just run the commands you were proposing and it worked!! I imagine that when you do this env -i bash --noprofile --norc the --norc part means to avoid my condarc file? This could be the issue, as I had to modify it for installing a package. See it here: https://github.com/VGP/vgp-assembly/issues/81

Could this be the reason of my issue? Having modified my condarc file? Right now my .condarc file looks like this:

auto_activate_base: true

channels:

- conda-forge

- bioconda

- biobuilds

- defaults

channel_priority: flexible
restore_free_channel: true
report_errors: true

1
Entering edit mode

OK, that gives you a bit of info! The --norc I mentioned is for bash rather than conda (similar idea, different software, "deeper" than conda). But yeah part of running those commands was leaving behind your ~/.condarc which could be related, and the timing sounds like it makes sense. I think that's more likely part of the problem than anything handlead by that env command, actually, but I thought starting completely fresh to get a working example made sense first. So... you could re-do those commands and just skip the env part and it will be slightly more similar to what you already have, which could narrow it down slightly more. Easy enough to try it since you can remove that whole conda-debug directory at any time and start that from scratch.

All this tells you it's definitely something about your ~/miniconda3 and/or ~/.condarc and/or other ~/. files but it's going to be hard to tell precisely what. My hunch is that the state of the conda package's files within the base environment got messed up somehow, possibly related to the .condarc changes. If so, though, it's hard to revert back from that, since you can't just change the .condarc back to get the previous state of the environment's files.

I think your options are:

1. Remove (or just set aside for safekeeping until you're sure!) however much of your existing conda files you need to just get things working again. This is probably the faster and easier way if you can spare ditching your old conda setup.
2. Keep digging through files and compare-and-contrasting with an isolated test conda install to figure out exactly what's wrong in these files. The answer is hiding in there somewhere. This could take a long while, though, and I think I'm out of obvious suggestions from this end.
0
Entering edit mode

Again, thanks a lot for you answer! I would like to solve the problem as soon as possible, so I am thinking in removing it. But what do you mean? To remove all conda and install it again, or there is a way to only remove the .condarc file and create a new one? If I remove conda, I understand that all the environments will be lost?

2
Entering edit mode
3 months ago
Jesse ▴ 450

I'd suggest moving (rather than deleting) existing conda files, including .condarc and others, into a directory while you try things out with a new install. Then you have the option to reverse the changes if you need to, and your environments aren't irreversibly lost.

Something like:

conda init --reverse
mkdir ~/conda-backup
mv ~/miniconda3 ~/conda-backup
mv ~/.condarc ~/conda-backup
mv ~/.conda ~/conda-backup


Then if you log out and back in there should be no conda loaded and you can proceed with an installation as if it's the first setup. But, if need be you could reverse those steps above and get back to what you had before. (Apparently there's also a package called anaconda-clean I just noticed that sounds like it aims for the same idea, but I don't have experience with that personally.)

0
Entering edit mode

excellent, thanks a lot for your help! I'll try and I will let you know!

0
Entering edit mode

Dear Jesse I am extremelly thankful for your help, I created a new conda and it worked!!!! You cannot imagine how happy we are!! It has been a nightmare, thanks again!!!

0
Entering edit mode

Sure thing! Glad to hear it.