conda create env using .yml file leads to dependency conflicts
2
1
Entering edit mode
14 months ago
Kevin ▴ 10

Hi all,

I'm trying to create a conda environment through a .yml file that has all the required dependencies for a certain project but I run into environment conflicts. I figured out the source of this conflict, which stems from two specific dependencies, but for some reason, creating an environment with these dependencies alone doesn't lead to an error, but does when I run it with all the other dependencies.

Here's what I mean:

This is my .yml file for the environment I want to create.

name: master
channels:
  - r
  - bioconda
  - conda-forge
  - defaults
dependencies:
  - bedtools=2.30.0
  - htseq=2.0.2
  - samtools=1.16.1
  - r-base=4.2.2
  - r-dplyr=1.0.10
  - r-ggplot2=3.4.0
  - r-ggridges=0.5.4
  - r-biocmanager=1.30.19
  - bioconductor-rsamtools=2.14.0
  - bioconductor-rsubread=2.8.1
  - bioconductor-deseq2=1.38.0
  - bioconductor-biocgenerics=0.44.0
  - bioconductor-genomicalignments=1.34.0
  - bioconductor-genomicranges=1.50.0

After running the code conda env create -f .yml it leads to an error basically stating

Found conflicts! Looking for incompatible packages.

The packages that I've determined are the issue are samtools and rsubread. When I run a subset of the .yml file without them, it works. When I run a .yml file with only samtools and rsubread, that also works. So I don't know why its creating a conflict. Both of the .yml files below work.

name: testingenv1
channels:
  - r
  - bioconda
  - conda-forge
  - defaults
dependencies:
  - bedtools=2.30.0
  - htseq=2.0.2
  - r-base=4.2.2
  - r-dplyr=1.0.10
  - r-ggplot2=3.4.0
  - r-ggridges=0.5.4
  - r-biocmanager=1.30.19
  - bioconductor-rsamtools=2.14.0
  - bioconductor-deseq2=1.38.0
  - bioconductor-biocgenerics=0.44.0
  - bioconductor-genomicalignments=1.34.0
  - bioconductor-genomicranges=1.50.0

name: testingenv2
channels:
  - r
  - bioconda
  - conda-forge
  - defaults
dependencies:
  - samtools=1.16.1
  - bioconductor-rsubread=2.8.1

I've tried searching the web for my own solution to no avail. I've even tried package clients like mamba instead of conda since I've heard that conda's error reporting is pretty bad, but mamba leads to other issues.

Any help would be appreciated!
yml conda • 4.9k views
ADD COMMENT •
0
Entering edit mode

Do you really need these specific versions of all packages? Can you change some = to >= or even drop reliance on specific version numbers entirely?

ADD REPLY •
0
Entering edit mode

Good idea. I'll see which ones I can drop reliance to. Thanks!

ADD REPLY •
0
Entering edit mode

For future use, you can either run conda env export --from-history > myenv.yml or conda env export --no-builds > myenv.yml to export your environment configuration without the build information. This may help you avoid conflicts in the future.

ADD REPLY •
2
Entering edit mode
14 months ago
Ram 43k

I checked the dependencies of bioconductor-rsubread=2.8.1 using mamba search bioconductor-rsubread=2.8.1 --info and this is what I see:

bioconductor-rsubread 2.8.1 r41haba8685_0
-----------------------------------------
file name   : bioconductor-rsubread-2.8.1-r41haba8685_0.tar.bz2
name        : bioconductor-rsubread
version     : 2.8.1
build       : r41haba8685_0
build number: 0
size        : 11.2 MB
license     : GPL (>=3)
subdir      : osx-64
url         : https://conda.anaconda.org/bioconda/osx-64/bioconductor-rsubread-2.8.1-r41haba8685_0.tar.bz2
md5         : 3fb683d51e576d3ecc89309052412dd4
timestamp   : 2022-02-24 15:51:46 UTC
dependencies:
  - libblas >=3.8.0,<4.0a0
  - liblapack >=3.8.0,<4.0a0
  - r-base >=4.1,<4.2.0a0
  - r-matrix

See how it needs r-base >=4.1 but less than 4.2.0a0? Your r-base=4.2.2 clashes with that. mamba search samtools=1.16.1 --info reveals:

samtools 1.16.1 h7e39424_1
--------------------------
file name   : samtools-1.16.1-h7e39424_1.tar.bz2
name        : samtools
version     : 1.16.1
build       : h7e39424_1
build number: 1
size        : 421 KB
license     : MIT
subdir      : osx-64
url         : https://conda.anaconda.org/bioconda/osx-64/samtools-1.16.1-h7e39424_1.tar.bz2
md5         : a65669266b75d5a7bdc5b62caceba094
timestamp   : 2022-10-10 20:04:31 UTC
dependencies:
  - htslib >=1.16,<1.17.0a0
  - libzlib >=1.2.12,<1.3.0a0
  - ncurses >=6.3,<7.0a0
  - zlib >=1.2.12,<1.3.0a0

While it does not directly clash with any other requirement, it is possible that its dependencies could have other dependencies that clash with requirements or dependencies of other requirements.

Remove the version numbers from samtools and rsubread, and you should be all set.
ADD COMMENT •
0
Entering edit mode

Ah got it. I didn't know I could directly check sub dependencies. Thanks so much for the help!

ADD REPLY •
1
Entering edit mode
14 months ago
cfos4698 ★ 1.1k

Sometimes some versions of programs/libraries just won't be compatible. How did you come to those specific versions of the programs you want?

I removed the version pinning for troubleshooting, changes the channel priorities, and used mamba. The environment was created successfully. As per the comments below, here is an example yaml file with compatible pinned versions (for _all_ dependencies of dependencies) that will resolve:

name: master
channels:
  - conda-forge
  - bioconda
  - r
  - defaults
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=2_gnu
  - _r-mutex=1.0.1=anacondar_1
  - argcomplete=2.0.0=pyhd8ed1ab_0
  - bedtools=2.30.0=h468198e_3
  - binutils_impl_linux-64=2.40=hf600244_0
  - bioconductor-annotate=1.76.0=r42hdfd78af_0
  - bioconductor-annotationdbi=1.60.0=r42hdfd78af_0
  - bioconductor-biobase=2.58.0=r42hc0cfd56_0
  - bioconductor-biocgenerics=0.44.0=r42hdfd78af_0
  - bioconductor-biocparallel=1.32.5=r42hc247a5b_0
  - bioconductor-biostrings=2.66.0=r42hc0cfd56_0
  - bioconductor-data-packages=20230202=hdfd78af_0
  - bioconductor-delayedarray=0.24.0=r42hc0cfd56_0
  - bioconductor-deseq2=1.38.0=r42hc247a5b_0
  - bioconductor-genefilter=1.80.0=r42h38f54d8_0
  - bioconductor-geneplotter=1.76.0=r42hdfd78af_0
  - bioconductor-genomeinfodb=1.34.8=r42hdfd78af_0
  - bioconductor-genomeinfodbdata=1.2.9=r42hdfd78af_0
  - bioconductor-genomicalignments=1.34.0=r42hc0cfd56_0
  - bioconductor-genomicranges=1.50.0=r42hc0cfd56_0
  - bioconductor-iranges=2.32.0=r42hc0cfd56_0
  - bioconductor-keggrest=1.38.0=r42hdfd78af_0
  - bioconductor-matrixgenerics=1.10.0=r42hdfd78af_0
  - bioconductor-rhtslib=2.0.0=r42hc0cfd56_0
  - bioconductor-rsamtools=2.14.0=r42hc247a5b_0
  - bioconductor-rsubread=2.12.0=r42hc0cfd56_0
  - bioconductor-s4vectors=0.36.0=r42hc0cfd56_0
  - bioconductor-summarizedexperiment=1.28.0=r42hdfd78af_0
  - bioconductor-xvector=0.38.0=r42hc0cfd56_0
  - bioconductor-zlibbioc=1.44.0=r42hc0cfd56_0
  - brotli=1.0.9=h166bdaf_8
  - brotli-bin=1.0.9=h166bdaf_8
  - bwidget=1.9.14=ha770c72_1
  - bzip2=1.0.8=h7f98852_4
  - c-ares=1.18.1=h7f98852_0
  - ca-certificates=2022.12.7=ha878542_0
  - cairo=1.16.0=ha61ee94_1014
  - certifi=2022.12.7=pyhd8ed1ab_0
  - curl=7.87.0=h6312ad2_0
  - cycler=0.11.0=pyhd8ed1ab_0
  - expat=2.5.0=h27087fc_0
  - font-ttf-dejavu-sans-mono=2.37=hab24e00_0
  - font-ttf-inconsolata=3.000=h77eed37_0
  - font-ttf-source-code-pro=2.038=h77eed37_0
  - font-ttf-ubuntu=0.83=hab24e00_0
  - fontconfig=2.14.2=h14ed4e7_0
  - fonts-conda-ecosystem=1=0
  - fonts-conda-forge=1=0
  - freetype=2.12.1=hca18f0e_1
  - fribidi=1.0.10=h36c2ea0_0
  - gcc_impl_linux-64=12.2.0=hcc96c02_19
  - gettext=0.21.1=h27087fc_0
  - gfortran_impl_linux-64=12.2.0=h55be85b_19
  - graphite2=1.3.13=h58526e2_1001
  - gsl=2.7=he838d99_0
  - gxx_impl_linux-64=12.2.0=hcc96c02_19
  - harfbuzz=6.0.0=h8e241bc_0
  - htslib=1.16=h6bc39ce_0
  - icu=70.1=h27087fc_0
  - importlib-metadata=4.13.0=pyha770c72_0
  - importlib_metadata=4.13.0=hd8ed1ab_0
  - jpeg=9e=h0b41bf4_3
  - jq=1.6=h36c2ea0_1000
  - kernel-headers_linux-64=2.6.32=he073ed8_15
  - keyutils=1.6.1=h166bdaf_0
  - krb5=1.20.1=hf9c8cef_0
  - lcms2=2.14=h6ed2654_0
  - ld_impl_linux-64=2.40=h41732ed_0
  - lerc=4.0.0=h27087fc_0
  - libblas=3.9.0=16_linux64_openblas
  - libbrotlicommon=1.0.9=h166bdaf_8
  - libbrotlidec=1.0.9=h166bdaf_8
  - libbrotlienc=1.0.9=h166bdaf_8
  - libcblas=3.9.0=16_linux64_openblas
  - libcurl=7.87.0=h6312ad2_0
  - libdeflate=1.13=h166bdaf_0
  - libedit=3.1.20191231=he28a2e2_2
  - libev=4.33=h516909a_1
  - libffi=3.4.2=h7f98852_5
  - libgcc-devel_linux-64=12.2.0=h3b97bd3_19
  - libgcc-ng=12.2.0=h65d4601_19
  - libgfortran-ng=12.2.0=h69a702a_19
  - libgfortran5=12.2.0=h337968e_19
  - libglib=2.74.1=h606061b_1
  - libgomp=12.2.0=h65d4601_19
  - libiconv=1.17=h166bdaf_0
  - liblapack=3.9.0=16_linux64_openblas
  - libnghttp2=1.51.0=hdcd2b5c_0
  - libnsl=2.0.0=h7f98852_0
  - libopenblas=0.3.21=pthreads_h78a6416_3
  - libpng=1.6.39=h753d276_0
  - libsanitizer=12.2.0=h46fd767_19
  - libsqlite=3.40.0=h753d276_0
  - libssh2=1.10.0=haa6b8db_3
  - libstdcxx-devel_linux-64=12.2.0=h3b97bd3_19
  - libstdcxx-ng=12.2.0=h46fd767_19
  - libtiff=4.4.0=h0e0dad5_3
  - libuuid=2.32.1=h7f98852_1000
  - libwebp-base=1.2.4=h166bdaf_0
  - libxcb=1.13=h7f98852_1004
  - libxml2=2.10.3=h7463322_0
  - libzlib=1.2.13=h166bdaf_4
  - make=4.3=hd18ef5c_1
  - matplotlib-base=3.7.0=py310he60537e_0
  - munkres=1.1.4=pyh9f0ad1d_0
  - ncurses=6.3=h27087fc_1
  - oniguruma=6.9.8=h166bdaf_0
  - openjpeg=2.5.0=h7d73246_1
  - openssl=1.1.1t=h0b41bf4_0
  - packaging=23.0=pyhd8ed1ab_0
  - pango=1.50.13=hd33c08f_0
  - pcre2=10.40=hc3806b6_0
  - pip=23.0.1=pyhd8ed1ab_0
  - pixman=0.40.0=h36c2ea0_0
  - pthread-stubs=0.4=h36c2ea0_1001
  - pyparsing=3.0.9=pyhd8ed1ab_0
  - python=3.10.8=h257c98d_0_cpython
  - python-dateutil=2.8.2=pyhd8ed1ab_0
  - python_abi=3.10=3_cp310
  - r-askpass=1.1=r42h06615bd_3
  - r-base=4.2.2=h6b4767f_2
  - r-bh=1.81.0_1=r42hc72bb7e_0
  - r-biocmanager=1.30.19=r42hc72bb7e_0
  - r-bit=4.0.5=r42h06615bd_0
  - r-bit64=4.0.5=r42h06615bd_1
  - r-bitops=1.0_7=r42h06615bd_1
  - r-blob=1.2.3=r42hc72bb7e_1
  - r-cachem=1.0.6=r42h06615bd_1
  - r-cli=3.6.0=r42h38f115c_0
  - r-codetools=0.2_19=r42hc72bb7e_0
  - r-colorspace=2.1_0=r42h133d619_0
  - r-cpp11=0.4.3=r42hc72bb7e_0
  - r-crayon=1.5.2=r42hc72bb7e_1
  - r-curl=4.3.3=r42h06615bd_1
  - r-dbi=1.1.3=r42hc72bb7e_1
  - r-dplyr=1.1.0=r42h38f115c_0
  - r-ellipsis=0.3.2=r42h06615bd_1
  - r-fansi=1.0.4=r42h133d619_0
  - r-farver=2.1.1=r42h7525677_1
  - r-fastmap=1.1.0=r42h7525677_1
  - r-formatr=1.14=r42hc72bb7e_0
  - r-futile.logger=1.4.3=r42hc72bb7e_1004
  - r-futile.options=1.0.1=r42hc72bb7e_1003
  - r-generics=0.1.3=r42hc72bb7e_1
  - r-ggplot2=3.4.1=r42hc72bb7e_0
  - r-ggridges=0.5.4=r42hc72bb7e_1
  - r-glue=1.6.2=r42h06615bd_1
  - r-gtable=0.3.1=r42hc72bb7e_1
  - r-httr=1.4.4=r42hc72bb7e_1
  - r-isoband=0.2.7=r42h38f115c_1
  - r-jsonlite=1.8.4=r42h133d619_0
  - r-labeling=0.4.2=r42hc72bb7e_2
  - r-lambda.r=1.2.4=r42hc72bb7e_2
  - r-lattice=0.20_45=r42h06615bd_1
  - r-lifecycle=1.0.3=r42hc72bb7e_1
  - r-locfit=1.5_9.7=r42h133d619_0
  - r-magrittr=2.0.3=r42h06615bd_1
  - r-mass=7.3_58.2=r42h133d619_0
  - r-matrix=1.5_3=r42h5f7b363_0
  - r-matrixstats=0.63.0=r42h06615bd_0
  - r-memoise=2.0.1=r42hc72bb7e_1
  - r-mgcv=1.8_41=r42h5f7b363_0
  - r-mime=0.12=r42h06615bd_1
  - r-munsell=0.5.0=r42hc72bb7e_1005
  - r-nlme=3.1_162=r42hac0b197_0
  - r-openssl=2.0.5=r42hb1dc35e_0
  - r-pillar=1.8.1=r42hc72bb7e_1
  - r-pkgconfig=2.0.3=r42hc72bb7e_2
  - r-plogr=0.2.0=r42hc72bb7e_1004
  - r-plyr=1.8.8=r42h7525677_0
  - r-png=0.1_8=r42h10cf519_0
  - r-r6=2.5.1=r42hc72bb7e_1
  - r-rcolorbrewer=1.1_3=r42h785f33e_1
  - r-rcpp=1.0.10=r42h38f115c_0
  - r-rcpparmadillo=0.11.4.4.0=r42h358215d_0
  - r-rcurl=1.98_1.10=r42h133d619_0
  - r-rlang=1.0.6=r42h7525677_1
  - r-rsqlite=2.2.20=r42h38f115c_0
  - r-scales=1.2.1=r42hc72bb7e_1
  - r-snow=0.4_4=r42hc72bb7e_1
  - r-survival=3.5_3=r42h133d619_0
  - r-sys=3.4.1=r42h06615bd_0
  - r-tibble=3.1.8=r42h06615bd_1
  - r-tidyselect=1.2.0=r42hc72bb7e_0
  - r-utf8=1.2.3=r42h133d619_0
  - r-vctrs=0.5.2=r42h38f115c_0
  - r-viridislite=0.4.1=r42hc72bb7e_1
  - r-withr=2.5.0=r42hc72bb7e_1
  - r-xml=3.99_0.13=r42hed7f998_0
  - r-xtable=1.8_4=r42hc72bb7e_4
  - readline=8.1.2=h0f457ee_0
  - samtools=1.16.1=h6899075_1
  - sed=4.8=he412f7d_0
  - setuptools=67.4.0=pyhd8ed1ab_0
  - six=1.16.0=pyh6c4a22f_0
  - sysroot_linux-64=2.12=he073ed8_15
  - tk=8.6.12=h27826a3_0
  - tktable=2.10=hb7b940f_3
  - toml=0.10.2=pyhd8ed1ab_0
  - tzdata=2022g=h191b570_0
  - wheel=0.38.4=pyhd8ed1ab_0
  - xmltodict=0.13.0=pyhd8ed1ab_0
  - xorg-kbproto=1.0.7=h7f98852_1002
  - xorg-libice=1.0.10=h7f98852_0
  - xorg-libsm=1.2.3=hd9c2040_1000
  - xorg-libx11=1.7.2=h7f98852_0
  - xorg-libxau=1.0.9=h7f98852_0
  - xorg-libxdmcp=1.1.3=h7f98852_0
  - xorg-libxext=1.3.4=h0b41bf4_2
  - xorg-libxrender=0.9.10=h7f98852_1003
  - xorg-libxt=1.2.1=h7f98852_2
  - xorg-renderproto=0.11.1=h7f98852_1002
  - xorg-xextproto=7.3.0=h0b41bf4_1003
  - xorg-xproto=7.0.31=h7f98852_1007
  - xz=5.2.6=h166bdaf_0
  - yaml=0.2.5=h7f98852_2
  - yq=3.1.1=pyhd8ed1ab_0
  - zipp=3.14.0=pyhd8ed1ab_0
  - zlib=1.2.13=h166bdaf_4
  - zstd=1.5.2=h3eb15da_6
  - pip:
    - contourpy==1.0.7
    - fonttools==4.38.0
    - htseq==2.0.2
    - kiwisolver==1.4.4
    - matplotlib==3.7.0
    - numpy==1.24.2
    - pillow==9.2.0
    - pysam==0.20.0
    - pyyaml==6.0
    - unicodedata2==15.0.0
ADD COMMENT •
0
Entering edit mode

Sometimes some versions of programs/libraries just won't be compatible.

OP figured out incompatibilities more specifically than this statement says - it's not like with people where "sometimes some people aren't just compatible", it's software so the dependencies are well laid out. Removing versions from all requirements is an easy but quite dirty fix. Also, it does not answer why this particular set of requirements clashed or why removing those two made it work.

ADD REPLY •
1
Entering edit mode

Fair. I was following up with this yaml with specific verions that _are_ compatible, but I accidentally posted early (hence the initial edit). FWIW, here is an example yaml with pinned dependencies that works to highlight the different versions that would be required to successfully resolve:

name: master
channels:
  - conda-forge
  - bioconda
  - r
  - defaults
dependencies:
  - bedtools=2.30.0=h468198e_3
  - bioconductor-biocgenerics=0.44.0=r42hdfd78af_0
  - bioconductor-deseq2=1.38.0=r42hc247a5b_0
  - bioconductor-genomicalignments=1.34.0=r42hc0cfd56_0
  - bioconductor-genomicranges=1.50.0=r42hc0cfd56_0
  - bioconductor-rsamtools=2.14.0=r42hc247a5b_0
  - bioconductor-rsubread=2.12.0=r42hc0cfd56_0
  - pip=23.0.1=pyhd8ed1ab_0
  - r-base=4.2.2=h6b4767f_2
  - r-biocmanager=1.30.19=r42hc72bb7e_0
  - r-dplyr=1.1.0=r42h38f115c_0
  - r-ggplot2=3.4.1=r42hc72bb7e_0
  - r-ggridges=0.5.4=r42hc72bb7e_1
  - samtools=1.16.1=h6899075_1
  - pip:
    - htseq==2.0.2
ADD REPLY •
0
Entering edit mode

Edit your answer and replace it with this list. Also, do you know why your htseq dependency added a pip dependency?

ADD REPLY •
0
Entering edit mode

Done - replaced with full dependency yaml. Also, I'm unsure exactly why mamba resolves the env using pip for htseq rather than by using the htseq version on bioconda. I _think_ it's another dependency issue, but now it's getting beyond me :)

ADD REPLY •
0
Entering edit mode

The idea with adding specific versions of each dependency was to standardize the environment across different operating systems and also ensure the code to be more robust so that future dependencies don't clash. I'm ultimately writing a code package that is meant to be published so defining the specific dependencies also ensures reproducibility.

ADD REPLY •
0
Entering edit mode

Apologies, I should have been more clear. I agree that specifying dependencies is a good thing. I was more meaning why _those_ specific versions. For example, why specifically r-base=4.2.2 and bioconductor-rsubread=2.8.1? If I get dependency issues with specific versions I troubleshoot by removing pinning of versions, seeing which versions _are_compatible, and then seeing if the resolved environment versions are acceptable. If they are, I'll make a new environment.yml file pinning _those_ versions. If not, I'll look further into which versions are causing the original issues. Make sense? :)

ADD REPLY •
0
Entering edit mode

I gotchu, tbh for most of the dependencies, I just picked the most up to date one I found in the anaconda package repository that was cross platform for both osx and linux. Rsubread 2.8.1 is the latest one available for osx, whereas linux has 2.12.0.

ADD REPLY •

Login before adding your answer.

Similar Posts
Loading Similar Posts
Traffic: 1909 users visited in the last hour
Content Search
Users
Tags
Badges
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6