Question: I'm putting a pipeline into docker, where do I store the tar files for my dependencies, on GitHub or within the Docker image?
0
gravatar for SaltedPork
11 months ago by
SaltedPork80
SaltedPork80 wrote:

Lets say my pipeline requires me to install Samtools from the tar file. Would it make more sense to keep this tar file within the docker image in a folder, or should I put it on GitHub and have it be downloaded when using a command like ?

RUN git clone www.linktogithub.com
docker pipeline • 464 views
ADD COMMENTlink modified 11 months ago by Eric Lim1.2k • written 11 months ago by SaltedPork80
1
gravatar for Eric Lim
11 months ago by
Eric Lim1.2k
Boston
Eric Lim1.2k wrote:

I typically use conda.

 RUN mkdir -p /tmp/conda-build && wget -nv https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh && bash Miniconda3-latest-Linux-x86_64.sh -b -p /miniconda3
RUN cp /miniconda3/bin/conda /miniconda3/bin/conda3
ENV PATH=/miniconda3/bin:$PATH
RUN conda3 update conda
RUN conda3 config --add channels bioconda
RUN conda3 install samtools -c bioconda -y

If you need to clone a private repo, you can use multi-stage build with --squash to hide repository keys from your image.

ADD COMMENTlink modified 11 months ago • written 11 months ago by Eric Lim1.2k

What about for programs that can't be downloaded using conda, or apt-get and their only source is a tarball, where do you keep those?

ADD REPLYlink written 11 months ago by SaltedPork80
1

Totally up to you.

If the file is local, you can use ADD or COPY. Look up the documentation to determine which one is more applicable to your use case.

If it's stored somewhere online accessible without credentials, you can just wget or git cloneand proceed to the installation as usual.

I use quite a bit of multi-stage builds to reduce the size of the final image, i.e.,

FROM ubuntu as intermediate
COPY src.zip file.zip
RUN unzip file.zip
# proceed to installation

FROM ubuntu
COPY --from intermediate /build-output/ /usr/bin/

When your image is getting large, this is an elegant way to do it.

Hope it helps.

ADD REPLYlink modified 11 months ago • written 11 months ago by Eric Lim1.2k
1

Yep, it sure does help, thanks.

ADD REPLYlink written 11 months ago by SaltedPork80
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 1090 users visited in the last hour