Question: I'm putting a pipeline into docker, where do I store the tar files for my dependencies, on GitHub or within the Docker image?
0
gravatar for SaltedPork
4 days ago by
SaltedPork50
SaltedPork50 wrote:

Lets say my pipeline requires me to install Samtools from the tar file. Would it make more sense to keep this tar file within the docker image in a folder, or should I put it on GitHub and have it be downloaded when using a command like ?

RUN git clone www.linktogithub.com
docker pipeline • 116 views
ADD COMMENTlink modified 4 days ago by Eric Lim370 • written 4 days ago by SaltedPork50
1
gravatar for Eric Lim
4 days ago by
Eric Lim370
Stoke Therapeutics, Inc
Eric Lim370 wrote:

I typically use conda.

 RUN mkdir -p /tmp/conda-build && wget -nv https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh && bash Miniconda3-latest-Linux-x86_64.sh -b -p /miniconda3
RUN cp /miniconda3/bin/conda /miniconda3/bin/conda3
ENV PATH=/miniconda3/bin:$PATH
RUN conda3 update conda
RUN conda3 config --add channels bioconda
RUN conda3 install samtools -c bioconda -y

If you need to clone a private repo, you can use multi-stage build with --squash to hide repository keys from your image.

ADD COMMENTlink modified 4 days ago • written 4 days ago by Eric Lim370

What about for programs that can't be downloaded using conda, or apt-get and their only source is a tarball, where do you keep those?

ADD REPLYlink written 4 days ago by SaltedPork50
1

Totally up to you.

If the file is local, you can use ADD or COPY. Look up the documentation to determine which one is more applicable to your use case.

If it's stored somewhere online accessible without credentials, you can just wget or git cloneand proceed to the installation as usual.

I use quite a bit of multi-stage builds to reduce the size of the final image, i.e.,

FROM ubuntu as intermediate
COPY src.zip file.zip
RUN unzip file.zip
# proceed to installation

FROM ubuntu
COPY --from intermediate /build-output/ /usr/bin/

When your image is getting large, this is an elegant way to do it.

Hope it helps.

ADD REPLYlink modified 4 days ago • written 4 days ago by Eric Lim370

Yep, it sure does help, thanks.

ADD REPLYlink written 4 days ago by SaltedPork50
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 1222 users visited in the last hour