Question: I'm putting a pipeline into docker, where do I store the tar files for my dependencies, on GitHub or within the Docker image?
0
gravatar for SaltedPork
8 months ago by
SaltedPork80
SaltedPork80 wrote:

Lets say my pipeline requires me to install Samtools from the tar file. Would it make more sense to keep this tar file within the docker image in a folder, or should I put it on GitHub and have it be downloaded when using a command like ?

RUN git clone www.linktogithub.com
docker pipeline • 371 views
ADD COMMENTlink modified 8 months ago by Eric Lim960 • written 8 months ago by SaltedPork80
1
gravatar for Eric Lim
8 months ago by
Eric Lim960
Boston
Eric Lim960 wrote:

I typically use conda.

 RUN mkdir -p /tmp/conda-build && wget -nv https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh && bash Miniconda3-latest-Linux-x86_64.sh -b -p /miniconda3
RUN cp /miniconda3/bin/conda /miniconda3/bin/conda3
ENV PATH=/miniconda3/bin:$PATH
RUN conda3 update conda
RUN conda3 config --add channels bioconda
RUN conda3 install samtools -c bioconda -y

If you need to clone a private repo, you can use multi-stage build with --squash to hide repository keys from your image.

ADD COMMENTlink modified 8 months ago • written 8 months ago by Eric Lim960

What about for programs that can't be downloaded using conda, or apt-get and their only source is a tarball, where do you keep those?

ADD REPLYlink written 8 months ago by SaltedPork80
1

Totally up to you.

If the file is local, you can use ADD or COPY. Look up the documentation to determine which one is more applicable to your use case.

If it's stored somewhere online accessible without credentials, you can just wget or git cloneand proceed to the installation as usual.

I use quite a bit of multi-stage builds to reduce the size of the final image, i.e.,

FROM ubuntu as intermediate
COPY src.zip file.zip
RUN unzip file.zip
# proceed to installation

FROM ubuntu
COPY --from intermediate /build-output/ /usr/bin/

When your image is getting large, this is an elegant way to do it.

Hope it helps.

ADD REPLYlink modified 8 months ago • written 8 months ago by Eric Lim960
1

Yep, it sure does help, thanks.

ADD REPLYlink written 8 months ago by SaltedPork80
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 2.3.0
Traffic: 1426 users visited in the last hour