I'm putting a pipeline into docker, where do I store the tar files for my dependencies, on GitHub or within the Docker image?
1
0
Entering edit mode
3.8 years ago
SaltedPork ▴ 120

Lets say my pipeline requires me to install Samtools from the tar file. Would it make more sense to keep this tar file within the docker image in a folder, or should I put it on GitHub and have it be downloaded when using a command like ?

RUN git clone www.linktogithub.com
docker pipeline • 1.6k views
ADD COMMENT
1
Entering edit mode
3.8 years ago
Eric Lim ★ 1.7k

I typically use conda.

 RUN mkdir -p /tmp/conda-build && wget -nv https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh && bash Miniconda3-latest-Linux-x86_64.sh -b -p /miniconda3
RUN cp /miniconda3/bin/conda /miniconda3/bin/conda3
ENV PATH=/miniconda3/bin:$PATH
RUN conda3 update conda
RUN conda3 config --add channels bioconda
RUN conda3 install samtools -c bioconda -y

If you need to clone a private repo, you can use multi-stage build with --squash to hide repository keys from your image.

ADD COMMENT
0
Entering edit mode

What about for programs that can't be downloaded using conda, or apt-get and their only source is a tarball, where do you keep those?

ADD REPLY
1
Entering edit mode

Totally up to you.

If the file is local, you can use ADD or COPY. Look up the documentation to determine which one is more applicable to your use case.

If it's stored somewhere online accessible without credentials, you can just wget or git cloneand proceed to the installation as usual.

I use quite a bit of multi-stage builds to reduce the size of the final image, i.e.,

FROM ubuntu as intermediate
COPY src.zip file.zip
RUN unzip file.zip
# proceed to installation

FROM ubuntu
COPY --from intermediate /build-output/ /usr/bin/

When your image is getting large, this is an elegant way to do it.

Hope it helps.

ADD REPLY
1
Entering edit mode

Yep, it sure does help, thanks.

ADD REPLY

Login before adding your answer.

Traffic: 1437 users visited in the last hour
Help About
FAQ
Access RSS
API
Stats

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.

Powered by the version 2.3.6