Docker里运行Docker docker in docker(dind)
目的
制作一个可以build docker镜像的docker镜像,jenkins CI服务节点,部署到阿里云的容器服务集群里.
阿里云官方有完整的镜像,master和slave的都有,时间稍微久远了一点,所以自己研究一下build个最新的版本.
关于 docker in docker
docker运行在docker里面分两种情况
- (dind) docker inside docker
- (dood) docker outside of docker
dood的好处是内部容器并不需要真正的安装docker而是挂载父级docker的socket和执行文件就可以,但是缺点也非常明显,就是内部容器必须和外部的docker环境保持一致,不然会报各种错误,缺少库文件什么的,而且安全性有很大问题,Docker-in-Docker ,贴吧翻译版本, 这个项目的作者本身就不建议这么去使用docker(dind的方式),有安全性问题,存储问题
看了这些文章之后,得出的结论如下
- 真正的docker in docker 在docker官方有镜像直接就能用 官方dind镜像
- docker in docker dind 是用来学习的,在实际生产环境中一般用不到
- docker in docker dind 是有很多问题的(意想不到的), 比如存储空间,安全等等
- 如果只是使用CI如jenkins这种,在容器里需要build镜像,那么不需要纯粹的dind, 可以变通的使用host的docker(通过二进制docker文件和宿主的sock来实现)
那么解决方案是什么
原文
The solution
Let’s take a step back here. Do you really want Docker-in-Docker? Or do you just want to be able to run Docker (specifically: build, run, sometimes push containers and images) from your CI system, while this CI system itself is in a container?
I’m going to bet that most people want the latter. All you want is a solution so that your CI system like Jenkins can start containers.
And the simplest way is to just expose the Docker socket to your CI container, by bind-mounting it with the -v flag.
Simply put, when you start your CI container (Jenkins or other), instead of hacking something together with Docker-in-Docker, start it with:
docker run -v /var/run/docker.sock:/var/run/docker.sock ...
Now this container will have access to the Docker socket, and will therefore be able to start containers. Except that instead of starting “child” containers, it will start “sibling” containers.
Try it out, using the docker official image (which contains the Docker binary):
docker run -v /var/run/docker.sock:/var/run/docker.sock \
-ti docker
This looks like Docker-in-Docker, feels like Docker-in-Docker, but it’s not Docker-in-Docker: when this container will create more containers, those containers will be created in the top-level Docker. You will not experience nesting side effects, and the build cache will be shared across multiple invocations.
Former versions of this post advised to bind-mount the docker binary from the host to the container. This is not reliable anymore, because the Docker Engine is no longer distributed as (almost) static libraries.
If you want to use e.g. Docker from your Jenkins CI system, you have multiple options:
- installing the Docker CLI using your base image’s packaging system (i.e. if your image is based on Debian, use .deb packages),
- using the Docker API.
翻译后
解决方案
退一步, 是需要一个真正的 docker in docker,还是只想在类似jenkins这样的CI系统里要运行docker命令(build push images)
我想大部分人只是需要后者,你所需要的是你的ci系统可以启动容器,或者构建容器(这是我加的)
最简单的方式是启动容器的时候以-v的方式挂载宿主机的docker的socket给含有jenkins这种CI的容器使用
简单的说,启动你的CI 容器(jenkins或者其他), 而不是用docker-in-docker上hacking, 启动方法:
docker run -v /var/run/docker.sock:/var/run/docker.sock ...
这样这个容器就会可以访问到宿主机的docker socket并使用它,因为这个容器就有能力启动容器,这样他启动和操作的容器和他本身是兄弟关系,是平级的而不是父子关系.
试试官方的docker镜像, 包含二进制可执行程序
docker run -v /var/run/docker.sock:/var/run/docker.sock -ti docker
插个楼,我再本机上做的实验
➜ jenkins docker run -v /var/run/docker.sock:/var/run/docker.sock \
-ti docker:17.05
Unable to find image 'docker:17.05' locally
17.05: Pulling from library/docker
cfc728c1c558: Pull complete 81d7b2e827ca: Pull complete 144b6478a622: Pull complete e04c57814bdc: Pull complete Digest: sha256:da22b37a64e68f7b2199ba619f61e0b82f95d6fa4ed812b2d6c19e486d933ed0 Status: Downloaded newer image for docker:17.05 / # which docker /usr/local/bin/docker / # docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES cae7df10f791 docker:17.05 "docker-entrypoint..." 20 seconds ago Up 19 seconds hungry_banach d0c7c57ee2c9 jenkinsci/jenkins:lts "/bin/tini -- /usr..." 18 hours ago Up 8 hours 50000/tcp, 0.0.0.0:49001->8080/tcp vigilant_pasteur / # docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES cae7df10f791 docker:17.05 "docker-entrypoint..." 23 seconds ago Up 22 seconds hungry_banach 512c903e80da django:test03 "/bin/bash" 10 hours ago Exited (0) 6 hours ago testdjango01 d0c7c57ee2c9 jenkinsci/jenkins:lts "/bin/tini -- /usr..." 18 hours ago Up 8 hours 50000/tcp, 0.0.0.0:49001->8080/tcp vigilant_pasteur cb5527ee98c5 jenkinsci/jenkins:lts "/bin/tini -- /usr..." 18 hours ago Created vigilant_easley fcd7e2089a0d 12c901436f44 "/bin/sh -c 'echo ..." 19 hours ago Exited (127) 19 hours ago amazing_easley 0fb85d775afc 04b5d101ead1 "/bin/sh -c 'set -..." 20 hours ago Exited (2) 19 hours ago gracious_swirles 2ff8dce9c205 ee4682994d35 "/bin/bash" 20 hours ago Exited (0) 19 hours ago testpython2 可以清楚的看到,这些结果都是宿主机上的docker的结果,所以这里使用的docker命令实际上的空间和进程都是在宿主机上的.
可以看一些这个镜像的Dockfile
穿越回来,继续翻译
这看起来像是docker-in-docker,但是实际上不是,当你要在这个容器里创建新的容器的时候,这些容器并不是在容器里被创建的,而是在host机器上创建的,你将不会受到内嵌的影响,构建缓存也在多实例间共享.
这篇文章的老版本建议bind docker的二进制文件从host到容器,这个说法已经不再可靠,因为docker 引擎不再发布静态库,所以单纯用挂载的方式使docker的二进制文件在容器和host间共享的方式不存在了,取而代之的是直接在容器内解压一个编译好的docker.
所以现在如果你想使用docker在jenkins CI的容器里,你有以下选择:
- 安装docker cli,在你的基础docker镜像里(比如,你使用debian, 就使用 deb packages)
- 使用remote api
我的方案
好了看了这么多,是时候自己构建一个供jenkins子节点使用的镜像了,包含docker,可以build和push镜像
我的场景如下
- 阿里云上用几台机器组件了容器的集群
- 现在要搭建一个CI的demo(jenkins的)
- jenkins是master agent模式,不同节点环境不一样的,有的是python的,有的是java的
- 子节点在有代码更新的时候,触发任务,任务很简单,就是用Dockerfile构建新的镜像
- 新的镜像构建完成以后,直接推送(走阿里云的内网,不用流量费)
- 镜像有更新,触发容器集群重新部署应用.
需要的镜像
- 1个jenkins master容器
- 1个jenkins agent容器 用来build镜像并推送到镜像仓库
- 一个python+mysql 编排
最后附上我已经配置好的Dockerfile(已经在阿里云的容器里运行)
FROM ubuntu:xenial
MAINTAINER Vincent Wantchalk <ohergal@gmail.com>
RUN apt-get update
RUN apt-get -y install wget \
apt-utils \
locales \
tzdata \
\
build-essential \
apt-transport-https \
ca-certificates \
curl \
git \
openssh-server \
zlib1g \
zlib1g.dev \
openssl \
libssl-dev \
iptables && \
rm -rf /var/lib/apt/lists/*
RUN locale-gen en_US.UTF-8
RUN dpkg-reconfigure locales
RUN echo "Asia/shanghai" > /etc/timezone
RUN ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime \
&& dpkg-reconfigure -f noninteractive tzdata
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8
# To avoid annoying "perl: warning: Setting locale failed." errors,
# do not allow the client to pass custom locals, see:
# http://stackoverflow.com/a/2510548/15677
# RUN sed -i 's/^AcceptEnv LANG LC_\*$//g' /etc/ssh/sshd_config
RUN apt-get -q update &&\
DEBIAN_FRONTEND="noninteractive" apt-get -q install -y -o Dpkg::Options::="--force-confnew" --no-install-recommends software-properties-common && \
add-apt-repository -y ppa:openjdk-r/ppa &&\
apt-get -q update &&\
DEBIAN_FRONTEND="noninteractive" apt-get -q install -y -o Dpkg::Options::="--force-confnew" --no-install-recommends openjdk-8-jdk && \
apt-get -q clean -y && rm -rf /var/lib/apt/lists/* && rm -f /var/cache/apt/*.bin && \
sed -i 's|session required pam_loginuid.so|session optional pam_loginuid.so|g' /etc/pam.d/sshd &&\
mkdir -p /var/run/sshd
# clean compile tools
RUN apt-get purge -y --auto-remove build-essential libssl-dev
RUN useradd -m -d /home/jenkins -s /bin/bash jenkins && \
echo "jenkins:jenkins" | chpasswd
# install docker binary files
ENV DOCKER_BUCKET get.docker.com
ENV DOCKER_VERSION 17.05.0-ce
ENV DOCKER_SHA256_x86_64 340e0b5a009ba70e1b644136b94d13824db0aeb52e09071410f35a95d94316d9
ENV DOCKER_SHA256_armel 59bf474090b4b095d19e70bb76305ebfbdb0f18f33aed2fccd16003e500ed1b7
RUN curl -fSL "https://${DOCKER_BUCKET}/builds/Linux/x86_64/docker-${DOCKER_VERSION}.tgz" -o docker.tgz \
&& echo "${DOCKER_SHA256_x86_64} *docker.tgz" | sha256sum -c - \
&& tar -xzvf docker.tgz \
&& mv docker/* /usr/local/bin/ \
&& rmdir docker \
&& rm docker.tgz \
&& chmod +x /usr/local/bin/docker
# && docker -v \
# && docker ps -a
RUN echo "jenkins ALL=NOPASSWD: ALL" >> /etc/sudoers
# Standard SSH port
EXPOSE 22
COPY docker-entrypoint.sh /usr/local/bin/
ENTRYPOINT ["docker-entrypoint.sh"]
# Default command
CMD ["/usr/sbin/sshd", "-D"]