docker 命令实用指南

Docker 简介

省略。

Docker 命令

• run - docker images 运行
• images - docker images 查看
• ps - docker containers 查看
• rm - docker containers 删除
• rmi - docker images 删除
• pull - dockerhub images 拉取
• push - dockerhub images 更新
• commit - dockerhub images 提交
• exec - docker containers 运行时修改

镜像

创建镜像

方式一：下载

# 下载指定版本容器镜像
docker pull gitlab/gitlab-ce:11.2.3-ce.0

方式二：从已经创建的容器中更新镜像，并且提交这个镜像

docker commit -m="First Docker" -a="wcjiang" a6b0a6cfdacf wcjiang/nginx:v1.2.1

上面命令参数说明：

• m 提交的描述信息
• a 指定镜像作者
• a6b0a6cfdacf 记住这个是容器id，不是镜像id
• wcjiang/nginx:v1.2.1 创建的目标镜像名

方式三：使用 Dockerfile 指令来创建一个新的镜像

参考 https://github.com/jaywcjlove/docker-tutorial

发布镜像

参考 https://github.com/jaywcjlove/docker-tutorial

导出镜像为本地文件

常用于分享。

[Saving Images and Containers as Tar Files for Sharing | dockerlabs (collabnix.com)](https://dockerlabs.collabnix.com/beginners/saving-images-as-tar/#:~:text=The docker export - Export a,a tar archive or STDIN)

容器

容器管理

# 列出包括未运行的容器
docker ps
# 查看所有容器(包括正在运行和已停止的)
docker ps -a

# 列出本机正在运行的容器
docker container ls

docker start [containerID/Names] # 启动容器
docker stop [containerID/Names]  # 停止容器
docker rm [containerID/Names]    # 删除容器
docker logs [containerID/Names]  # 查看日志

容器使用

创建

# 创建一个守护状态的 Docker 容器
docker run -itd my-nginx /bin/bash

进入容器

# 进入容器
docker exec -it [containerID/Names] /bin/bash  

一次性使用

# 在docker容器中运行hello world!
docker run centos echo "hello world"  
# 在docker容器中，安装wget软件
docker run centos yum install -y wget 

文件拷贝

从主机复制到容器 sudo docker cp host_path containerID:container_path
从容器复制到主机 sudo docker cp containerID:container_path host_path

# 从正在运行的 Docker 容器里面，将文件拷贝到本机，注意后面有个【点】拷贝到当前目录
docker container cp [containID]:[/path/to/file] .

Docker GPU 使用

https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/user-guide.html

例子

# 参数说明：
# --rm		Automatically remove the container when it exits

1. Starting a GPU enabled CUDA container; using --gpus （常用）

   docker run --rm --gpus all nvidia/cuda nvidia-smi
   

2. Using NVIDIA_VISIBLE_DEVICES and specify the nvidia runtime （常用）

   docker run --rm --runtime=nvidia \
     -e NVIDIA_VISIBLE_DEVICES=all nvidia/cuda nvidia-smi
   

3. Start a GPU enabled container on two GPUs （常用）

   docker run --rm --gpus 2 nvidia/cuda nvidia-smi
   

4. Starting a GPU enabled container on specific GPUs

   docker run --gpus '"device=1,2"' \
     nvidia/cuda nvidia-smi --query-gpu=uuid --format=csv
       
   # uuid
   # GPU-ad2367dd-a40e-6b86-6fc3-c44a2cc92c7e
   # GPU-16a23983-e73e-0945-2095-cdeb50696982
   

5. Alternatively, you can also use NVIDIA_VISIBLE_DEVICES （常用）

   docker run --rm --runtime=nvidia \
     -e NVIDIA_VISIBLE_DEVICES=1,2 \
     nvidia/cuda nvidia-smi --query-gpu=uuid --format=csv
       
   # uuid
   # GPU-ad2367dd-a40e-6b86-6fc3-c44a2cc92c7e
   # GPU-16a23983-e73e-0945-2095-cdeb50696982
   

6. Query the GPU UUID using nvidia-smi and then specify that to the container

   nvidia-smi -i 3 --query-gpu=uuid --format=csv
   
   # uuid
   # GPU-18a3e86f-4c0e-cd9f-59c3-55488c4b0c24
   
   docker run --gpus device=GPU-18a3e86f-4c0e-cd9f-59c3-55488c4b0c24 \
     nvidia/cuda nvidia-smi
   

Tutorials

Dockerfile

FROM centos:7.2.1511

# yum basic tool
RUN yum clean all \
 && yum makecache \
 && yum install -y wget \
 && mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup \
 && wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo \
 && yum makecache \
 && yum install -y gcc gcc-c++ python-devel bzip2 \
 && yum install -y make \
 && yum install -y epel-release \
 && yum install -y python-devel \
 && yum install -y python-pip \
 && yum install -y mysql-devel \
 && yum clean all \
 && pip install --upgrade pip -i http://pypi.douban.com/simple/ --trusted-host pypi.douban.com \
 && pip install setuptools==33.1.1 -i http://pypi.douban.com/simple/ --trusted-host pypi.douban.com

##守护进程，可配置多进程
#RUN yum install -y python-setuptools
#    && pip install supervisor -i http://pypi.douban.com/simple/ --trusted-host pypi.douban.com \
#    && mkdir -m 777 /tmp
#    && mkdir -m 755 -p /etc/supervisor/conf.d
#COPY supervisor.conf /etc/supervisor/supervisord.conf
#
##镜像启动命令
#CMD ["supervisord","-c","/etc/supervisor/supervisord.conf"]
COPY resource/CRF++-0.58.tar.gz /tmp

# MySQL and CRFPP
RUN pip install MySQL-python -i http://pypi.douban.com/simple/ --trusted-host pypi.douban.com \
 && pip install Cython -i http://pypi.douban.com/simple/ --trusted-host pypi.douban.com \
 && cd /tmp \
 && tar -xzvf CRF++-0.58.tar.gz \
 && cd CRF++-0.58 \
 && ./configure \
 && make \
 && make install \
 && cd python \
 && python setup.py build \
 && python setup.py install \
 && cd .. \
 && rm -rf CRF++-0.58 \
 && rm -f CRF++-0.58.tar.gz \
 && mkdir -p /code/example_project/src \
 && mkdir -p /code/example_project/conf \
 && mkdir -p /code/example_project/admin
COPY setup.py /code/example_project
COPY src /code/example_project/src
COPY conf /code/example_project/conf
COPY admin /code/example_project/admin
RUN cd /code/example_project \
 && python2 setup.py build_ext
CMD cd /code/example_project/admin \
 && sh start_server.sh

build_docker_image.sh

#!/usr/bin/env bash
#检查是否输入镜像tag的version 信息
version=$1
if [ -z $version ]
then
    version="pingan_20180131"
fi
cd ..
# 清理日志，也可以在后面做文件模型拷贝等批处理信息
rm -rf log/*
docker build -t dockerhub.example.com/pingan_contract/seq2seq_offline:$version .

start_docker_server.sh

#!/usr/bin/env bash
name=`docker ps -a|grep pingan_contract_seq2seq_offline|awk '{print $NF}'`
if [ -z $name ]
then
    docker run -itd -m 256g -v /Users/lotus/PycharmProjects/example_project/admin:/code/example_project/admin -v /Users/lotus/PycharmProjects/example_project/log:/code/example_project/log -v /Users/lotus/PycharmProjects/example_project/data:/code/example_project/data --name example_project_seq2seq_offline dockerhub.example.com/example_project/seq2seq_offline:example_20180131
else
    docker start example_project_seq2seq_offline
fi

start_server.sh

#!/usr/bin/env bash

base_home="/code/example_project"
base_source_path="${base_home}/src"
base_data_path="${base_home}/data"
base_conf_path="${base_home}/conf"

cd ${base_source_path}
python3 main.py

stop_docker_server.sh

#!/usr/bin/env bash
docker stop example_project_seq2seq_offline

stop_server.sh

#!/usr/bin/env bash
ps aux|grep python|grep main|awk '{print $2}'|xargs kill -9

docker run 例子2

docker run --runtime=nvidia -it -d --name alex_conda \
  -p <port>:8888 \
  -v /etc/passwd:/etc/passwd:ro \
  -v /etc/group:/etc/group:ro \
  -v $HOME:$HOME \
  -v /mnt:/mnt \
  -v conda_pkgs:/opt/conda/pkgs \
  -v conda_envs:/opt/conda/envs \
  -u `id -u`:`id -g` \
  -w $HOME \
  conda_jupyter

docker run --runtime=nvidia -it -d --name example_container -p 6688:8888 -p 6622:22 -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro -v $HOME:$HOME -v /mnt:/mnt -v conda_pkgs:/opt/conda/pkgs -v conda_envs:/opt/conda/envs -u `id -u`:`id -g` -w $HOME conda_jupyter

# 例子：root 身份进入 容器
docker exec -u 0 -it example_container bash

docker exec：使用 root 身份进入容器

docker exec -u 0 -it 容器名 bash

docker 开启 ssh 服务

https://www.jianshu.com/p/94a0a95c9a2c

deep learning dockerfile资源

AWS

https://github.com/aws/deep-learning-containers.git

# 获取 dockerfile
# 位置：https://github.com/aws/deep-learning-containers/blob/master/pytorch/training/docker/1.11/py3/cu113/Dockerfile.gpu
git clone https://github.com/aws/deep-learning-containers.git

# build image
cd deep-learning-containers
cp ./pytorch/training/docker/build_artifacts/changehostname.c ./src
cp ./pytorch/training/docker/build_artifacts/start_with_right_hostname.sh ./src
cd ./src

# build 镜像
# 如果失败，重试几次就可以，需要保证网速
nohup docker build -f /home/xiaqi.xia/workspaces/deep-learning-containers/pytorch/training/docker/1.11/py3/cu113/Dockerfile.gpu -t aws/deep-learning-containers:pytorch1.11-cuda11.3.1-ubuntu20.04 . &

# 创建容器
docker run --runtime=nvidia -it -d --name pytorch_middle_language -p 7788:8888 -p 7722:22 -v /etc/passwd:/etc/passwd:ro -v /etc/group:/etc/group:ro -v $HOME:$HOME -v /mnt:/mnt -v conda_pkgs:/opt/conda/pkgs -v conda_envs:/opt/conda/envs -u `id -u`:`id -g` -w $HOME aws/deep-learning-containers:pytorch1.11-cuda11.3.1-ubuntu20.04

# 进入容器
docker exec -it pytorch_middle_language '/bin/bash'
# root 用户 进入
docker exec -u 0 -it pytorch_middle_language bash

# 开启 ssh 服务
https://www.jianshu.com/p/94a0a95c9a2c

# 创建 conda 虚拟环境
conda create --name kg-bart --clone base

# 服务器上命令方式执行
CUDA_VISIBLE_DEVICES=0 python run_seq2seq.py --data_dir ../dataset/commongen_data/commongen --output_dir ../output/KGBart --log_dir ../log/KGBart --model_recover_path ../output/Pretraining_KG/best_model/model.best.bin --fp16 True --max_seq_length 32 --max_position_embeddings 64 --max_len_a 32 --max_len_b 64 --max_pred 64 --train_batch_size 60 --eval_batch_size 48 --gradient_accumulation_steps 6 --learning_rate 0.00001 --warmup_proportion 0.1 --label_smoothing 0.1 --num_train_epochs 10

anibali/docker-pytorch

https://github.com/anibali/docker-pytorch/blob/master/dockerfiles/1.11.0-cuda11.5-ubuntu20.04/Dockerfile

Solved Bugs

解决：docker build E: Failed to fetch http://security.ubuntu.com

方法一

https://stackoverflow.com/questions/37706635/in-docker-apt-get-install-fails-with-failed-to-fetch-http-archive-ubuntu-com

FROM ubuntu:14.04
RUN sed -i'' 's/archive\.ubuntu\.com/us\.archive\.ubuntu\.com/' /etc/apt/sources.list
RUN apt-get -y update

方法二

如果方法一不管用，使用方法二。apt get - Ubuntu apt-get unable to fetch packages - Stack Overflow

# add the following to your resolve.conf file /etc/resolv.conf
# nameserver 8.8.8.8
# nameserver 8.8.4.4
RUN echo "nameserver 8.8.8.8" >> /etc/resolv.conf && echo "nameserver 8.8.4.4" >> /etc/resolv.conf

解决：docker build E: Failed to fetch https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/

https://github.com/deepmind/alphafold/issues/4

# 方法一：（已尝试）
Try this:
Modify the Dockerfile like this if you are in China:

.................  # some command

RUN echo deb https://developer.download.nvidia.cn/compute/cuda/repos/ubuntu2004/x86_64 / | tee /etc/apt/sources.list.d/cuda.list    # HERE! Add this before apt-get update

RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y \
.................  # some command
And you may also unable to access the github depository. So you can modify the website of https://github.com/soedinglab/hh-suite.git to https://github.com.cnpmjs.org/soedinglab/hh-suite.git

# 方法二：（未尝试）
docker build —add-host “developer.download.nvidia.com:152.195.19.142 -f docker/Dockerfile -t alphafold .

其他教程

史上讲解最好的 Docker 教程，从入门到精通（建议收藏的教程） (qq.com)

References

1. https://github.com/jaywcjlove/docker-tutorial
2. https://github.com/NVIDIA/nvidia-docker