katacontainers v2编译

郑茂勋

2023-12-01

开发环境

设备信息

操作系统：Centos 8 发行版

Hypervisor：VMWare

[root@localhost ~]# cat /etc/*release
CentOS Linux release 8.2.2004 (Core) 
NAME="CentOS Linux"
VERSION="8 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="8"

CentOS Linux release 8.2.2004 (Core) 
CentOS Linux release 8.2.2004 (Core) 
[root@localhost ~]#
[root@localhost ~]# uname -a
Linux 192.168.72.138 4.18.0-193.el8.x86_64 #1 SMP Fri May 8 10:59:10 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
[root@localhost ~]# lscpu
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              2
On-line CPU(s) list: 0,1
Thread(s) per core:  1
Core(s) per socket:  2
Socket(s):           1
NUMA node(s):        1
Vendor ID:           GenuineIntel
CPU family:          6
Model:               142
Model name:          Intel(R) Core(TM) i7-7500U CPU @ 2.70GHz
Stepping:            9
CPU MHz:             2904.008
BogoMIPS:            5808.01
Virtualization:      VT-x
Hypervisor vendor:   VMware
Virtualization type: full
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            4096K
NUMA node0 CPU(s):   0,1
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc cpuid aperfmperf pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch epb invpcid_single pti tpr_shadow vnmi ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid rdseed adx smap xsaveopt dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp
[root@localhost ~]# free -h
           total        used        free      shared  buff/cache   available
Mem:          3.7Gi       1.1Gi       299Mi       4.0Mi       2.3Gi       2.3Gi
Swap:         3.9Gi        16Mi       3.9Gi
[root@localhost ~]# df -h
Filesystem           Size  Used Avail Use% Mounted on
devtmpfs             1.9G     0  1.9G   0% /dev
tmpfs                1.9G     0  1.9G   0% /dev/shm
tmpfs                1.9G  9.8M  1.9G   1% /run
tmpfs                1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/mapper/cl-root   46G   11G   35G  24% /
/dev/sda1            976M  193M  716M  22% /boot
tmpfs                376M  1.2M  375M   1% /run/user/42
tmpfs                376M  4.0K  376M   1% /run/user/0
overlay               46G   11G   35G  24% /var/lib/docker/overlay2/dfa135f70d10d320af252c3773ed7a4d200f9378d5614a3220cfa3b8bc8e6765/merged
[root@localhost ~]#

安装环境依赖组件

yum install -y gcc make git openssl

编译环境搭建

下载kata安装包

wget https://github.com/kata-containers/kata-containers/archive/refs/tags/2.0.2.tar.gz

安装golang开发环境

wget https://dl.google.com/go/go1.16.3.linux-amd64.tar.gz

解压golang安装包

tar -C /usr/local -xf go1.16.3.linux-amd64.tar.gz

配置golang开发环境

mkdir /root/go
echo "export GOROOT=/usr/local/go" >> /etc/profile
echo "export GOPATH=/root/go" >> /etc/profile
echo "export GO111MODULE=on" >> /etc/profile
echo "export PATH=$PATH:$GOPATH/bin" >> /etc/profile
source /etc/profile

安装rust环境

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
根据提示选择默认配置：1
安装完成后配置环境变量
source $HOME/.cargo/env
创建config文件，配置rust加速源
[root@localhost ~]# cat /root/.cargo/config 
[source.crates-io]
registry = "https://github.com/rust-lang/crates.io-index"
replace-with = 'ustc'
[source.ustc]
registry = "git://mirrors.ustc.edu.cn/crates.io-index"

编译kata-containers v2

kata-containers release 2.0.2版本顶层目录make编译会因为rust编译失败，所以分别进入不同组件目录进行编译；或者编辑顶级目录的Makefile注释“TOOLS += agent-ctl”配置，取消agent-ctl编译可以编译出kata-runtime和kata-agent

编译kata-runtime

cd kata-containers-2.0.2/src/runtime
make
第一次编译会出现找不到yq命令错误提示：INFO: yq was not found, installing it
编译脚本会在golang环境中自动下载，编译完成后执行：make install 完成kata-runtime安装
安装完成后会显示二进制以及配置文件安装详情：
• Summary:
        destination install path (DESTDIR) : /
        binary installation path (BINDIR) : /usr/local/bin
        binaries to install :
         - /usr/local/bin/kata-runtime
         - /usr/local/bin/containerd-shim-kata-v2
         - /usr/local/bin/kata-monitor
         - /usr/libexec/kata-containers/kata-netmon
         - /usr/local/bin/data/kata-collect-data.sh
        configs to install (CONFIGS) :
         - cli/config/configuration-acrn.toml
         - cli/config/configuration-clh.toml
         - cli/config/configuration-fc.toml
         - cli/config/configuration-qemu.toml
        install paths (CONFIG_PATHS) :
         - /usr/share/defaults/kata-containers/configuration-acrn.toml
         - /usr/share/defaults/kata-containers/configuration-clh.toml
         - /usr/share/defaults/kata-containers/configuration-fc.toml
         - /usr/share/defaults/kata-containers/configuration-qemu.toml
        alternate config paths (SYSCONFIG_PATHS) : 
         - /etc/kata-containers/configuration-acrn.toml
         - /etc/kata-containers/configuration-clh.toml
         - /etc/kata-containers/configuration-fc.toml
         - /etc/kata-containers/configuration-qemu.toml
        default install path for qemu (CONFIG_PATH) : /usr/share/defaults/kata-containers/configuration.toml
        default alternate config path (SYSCONFIG) : /etc/kata-containers/configuration.toml
        qemu hypervisor path (QEMUPATH) : /usr/bin/qemu-system-x86_64
        cloud-hypervisor hypervisor path (CLHPATH) : /usr/bin/cloud-hypervisor
        firecracker hypervisor path (FCPATH) : /usr/bin/firecracker
        acrn hypervisor path (ACRNPATH) : /usr/bin/acrn-dm
        assets path (PKGDATADIR) : /usr/share/kata-containers
        shim path (PKGLIBEXECDIR) : /usr/libexec/kata-containers

执行kata-runtime kata-check检测kata-runtime运行环境是否正常，如果运行在vmware环境中会因为vmware加载了一些 vmware的模块并使用 vsock从而产生冲突

WARN[0000] modprobe insert module failed: modprobe: ERROR: could not insert 'vhost_vsock': Device or resource busy  arch=amd64 error="exit status 1" module=vhost_vsock name= pid=159576 source=runtime
ERRO[0000] kernel property not found                     arch=amd64 description="Host Support for Linux VM Sockets" name=vhost_vsock pid=159576 source=runtime type=module

创建/etc/modprobe.d/blacklist-vmware.conf文件，限制系统启动加载vmware模块

blacklist vmw_vsock_virtio_transport_common
blacklist vmw_vsock_vmci_transport

编译kata-agent

cd kata-containers-2.0.2/src/agent
make
第一次编译出现如下错误：
error[E0463]: can't find crate for `core`
  |
  = note: the `x86_64-unknown-linux-musl` target may not be installed

rust安装x86_64-unknown-linux-musl：
rustup target add x86_64-unknown-linux-musl
重新编译

编译rootfs

编译rootfs过程会下载、安装依赖组件，最好使用docker环境进行编译避免安装过多组件在系统，所以在准备开发环境时卸载了podman，安装了docker

cd /root/katav2/kata-containers-2.0.2/tools/osbuilder/rootfs-builder
编译rootfs
script -fec 'sudo -E USE_DOCKER=true SECCOMP=no ./rootfs.sh centos'
编译过程中可能会因为网络原因导致编译失败：
Step 6/12 : RUN cd /tmp ; curl -OL https://storage.googleapis.com/golang/go1.14.4.linux-amd64.tar.gz
 ---> Running in 473c0ff8c147
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:21 --:--:--     0
curl: (35) TCP connection reset by peer
重新编译

如果需要编译带console的镜像需要在制作镜像时指定添加额外的组件

script -fec 'USE_DOCKER=true EXTRA_PKGS="bash coreutils" ./rootfs.sh centos'

成功编译rootfs后会有如下提示，会在rootfs中自动安装kata-agent和kata-agent的systemd文件

Installing systemd unit files...
install -D -m 644 kata-agent.service /root/katav2/kata-containers-2.0.2/tools/osbuilder/rootfs-builder/rootfs/usr/lib/systemd/system/kata-agent.service || exit 1;      install -D -m 644 kata-containers.target /root/katav2/kata-containers-2.0.2/tools/osbuilder/rootfs-builder/rootfs/usr/lib/systemd/system/kata-containers.target || exit 1;
~/katav2/kata-containers-2.0.2/tools/osbuilder/rootfs-builder
[OK] Agent installed
INFO: Check init is installed
[OK] init is installed
INFO: Create /etc/resolv.conf file in rootfs if not exist
INFO: Creating summary file
INFO: Created summary file '/var/lib/osbuilder/osbuilder.yaml' inside rootfs
Script done, file is typescript

编译rootfs image

进入编译image目录
cd /root/katav2/kata-containers-2.0.2/tools/osbuilder/image-builder

执行编译脚本
script -fec 'sudo -E USE_DOCKER=true ./image_builder.sh /root/katav2/kata-containers-2.0.2/tools/osbuilder/rootfs-builder/rootfs'

成功后如下提示
OK!
1+0 records in
1+0 records out
2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.00454381 s, 462 MB/s
1044480+0 records in
1044480+0 records out
534773760 bytes (535 MB, 510 MiB) copied, 15.3046 s, 34.9 MB/s
Script done, file is typescript

编译kata kernel

cd /root/katav2/kata-containers-2.0.2/tools/packaging/kernel
配置kernel编译环境
./build-kernel.sh setup
出现如下错误是因为编译内核时找不到内核.config文件，原因是build-kernel.sh编译脚本中force_setup_generate_config变量默认false，导致没有执行到make oldconfig命令，所以没有.config文件
*** Configuration file ".config" not found!
***
*** Please run some configurator (e.g. "make oldconfig" or
*** "make menuconfig" or "make xconfig").
***
编译内核
./build-kernel.sh build
安装内核文件到/usr/share/kata-containers/目录
./build-kernel.sh install