从Fedora CoreOS(Fedora35)到CentOS-Stream9的重构

从Fedora CoreOS(Fedora35)到CentOS-Stream9的重构
荣涛 2022-03-16

文档修改日志

日期	修改内容	修改人	备注
2022-03-16	创建	荣涛
2022-03-17	systemd service失败	荣涛
2022-03-18	coreos-installer问题	荣涛

引言

Fedora-CoreOS特有的RPM包如下(忽略版本信息)：

bind-libs-32:9.16.21-1.fc35.x86_64
bind-license-32:9.16.21-1.fc35.noarch
bind-utils-32:9.16.21-1.fc35.x86_64
conmon-2:2.0.30-2.fc35.x86_64
container-selinux-2:2.170.0-1.fc35.noarch
containerd-1.5.7-1.fc35.x86_64
containernetworking-plugins-1.0.1-1.fc35.x86_64
containers-common-4:1-30.fc35.noarch
criu-3.16-2.fc35.x86_64
criu-libs-3.16-2.fc35.x86_64
crun-1.2-1.fc35.x86_64
fedora-coreos-pinger-0.0.4-12.fc35.x86_64
fedora-gpg-keys-35-1.noarch
fedora-release-common-35-33.noarch
fedora-release-coreos-35-33.noarch
fedora-release-identity-coreos-35-33.noarch
fedora-repos-35-1.noarch
fedora-repos-archive-35-1.noarch
fedora-repos-modular-35-1.noarch
fedora-repos-ostree-35-1.noarch
fwupd-1.6.4-1.fc35.x86_64
moby-engine-20.10.9-1.fc35.x86_64
podman-3:3.4.0-1.fc35.x86_64
runc-2:1.0.2-2.fc35.x86_64
skopeo-1:1.5.0-2.fc35.x86_64
systemd-resolved-249.4-2.fc35.x86_64
toolbox-0.0.99.2^3.git075b9a8d2779-4.fc35.x86_64
util-linux-core-2.37.2-1.fc35.x86_64

那么就需要将src源玛RPM包在CentOS-Stream-9上进程rpmbuild生成二进制包，这是最简单的方法。

二进制RPM包的重构

标准重构流程

构建过程简单可以概括为：

# 1. 下载源玛包（在Fedora虚拟机上或者指定dnf repo地址）
dnf download --source bind
# 2. 将源码包拷贝到用于构建的目标虚拟机或者服务器上
# 3. 安装源码包
rpm -ivh bind.xxxxxx.src.rpm
# 4. 安装依赖，注意，需要使能crb
cd ~/rpmbuild/SPEC
sudo dnf builddep --enablerepo=crb bind.spec
# 5. 如果依赖不能够满足，需要下载不满足的依赖，编译并安装
# 6. 编译
rpmbuild -bb bind.spec

bind

按照标准重构流程即可，这将生成以下RPM二进制包(忽略版本信息)：

├── noarch
│   ├── bind-dnssec-doc-9.16.24-1.el9.noarch.rpm
│   ├── bind-doc-9.16.24-1.el9.noarch.rpm
│   ├── bind-license-9.16.24-1.el9.noarch.rpm
│   └── python3-bind-9.16.24-1.el9.noarch.rpm
└── x86_64
    ├── bind-9.16.24-1.el9.x86_64.rpm
    ├── bind-chroot-9.16.24-1.el9.x86_64.rpm
    ├── bind-debuginfo-9.16.24-1.el9.x86_64.rpm
    ├── bind-debugsource-9.16.24-1.el9.x86_64.rpm
    ├── bind-devel-9.16.24-1.el9.x86_64.rpm
    ├── bind-dlz-filesystem-9.16.24-1.el9.x86_64.rpm
    ├── bind-dlz-filesystem-debuginfo-9.16.24-1.el9.x86_64.rpm
    ├── bind-dlz-ldap-9.16.24-1.el9.x86_64.rpm
    ├── bind-dlz-ldap-debuginfo-9.16.24-1.el9.x86_64.rpm
    ├── bind-dlz-mysql-9.16.24-1.el9.x86_64.rpm
    ├── bind-dlz-mysql-debuginfo-9.16.24-1.el9.x86_64.rpm
    ├── bind-dlz-sqlite3-9.16.24-1.el9.x86_64.rpm
    ├── bind-dlz-sqlite3-debuginfo-9.16.24-1.el9.x86_64.rpm
    ├── bind-dnssec-utils-9.16.24-1.el9.x86_64.rpm
    ├── bind-dnssec-utils-debuginfo-9.16.24-1.el9.x86_64.rpm
    ├── bind-libs-9.16.24-1.el9.x86_64.rpm
    ├── bind-libs-debuginfo-9.16.24-1.el9.x86_64.rpm
    ├── bind-pkcs11-9.16.24-1.el9.x86_64.rpm
    ├── bind-pkcs11-debuginfo-9.16.24-1.el9.x86_64.rpm
    ├── bind-pkcs11-devel-9.16.24-1.el9.x86_64.rpm
    ├── bind-pkcs11-libs-9.16.24-1.el9.x86_64.rpm
    ├── bind-pkcs11-libs-debuginfo-9.16.24-1.el9.x86_64.rpm
    ├── bind-pkcs11-utils-9.16.24-1.el9.x86_64.rpm
    ├── bind-pkcs11-utils-debuginfo-9.16.24-1.el9.x86_64.rpm
    ├── bind-utils-9.16.24-1.el9.x86_64.rpm
    └── bind-utils-debuginfo-9.16.24-1.el9.x86_64.rpm

conmon

依赖go-md2man，需要下载

golang-github-cpuguy83-md2man-2.0.1-1.fc35.src.rpm
    依赖于： golang-gopkg-russross-blackfriday-2-2.1.0-3.fc35.src.rpm

其中golang-gopkg-russross-blackfriday能够顺利安装，但是，重新编译后RPM提供的能力发生了变化(命令dnf provides查看)

• 在Fedora中提供的叫做：golang(github.com/russross/blackfriday/v2)
• 在CentOS-Stream中提供的叫做：golang-symlink(github.com/russross/blackfriday/v2)

所以需要修改golang-github-cpuguy83-md2man.spec文件。

rpmbuild后生成的RPM包有：

├── noarch
│   ├── compat-golang-github-cpuguy83-md2man-2-devel-2.0.1-1.el9.noarch.rpm
│   └── golang-github-cpuguy83-md2man-devel-2.0.1-1.el9.noarch.rpm
└── x86_64
    ├── golang-github-cpuguy83-md2man-2.0.1-1.el9.x86_64.rpm
    ├── golang-github-cpuguy83-md2man-debuginfo-2.0.1-1.el9.x86_64.rpm
    └── golang-github-cpuguy83-md2man-debugsource-2.0.1-1.el9.x86_64.rpm

注意，安装golang-github-cpuguy83-md2man-devel-2.0.1-1.el9.noarch.rpm会提示需要golang(github.com/russross/blackfriday/v2)，直接不安装就行了，转而安装compat-golang-github-cpuguy83-md2man-2-devel-2.0.1-1.el9.noarch.rpm.

然后rpmbuild conmon就可生成：

└── x86_64
    ├── conmon-2.1.0-2.el9.x86_64.rpm
    ├── conmon-debuginfo-2.1.0-2.el9.x86_64.rpm
    └── conmon-debugsource-2.1.0-2.el9.x86_64.rpm

container-selinux

安装标准流程构建即可，生成：

└── noarch
    └── container-selinux-2.180.0-1.el9.noarch.rpm

containerd

containerd依赖的就非常的多了：

No matching package to install: 'golang(github.com/containerd/aufs/plugin)'
No matching package to install: 'golang(github.com/containerd/btrfs)'
No matching package to install: 'golang(github.com/containerd/cgroups)'
No matching package to install: 'golang(github.com/containerd/cgroups/stats/v1)'
No matching package to install: 'golang(github.com/containerd/cgroups/v2)'
No matching package to install: 'golang(github.com/containerd/cgroups/v2/stats)'
No matching package to install: 'golang(github.com/containerd/console)'
No matching package to install: 'golang(github.com/containerd/continuity)'
No matching package to install: 'golang(github.com/containerd/continuity/fs)'
No matching package to install: 'golang(github.com/containerd/continuity/fs/fstest)'
No matching package to install: 'golang(github.com/containerd/continuity/sysx)'
No matching package to install: 'golang(github.com/containerd/continuity/testutil)'
No matching package to install: 'golang(github.com/containerd/continuity/testutil/loopback)'
No matching package to install: 'golang(github.com/containerd/fifo)'
No matching package to install: 'golang(github.com/containerd/go-cni)'
No matching package to install: 'golang(github.com/containerd/go-runc)'
No matching package to install: 'golang(github.com/containerd/imgcrypt)'
No matching package to install: 'golang(github.com/containerd/imgcrypt/images/encryption)'
No matching package to install: 'golang(github.com/containerd/nri)'
No matching package to install: 'golang(github.com/containerd/nri/types/v1)'
No matching package to install: 'golang(github.com/containerd/ttrpc)'
No matching package to install: 'golang(github.com/containerd/ttrpc/plugin)'
No matching package to install: 'golang(github.com/containerd/typeurl)'
No matching package to install: 'golang(github.com/containerd/zfs/plugin)'
No matching package to install: 'golang(github.com/containernetworking/plugins/pkg/ns)'
No matching package to install: 'golang(github.com/coreos/go-systemd/v22/daemon)'
No matching package to install: 'golang(github.com/davecgh/go-spew/spew)'
No matching package to install: 'golang(github.com/docker/go-events)'
No matching package to install: 'golang(github.com/docker/go-metrics)'
No matching package to install: 'golang(github.com/docker/go-units)'
No matching package to install: 'golang(github.com/emicklei/go-restful)'
No matching package to install: 'golang(github.com/fsnotify/fsnotify)'
No matching package to install: 'golang(github.com/gogo/googleapis/google/rpc)'
No matching package to install: 'golang(github.com/gogo/protobuf/gogoproto)'
No matching package to install: 'golang(github.com/gogo/protobuf/proto)'
No matching package to install: 'golang(github.com/gogo/protobuf/protoc-gen-gogo/descriptor)'
No matching package to install: 'golang(github.com/gogo/protobuf/protoc-gen-gogo/generator)'
No matching package to install: 'golang(github.com/gogo/protobuf/sortkeys)'
No matching package to install: 'golang(github.com/gogo/protobuf/types)'
No matching package to install: 'golang(github.com/gogo/protobuf/vanity)'
No matching package to install: 'golang(github.com/gogo/protobuf/vanity/command)'
No matching package to install: 'golang(github.com/google/go-cmp/cmp)'
No matching package to install: 'golang(github.com/google/go-cmp/cmp/cmpopts)'
No matching package to install: 'golang(github.com/google/uuid)'
No matching package to install: 'golang(github.com/grpc-ecosystem/go-grpc-middleware)'
No matching package to install: 'golang(github.com/grpc-ecosystem/go-grpc-prometheus)'
No matching package to install: 'golang(github.com/hashicorp/go-multierror)'
No matching package to install: 'golang(github.com/imdario/mergo)'
No matching package to install: 'golang(github.com/intel/goresctrl/pkg/rdt)'
No matching package to install: 'golang(github.com/klauspost/compress/zstd)'
No matching package to install: 'golang(github.com/moby/locker)'
No matching package to install: 'golang(github.com/moby/sys/mountinfo)'
No matching package to install: 'golang(github.com/moby/sys/signal)'
No matching package to install: 'golang(github.com/moby/sys/symlink)'
No matching package to install: 'golang(github.com/opencontainers/go-digest)'
No matching package to install: 'golang(github.com/opencontainers/go-digest/digestset)'
No matching package to install: 'golang(github.com/opencontainers/image-spec/identity)'
No matching package to install: 'golang(github.com/opencontainers/image-spec/specs-go)'
No matching package to install: 'golang(github.com/opencontainers/image-spec/specs-go/v1)'
No matching package to install: 'golang(github.com/opencontainers/runc/libcontainer/user)'
No matching package to install: 'golang(github.com/opencontainers/runtime-spec/specs-go)'
No matching package to install: 'golang(github.com/opencontainers/selinux/go-selinux)'
No matching package to install: 'golang(github.com/opencontainers/selinux/go-selinux/label)'
No matching package to install: 'golang(github.com/pelletier/go-toml)'
No matching package to install: 'golang(github.com/prometheus/client_golang/prometheus)'
No matching package to install: 'golang(github.com/sirupsen/logrus)'
No matching package to install: 'golang(github.com/stretchr/testify/assert)'
No matching package to install: 'golang(github.com/stretchr/testify/require)'
No matching package to install: 'golang(github.com/tchap/go-patricia/patricia)'
No matching package to install: 'golang(github.com/urfave/cli)'
No matching package to install: 'golang(github.com/vishvananda/netlink)'
No matching package to install: 'golang(go.etcd.io/bbolt)'
No matching package to install: 'golang(go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc)'
No matching package to install: 'golang(go.opentelemetry.io/otel)'
No matching package to install: 'golang(go.opentelemetry.io/otel/attribute)'
No matching package to install: 'golang(go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc)'
No matching package to install: 'golang(go.opentelemetry.io/otel/propagation)'
No matching package to install: 'golang(go.opentelemetry.io/otel/sdk/resource)'
No matching package to install: 'golang(go.opentelemetry.io/otel/sdk/trace)'
No matching package to install: 'golang(go.opentelemetry.io/otel/semconv/v1.4.0)'
No matching package to install: 'golang(go.opentelemetry.io/otel/trace)'
No matching package to install: 'golang(golang.org/x/net/context)'
No matching package to install: 'golang(golang.org/x/net/context/ctxhttp)'
No matching package to install: 'golang(golang.org/x/sync/errgroup)'
No matching package to install: 'golang(golang.org/x/sync/semaphore)'
No matching package to install: 'golang(golang.org/x/sys/execabs)'
No matching package to install: 'golang(golang.org/x/sys/unix)'
No matching package to install: 'golang(google.golang.org/grpc)'
No matching package to install: 'golang(google.golang.org/grpc/backoff)'
No matching package to install: 'golang(google.golang.org/grpc/codes)'
No matching package to install: 'golang(google.golang.org/grpc/credentials)'
No matching package to install: 'golang(google.golang.org/grpc/credentials/insecure)'
No matching package to install: 'golang(google.golang.org/grpc/grpclog)'
No matching package to install: 'golang(google.golang.org/grpc/health)'
No matching package to install: 'golang(google.golang.org/grpc/health/grpc_health_v1)'
No matching package to install: 'golang(google.golang.org/grpc/metadata)'
No matching package to install: 'golang(google.golang.org/grpc/status)'
No matching package to install: 'golang(google.golang.org/protobuf/proto)'
No matching package to install: 'golang(gotest.tools/v3/assert)'
No matching package to install: 'golang(gotest.tools/v3/assert/cmp)'
No matching package to install: 'golang(k8s.io/api/core/v1)'
No matching package to install: 'golang(k8s.io/apimachinery/pkg/api/errors)'
No matching package to install: 'golang(k8s.io/apimachinery/pkg/api/resource)'
No matching package to install: 'golang(k8s.io/apimachinery/pkg/apis/meta/v1)'
No matching package to install: 'golang(k8s.io/apimachinery/pkg/types)'
No matching package to install: 'golang(k8s.io/apimachinery/pkg/util/httpstream)'
No matching package to install: 'golang(k8s.io/apimachinery/pkg/util/httpstream/spdy)'
No matching package to install: 'golang(k8s.io/apimachinery/pkg/util/net)'
No matching package to install: 'golang(k8s.io/apimachinery/pkg/util/remotecommand)'
No matching package to install: 'golang(k8s.io/apimachinery/pkg/util/runtime)'
No matching package to install: 'golang(k8s.io/apimachinery/pkg/util/sets)'
No matching package to install: 'golang(k8s.io/apiserver/pkg/server/httplog)'
No matching package to install: 'golang(k8s.io/apiserver/pkg/util/wsstream)'
No matching package to install: 'golang(k8s.io/client-go/tools/remotecommand)'
No matching package to install: 'golang(k8s.io/client-go/util/cert)'
No matching package to install: 'golang(k8s.io/component-base/logs/logreduction)'
No matching package to install: 'golang(k8s.io/cri-api/pkg/apis/runtime/v1)'
No matching package to install: 'golang(k8s.io/cri-api/pkg/apis/runtime/v1alpha2)'
No matching package to install: 'golang(k8s.io/klog/v2)'
No matching package to install: 'golang(k8s.io/utils/clock)'
No matching package to install: 'golang(k8s.io/utils/clock/testing)'
No matching package to install: 'golang(k8s.io/utils/exec)'

‘golang(github.com/containerd/aufs/plugin)’

需要安装的包为：golang-github-containerd-aufs-1.0.0-2.fc35.src.rpm.

coreos-assembler的适配

锁定kernel

最新的cosa锁定了内核版本，如下：

rpm-ostree version: 2022.5
Previous commit: fa1820378f0e7f70420d6d05c164fd00e2c89006f290fa566171c6ed6ad2bbfe
Enabled rpm-md repositories: fedora-coreos-pool
Updating metadata for 'fedora-coreos-pool'... done
Importing rpm-md... done
rpm-md repo 'fedora-coreos-pool'; generated: 2022-03-16T06:12:50Z solvables: 405
error: Couldn't find locked package 'kernel-5.16.13-200.fc35' (pkgs matching NEVRA: 0; mismatched checksums: 0)
  Packages matching name and arch (1):
  kernel-5.15.13-0.el9.x86_64 (fedora-coreos-pool)

这在文件coreos-assembler/build.sh中有：

 37 install_rpms() {
 38     local builddeps
 39     local frozendeps
 40
 41     # freeze kernel due to https://github.com/coreos/coreos-assembler/issues/2707
 42     frozendeps=$(echo kernel{,-core,-modules}-5.15.18-200.fc35)

锁定systemd

以上两个问题通过修改manifest-lock.overrides.yaml文件即可修改。

cosa build错误

提示的错误为：

Skipping file /usr/etc/grub.d/10_reset_boot_success from checkout
Skipping file /usr/etc/grub.d/12_menu_auto_hide from checkout
Checking out packages... done
error: Checkout catatonit-0.1.7-1.el9.x86_64: Hardlinking f2/63877cc46bb5405bd2b7488104ea7a5461c78857ccc2b3ce0f2ccccbbe5ca2.file to catatonit: File exists
+ rc=1
+ set +x

原因是catatonit和podman-catatonit冲突了。

ISO装机启动服务失败

参见下图

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-6w4ZYF7p-1648375711367)(cclinux-coreos-2203-iso-install-systemd-service-failed.png)]

失败的systemd service有：

• systemd-udevd: /usr/lib/udev/rules.d/50-udev-default.rules:42 Unknown group ‘sgx’, ignoring
• kernel: Driver ‘pcspkr’ is already registered, aborting…
• system-tmpfiles: Failed to open directory ‘.X11-unix’: No such file or directory
• system-tmpfiles: Failed to open directory ‘.ICE-unix’: No such file or directory
• system-tmpfiles: Failed to open directory ‘.XIM-unix’: No such file or directory
• system-tmpfiles: Failed to open directory ‘.font-unix’: No such file or directory
• system-tmpfiles: Failed to open directory ‘.Test-unix’: No such file or directory
• system-resolved.service: Failed to set up mount namespacing: /run/systemd/unit-root/dev: Read-only file system
• system-resolved.service: Failed to step NAMESPACE spwaning /usr/lib/systemd/systemd-resolved: Read-only file system
• dbus-broker.service: Failed to set up mount namespacing: /run/systemd/unit-root/dev: Read-only file system
• dbus-broker.service: Failed to step NAMESPACE spwaning /usr/lib/systemd/dbus-broker-launch: Read-only file system
• systemd: Failed to start Network Name Resolution
• systemd: Failed to start D-Bus System Message Bus
• NetworkManager: bus-manager: cannot connect to D-Bus: Error receiving data: Connection reset by peer
• systemd: Failed to start Network Manager
• systemd: Failed to start D-Bus System Message Bus
• systemd: Failed to listen on D-Bus System Message Bus Socket
• login: pam_systemd(login:session): Faile to connect to system bus: Connect refused
• sudo: pam_systemd(sudo:session): Faile to connect to system bus: Connect refused

dbus-broker.service

dbus-broker.service依赖于dbus.socket.

发现/tmp没有写权限。2022-03-17 17:08
没有挂载，见tmp.mount，删除了systemd中对 tmp.mount注释的补丁并重新编译systemd，见提交 a76940。

coreos-installer的问题

coreos-installer后重启失败

因为重构后的coreos使用coreos-installer安装后，重启无法正常启动，卡住的位置：

systemd[1]: Startup finished in 1.726s (kernel) + 0(initrd) + 2.722s (userspace) = 4.449s

卡住一段时间后会自动再次重启，然后又卡在这个位置。那么我们先查看正常情况下的文件系统。

参考链接

Copyright (C) CESTC Com.