KubeOVN 支持 KubeVirt 网卡热插实践

谷梁楷

2023-12-01

本文描述基于 KubeOVN + KubeVirt 实现网卡热插的实践方法。
基于 containerd。

版本

kubeovn 基于 feature 提交
kubevirt 基于 commit 3bee1de7b4f8bab059817cf44d636d2521dec8a9

部署

kubeovn + kubevirt

kubeovn 基于社区 install.sh 脚本部署，然后 kube-ovn-controller 镜像替换
kubevirt 可以使用镜像 quay.io/kubevirt/virt-operator:20230320_3bee1de7b

kubevirt 需要开启 feature-gate

---
apiVersion: kubevirt.io/v1
kind: KubeVirt
metadata:
  name: kubevirt
  namespace: kubevirt
spec:
  configuration:
    developerConfiguration: 
      featureGates:
        - HotplugNICs

dynamic-networks-controller

git clone https://github.com/k8snetworkplumbingwg/multus-dynamic-networks-controller.git
cd multus-dynamic-networks-controller
kubectl apply -f manifests/dynamic-networks-controller.yaml

multus

git clone https://github.com/k8snetworkplumbingwg/multus-cni.git && cd multus-cni
cat ./deployments/multus-daemonset-thick.yml | kubectl apply -f -

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: net1
  namespace: default
spec:
  config: '{
    "cniVersion": "0.3.1",
    "name": "kube-ovn",
    "plugins":[
        {
            "type":"kube-ovn",
            "server_socket":"/run/openvswitch/kube-ovn-daemon.sock",
            "provider": "net1.default.ovn"
        },
        {
            "type":"portmap",
            "capabilities":{
                "portMappings":true
            }
        }
    ]
}'

使用

创建虚拟机

apiVersion: kubevirt.io/v1
kind: VirtualMachineInstance
metadata:
  name: testvmi-nocloud
spec:
  terminationGracePeriodSeconds: 30
  domain:
    resources:
      requests:
        memory: 1024M
    devices:
      disks:
      - name: containerdisk
        disk:
          bus: virtio
      - name: emptydisk
        disk:
          bus: virtio
      - disk:
          bus: virtio
        name: cloudinitdisk
  volumes:
  - name: containerdisk
    containerDisk:
      image: kubevirt/fedora-cloud-container-disk-demo:latest
  - name: emptydisk
    emptyDisk:
      capacity: "2Gi"
  - name: cloudinitdisk
    cloudInitNoCloud:
      userData: |-
        #cloud-config
        password: fedora
        chpasswd: { expire: False }

检查 pod

$ kubectl get pods -owide
virt-launcher-testvmi-nocloud-sl55d   2/2     Running   0          61s   10.16.0.49   node113   <none>           1/1

$ kubectl exec -it virt-launcher-testvmi-nocloud-sl55d ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default 
    link/ether 66:6d:26:df:ab:7c brd ff:ff:ff:ff:ff:ff
    inet 10.16.0.49/16 brd 10.16.255.255 scope global eth0
       valid_lft forever preferred_lft forever
3: k6t-eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default 
    link/ether 3e:4c:d9:14:6f:1f brd ff:ff:ff:ff:ff:ff
    inet 169.254.75.10/32 scope global k6t-eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::3c4c:d9ff:fe14:6f1f/64 scope link 
       valid_lft forever preferred_lft forever
4: tap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc fq_codel master k6t-eth0 state UP group default qlen 1000
    link/ether fa:5a:68:6c:06:50 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::f85a:68ff:fe6c:650/64 scope link 
       valid_lft forever preferred_lft forever
41: eth0-nic@if42: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master k6t-eth0 state UP group default 
    link/ether 3e:4c:d9:14:6f:1f brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::3c4c:d9ff:fe14:6f1f/64 scope link 
       valid_lft forever preferred_lft forever

检查虚拟机

$ virtctl console testvmi-nocloud
$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:00:00:3b:e5:7b brd ff:ff:ff:ff:ff:ff
    altname enp1s0
    inet 10.16.0.49/16 brd 10.16.255.255 scope global dynamic noprefixroute eth0
       valid_lft 86313549sec preferred_lft 86313549sec
    inet6 fe80::200:ff:fe3b:e57b/64 scope link 
       valid_lft forever preferred_lft forever

KubeOVN 添加第二张网卡

添加第二张网卡信息，选择上面的 network-attachment-definition 和 subnet

$ kubectl edit pod  virt-launcher-testvmi-nocloud-sl55d
......
k8s.v1.cni.cncf.io/networks: '[{"interface":"net1","name":"net1","namespace":"default"}]'
net1.default.ovn.kubernetes.io/logical_switch: subnet-ipv4
......

检查，新增 net1 192.168.2.15/24

$ kubectl exec -it virt-launcher-testvmi-nocloud-sl55d ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default 
    link/ether 66:6d:26:df:ab:7c brd ff:ff:ff:ff:ff:ff
    inet 10.16.0.49/16 brd 10.16.255.255 scope global eth0
       valid_lft forever preferred_lft forever
3: k6t-eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default 
    link/ether 3e:4c:d9:14:6f:1f brd ff:ff:ff:ff:ff:ff
    inet 169.254.75.10/32 scope global k6t-eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::3c4c:d9ff:fe14:6f1f/64 scope link 
       valid_lft forever preferred_lft forever
4: tap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc fq_codel master k6t-eth0 state UP group default qlen 1000
    link/ether fa:5a:68:6c:06:50 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::f85a:68ff:fe6c:650/64 scope link 
       valid_lft forever preferred_lft forever
41: eth0-nic@if42: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master k6t-eth0 state UP group default 
    link/ether 3e:4c:d9:14:6f:1f brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::3c4c:d9ff:fe14:6f1f/64 scope link 
       valid_lft forever preferred_lft forever
43: net1@if44: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default 
    link/ether 00:00:00:d6:bb:bd brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.2.15/24 brd 192.168.2.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::200:ff:fed6:bbbd/64 scope link 
       valid_lft forever preferred_lft forever

virtctl 添加 interface

添加到虚拟机内，命名 eth1

$ virtctl addinterface testvmi-nocloud   --network-name default/net1 --iface-name eth1

检查 pod

$ kubectl exec -it virt-launcher-testvmi-nocloud-sl55d ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default 
    link/ether 66:6d:26:df:ab:7c brd ff:ff:ff:ff:ff:ff
    inet 10.16.0.49/16 brd 10.16.255.255 scope global eth0
       valid_lft forever preferred_lft forever
3: k6t-eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default 
    link/ether 3e:4c:d9:14:6f:1f brd ff:ff:ff:ff:ff:ff
    inet 169.254.75.10/32 scope global k6t-eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::3c4c:d9ff:fe14:6f1f/64 scope link 
       valid_lft forever preferred_lft forever
4: tap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc fq_codel master k6t-eth0 state UP group default qlen 1000
    link/ether fa:5a:68:6c:06:50 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::f85a:68ff:fe6c:650/64 scope link 
       valid_lft forever preferred_lft forever
5: net1: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default 
    link/ether d6:d6:72:f1:12:5c brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.15/24 brd 192.168.2.255 scope global net1
       valid_lft forever preferred_lft forever
6: k6t-net1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default 
    link/ether 06:73:79:b3:c0:6e brd ff:ff:ff:ff:ff:ff
    inet 169.254.75.11/32 scope global k6t-net1
       valid_lft forever preferred_lft forever
    inet6 fe80::473:79ff:feb3:c06e/64 scope link 
       valid_lft forever preferred_lft forever
7: tap1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc fq_codel master k6t-net1 state UP group default qlen 1000
    link/ether a2:7b:a6:7c:5a:81 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a07b:a6ff:fe7c:5a81/64 scope link 
       valid_lft forever preferred_lft forever
41: eth0-nic@if42: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master k6t-eth0 state UP group default 
    link/ether 3e:4c:d9:14:6f:1f brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::3c4c:d9ff:fe14:6f1f/64 scope link 
       valid_lft forever preferred_lft forever
43: net1-nic@if44: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue master k6t-net1 state UP group default 
    link/ether 06:73:79:b3:c0:6e brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::473:79ff:feb3:c06e/64 scope link 
       valid_lft forever preferred_lft forever

检查虚拟机，新增网卡

[fedora@testvmi-nocloud ~]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:00:00:3b:e5:7b brd ff:ff:ff:ff:ff:ff
    altname enp1s0
    inet 10.16.0.49/16 brd 10.16.255.255 scope global dynamic noprefixroute eth0
       valid_lft 86312937sec preferred_lft 86312937sec
    inet6 fe80::200:ff:fe3b:e57b/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:00:00:d6:bb:bd brd ff:ff:ff:ff:ff:ff
    altname enp8s0
    inet 192.168.2.15/24 brd 192.168.2.255 scope global dynamic noprefixroute eth1
       valid_lft 86313539sec preferred_lft 86313539sec
    inet6 fe80::af9d:e259:1171:8b9/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
       
[fedora@testvmi-nocloud ~]$ ping 192.168.2.1
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=254 time=2.88 ms
64 bytes from 192.168.2.1: icmp_seq=2 ttl=254 time=1.34 ms

检查 libvirt xml，新增

......
    <interface type='ethernet'>
      <mac address='00:00:00:d6:bb:bd'/>
      <target dev='tap1' managed='no'/>
      <model type='virtio-non-transitional'/>
      <mtu size='1400'/>
      <alias name='ua-eth1'/>
      <rom enabled='no'/>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
    </interface>

......

检查 vmi，新增

......
        interfaces:
		......
        - bridge: {}
          name: eth1
    networks:
	......
    - multus:
        networkName: default/net1
      name: eth1
......

feature

热拔目前未实现，待 kubevirt 开发。

KubeOVN 支持 KubeVirt 网卡热插实践

版本

部署

kubeovn + kubevirt

dynamic-networks-controller

multus

使用

创建虚拟机

KubeOVN 添加第二张网卡

virtctl 添加 interface

feature

相关阅读

相关文章

相关问答

相关文档