Kuberntes 1.24版本部署MetalLB负载均衡器
邴子实
一、MetalLB简介
MetalLB 是为裸机Kubernetes集群实现的负载均衡器，使用标准路由协议ARP或BGP。Metallb 支持两种模式：
- Layer2：Layer2模式必须为裸金属服务器或者支持arp广播的网络，由于云的特殊arp机制，云平台无法支持Layer2模式
- BGP：bgp模式和Calico组网方式无法共存,需采用其它组网方式才能使用BGP模式
官方网站：https://metallb.universe.tf
项目地址：https://github.com/metallb/metallb
二、MetalLB安装
1.部署k8s集群
详见：https://blog.csdn.net/lic95/article/details/125044136
部署完成，详细信息如下：
[root@master01 ~]# kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
master01 Ready control-plane 2d23h v1.24.1 192.168.3.31 <none> AlmaLinux 8.6 (Sky Tiger) 4.18.0-372.9.1.el8.x86_64 containerd://1.6.4
master02 Ready control-plane 2d22h v1.24.1 192.168.3.32 <none> AlmaLinux 8.6 (Sky Tiger) 4.18.0-372.9.1.el8.x86_64 containerd://1.6.4
master03 Ready control-plane 2d22h v1.24.1 192.168.3.33 <none> AlmaLinux 8.6 (Sky Tiger) 4.18.0-372.9.1.el8.x86_64 containerd://1.6.4
node01 Ready <none> 2d22h v1.24.1 192.168.3.41 <none> AlmaLinux 8.6 (Sky Tiger) 4.18.0-372.9.1.el8.x86_64 containerd://1.6.4
node02 Ready <none> 2d22h v1.24.1 192.168.3.42 <none> AlmaLinux 8.6 (Sky Tiger) 4.18.0-372.9.1.el8.x86_64 containerd://1.6.4
node03 Ready <none> 2d22h v1.24.1 192.168.3.43 <none> AlmaLinux 8.6 (Sky Tiger) 4.18.0-372.9.1.el8.x86_64 containerd://1.6.4
node04 Ready <none> 2d22h v1.24.1 192.168.3.44 <none> AlmaLinux 8.6 (Sky Tiger) 4.18.0-372.9.1.el8.x86_64 containerd://1.6.4
node05 Ready <none> 2d22h v1.24.1 192.168.3.45 <none> AlmaLinux 8.6 (Sky Tiger) 4.18.0-372.9.1.el8.x86_64 containerd://1.6.4
2.部署测试应用echoserver
# 部署echoserver
kubectl create deploy echoserver --image=cilium/echoserver
# 查看部署情况
[root@master01 manifests]# kubectl get pods
NAME READY STATUS RESTARTS AGE
echoserver-8585bfb456-v6cx8 1/1 Running 0 8m42s
#修改为LoadBalancer
kubectl expose deployment echoserver --port=80 --type=LoadBalancer
#查看services,因MetalLB未部署，状态未pending
[root@master01 ~]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
echoserver LoadBalancer 172.18.149.83 <pending> 8080:30926/TCP 2m
kubernetes ClusterIP 172.18.0.1 <none> 443/TCP 36m
3.修改kube-proxy，启用严格ARP模式
# 修改mode和strictARP值,并校对
kubectl get configmap kube-proxy -n kube-system -o yaml | \
sed -e "s/strictARP: false/strictARP: true/" | \
kubectl diff -f - -n kube-system
# 执行修改操作
kubectl get configmap kube-proxy -n kube-system -o yaml | \
sed -e "s/strictARP: false/strictARP: true/" | \
kubectl apply -f - -n kube-system
#查看修改结果
kubectl edit configmap -n kube-system kube-proxy
4.安装Metallb
#Metallb 支持 yaml文件、Kustomize、Helm 和 MetalLB Operator多种安装方法，这里使用yaml方式进行安装。
# 下载release版本，并解压进入manifests目录
wget https://github.com/metallb/metallb/archive/refs/tags/v0.12.1.tar.gz
tar vzxf v0.12.1.tar.gz
cd metallb-0.12.1/manifests/
# 执行yaml文件进行安装
kubectl apply -f namespace.yaml
kubectl apply -f metallb.yaml
# 查看运行状态
[root@master01 manifests]# kubectl -n metallb-system get pods
NAME READY STATUS RESTARTS AGE
controller-7476b58756-mkxcn 1/1 Running 0 4m12s
speaker-79r6v 1/1 Running 0 4m12s
speaker-dwlnf 1/1 Running 0 4m12s
speaker-ppgmj 1/1 Running 0 4m12s
speaker-zgnjz 1/1 Running 0 4m12s
speaker-zp6fm 1/1 Running 0 4m12s
[root@master01 manifests]# kubectl -n metallb-system get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
controller 1/1 1 1 5m19s
[root@master01 manifests]# kubectl -n metallb-system get ds
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
speaker 5 5 5 5 5 kubernetes.io/os=linux 6m7s
三、配置Layer2模式
注意：Layer2模式必须为裸金属服务器或者支持arp广播的网络，由于云的特殊arp机制，云平台无法支持Layer2模式
1、修改默认配置文件，为MetalLB分配部分IP地址(必须同一网段)
[root@master01 manifests]# cat example-layer2-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
address-pools:
- name: my-ip-space
protocol: layer2
addresses:
- 192.168.3.80-192.168.3.90 #必须同一网段
2.执行yaml文件
[root@master01 manifests]# kubectl apply -f example-layer2-config.yaml
configmap/config configured
3.查看echoserver LoadBalancer地址
[root@master01 manifests]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
echoserver LoadBalancer 172.18.149.83 192.168.3.80 80:32563/TCP 26m
kubernetes ClusterIP 172.18.0.1 <none> 443/TCP 60m
4.查看测试结果,OK
[root@master01 manifests]# curl http://192.168.3.80
Hostname: echoserver-8585bfb456-v6cx8
Pod Information:
-no pod information available-
Server values:
server_version=nginx: 1.13.3 - lua: 10008
Request Information:
client_address=::ffff:10.244.241.64
method=GET
real path=/
query=
request_version=1.1
request_scheme=http
request_uri=http://192.168.3.80:80/
Request Headers:
accept=*/*
host=192.168.3.80
user-agent=curl/7.61.1
Request Body:
-no body in request-
5.关于nodeport
在创建LoadBalancer服务的时候，默认情况下k8s会帮我们自动创建一个nodeport服务，这个操作可以通过指定Service中的allocateLoadBalancerNodePorts字段来定义开关，默认情况下为true，需要关闭nodeport的话可以修改service中的spec.allocateLoadBalancerNodePorts字段，将其设置为false，那么在创建svc的时候就不会分配nodeport。
三、配置BGP模式
BGP模式需要和外部路由器进行通信才能正常工作，为了方便，我们采用frr搭建外部路由器和k8s集群通信，路由器地址为192.168.3.254，搭建方式为容器
bgp模式和Calico组网方式无法共存,需采用其它组网方式才能使用BGP模式
1.搭建集群时修改网络模式为flannel模式
#集群的时候采用flannel网络组网
kubectl delete -f https://docs.projectcalico.org/manifests/calico.yaml
kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml
2.在外部机器上安装frr容器
# 默认配置启动frr容器
[root@AlmaLinux-BGP ~]# docker run -d --name AlmaLinux-BGP frrouting/frr
4f0097d103d78dbfbb728771885fec4c46d50cbe64a3cd44bb7ba58cfd35021a
# 提取镜像默认配置到/etc/frr，持久化存储配置做准备
mkdir -p /etc/frr
docker cp AlmaLinux-BGP:/etc/frr/daemons /etc/frr/daemons
# 修改配置文件，启动BGP
sed -i 's#bgpd=no#bgpd=yes#g' /etc/frr/daemons
# 关闭默认配置frr容器
docker stop AlmaLinux-BGP && docker rm AlmaLinux-BGP
# 持久化存储启动frr容器
docker run -d --restart=always \
--name AlmaLinux-BGP \
-v /etc/frr:/etc/frr:Z \
--net=host \
--privileged \
frrouting/frr
3.通过命令docker exec -it AlmaLinux-BGP vtysh进入frr容器，配置BGP
# 配置如下
frr# show running-config
Building configuration...
Current configuration:
!
frr version 8.1_git
frr defaults traditional
hostname git-nas
no ipv6 forwarding
hostname frr
!
router bgp 65000
bgp router-id 192.168.3.254
no bgp ebgp-requires-policy
neighbor k8s peer-group
neighbor k8s remote-as 65001
neighbor k8s password metallb
neighbor 192.168.3.41 peer-group k8s
neighbor 192.168.3.41 password metallb
neighbor 192.168.3.42 peer-group k8s
neighbor 192.168.3.42 password metallb
neighbor 192.168.3.43 peer-group k8s
neighbor 192.168.3.43 password metallb
neighbor 192.168.3.44 peer-group k8s
neighbor 192.168.3.44 password metallb
neighbor 192.168.3.45 peer-group k8s
neighbor 192.168.3.45 password metallb
exit
!
end
frr#
4.安装Metallb及测试pod
# 部署echoserver
kubectl create deploy echoserver --image=cilium/echoserver
#修改为LoadBalancer
kubectl expose deployment echoserver --port=80 --type=LoadBalancer
# 下载release版本，并解压进入manifests目录
wget https://github.com/metallb/metallb/archive/refs/tags/v0.12.1.tar.gz
tar vzxf v0.12.1.tar.gz
cd metallb-0.12.1/manifests/
# 执行yaml文件进行安装
kubectl apply -f namespace.yaml
kubectl apply -f metallb.yaml
# 查看pod状态
[root@master01 ~]# kubectl get pods -n metallb-system
NAME READY STATUS RESTARTS AGE
controller-7476b58756-zjlnh 1/1 Running 0 10m
speaker-4qbh6 1/1 Running 0 10m
speaker-9bwv8 1/1 Running 0 10m
speaker-b4vcb 1/1 Running 0 10m
speaker-hp9qk 1/1 Running 0 10m
speaker-wdb78 1/1 Running 0 10m
5.配置BPG模式configmap
[root@master01 manifests]# cat example-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
# The peers section tells MetalLB what BGP routers to connect too. There
# is one entry for each router you want to peer with.
peers:
- peer-address: 192.168.3.254
peer-asn: 65000
my-asn: 65001
password: "metallb"
address-pools:
- name: default
protocol: bgp
addresses:
- 192.168.0.100-192.168.0.200
6.执行操作
[root@master01 manifests]# kubectl apply -f example-config.yaml
configmap/config configured
7.修改配置后删除Metallb 删除控制器原来的pod,k8s自动重启新pod
[root@master01 manifests]# kubectl delete pod -n metallb-system -l app=metallb,component=controller
8.查看测试pod,分配到了新规划的IP
[root@master01 ~]# kubectl get svc -n default
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
echoserver LoadBalancer 172.18.3.41 192.168.0.100 80:30600/TCP 12m
kubernetes ClusterIP 172.18.0.1 <none> 443/TCP 22m
9.查看frr状态
# 查看frr BGP状态，已经和node1到node5建立了BGP邻居关系
frr# show ip bgp peer-group
BGP peer-group k8s, remote AS 65001
Peer-group type is external
Configured address-families: IPv4 Unicast;
Peer-group members:
192.168.3.41 Established
192.168.3.42 Established
192.168.3.43 Established
192.168.3.44 Established
192.168.3.45 Established
frr#
#查看路由状态
frr# show ip route bgp
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
B>* 192.168.0.100/32 [20/0] via 192.168.3.41, ens3, weight 1, 00:08:52
* via 192.168.3.42, ens3, weight 1, 00:08:52
* via 192.168.3.43, ens3, weight 1, 00:08:52
* via 192.168.3.44, ens3, weight 1, 00:08:52
* via 192.168.3.45, ens3, weight 1, 00:08:52
frr#
#查看路由表
frr# show ip bgp
BGP table version is 3, local router ID is 192.168.3.40, vrf id 0
Default local pref 100, local AS 65000
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*= 192.168.0.100/32 192.168.3.43 0 65001 ?
*= 192.168.3.45 0 65001 ?
*= 192.168.3.42 0 65001 ?
*> 192.168.3.41 0 65001 ?
*= 192.168.3.44 0 65001 ?
Displayed 1 routes and 5 total paths
frr#
8.查看测试结果，OK
root@git-nas:~# curl http://192.168.0.100
Hostname: echoserver-8585bfb456-8vn9r
Pod Information:
-no pod information available-
Server values:
server_version=nginx: 1.13.3 - lua: 10008
Request Information:
client_address=::ffff:10.244.5.0
method=GET
real path=/
query=
request_version=1.1
request_scheme=http
request_uri=http://192.168.0.100:80/
Request Headers:
accept=*/*
host=192.168.0.100
user-agent=curl/7.81.0
类似资料：