mysql - 使用 openwrt + netmaker 组网时发生非常怪异的 tcp 丢包?

Question

网络拓扑图

阿里云节点环境

• 操作系统：ubuntu 22.04LTS

• 网络配置：

  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000  link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00  inet 127.0.0.1/8 scope host lo     valid_lft forever preferred_lft forever  inet6 ::1/128 scope host     valid_lft forever preferred_lft forever2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000  link/ether 00:16:3e:08:3c:50 brd ff:ff:ff:ff:ff:ff  altname enp0s3  altname ens3  inet 172.19.27.60/20 metric 100 brd 172.19.31.255 scope global dynamic eth0     valid_lft 295876523sec preferred_lft 295876523sec  inet6 fe80::216:3eff:fe08:3c50/64 scope link     valid_lft forever preferred_lft forever38: netmaker: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000  link/none  inet 192.168.10.1/24 brd 192.168.10.255 scope global netmaker     valid_lft forever preferred_lft forever

• 路由信息：

  default via 172.19.31.253 dev eth0 proto dhcp src 172.19.27.60 metric 100100.100.2.136 via 172.19.31.253 dev eth0 proto dhcp src 172.19.27.60 metric 100100.100.2.138 via 172.19.31.253 dev eth0 proto dhcp src 172.19.27.60 metric 100172.19.16.0/20 dev eth0 proto kernel scope link src 172.19.27.60 metric 100172.19.31.253 dev eth0 proto dhcp scope link src 172.19.27.60 metric 100192.168.10.0/24 dev netmaker proto kernel scope link src 192.168.10.1192.168.31.0/24 via 192.168.10.1 dev netmaker

  办公室节点环境

• 操作系统：openwrt 23.05

• 网络配置：

  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000  link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00  inet 127.0.0.1/8 scope host lo     valid_lft forever preferred_lft forever  inet6 ::1/128 scope host     valid_lft forever preferred_lft forever2: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN qlen 1000  link/sit 0.0.0.0 brd 0.0.0.03: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br-lan state UP qlen 1000  link/ether 00:e0:4c:69:1a:62 brd ff:ff:ff:ff:ff:ff4: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000  link/ether 30:9c:23:a3:db:56 brd ff:ff:ff:ff:ff:ff  inet6 fe80::329c:23ff:fea3:db56/64 scope link     valid_lft forever preferred_lft forever24: br-lan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000  link/ether 00:e0:4c:69:1a:62 brd ff:ff:ff:ff:ff:ff  inet 192.168.31.1/24 brd 192.168.31.255 scope global br-lan     valid_lft forever preferred_lft forever  inet6 240e:370:4312:eee0::1/64 scope global dynamic noprefixroute     valid_lft 256217sec preferred_lft 169817sec  inet6 fe80::2e0:4cff:fe69:1a62/64 scope link     valid_lft forever preferred_lft forever25: pppoe-wan: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1492 qdisc fq_codel state UNKNOWN qlen 3  link/ppp  inet 100.64.215.177 peer 100.64.128.1/32 scope global pppoe-wan     valid_lft forever preferred_lft forever  inet6 240e:370:4312:eee1::1/64 scope global dynamic noprefixroute     valid_lft 256217sec preferred_lft 169817sec  inet6 240e:370:4211:c759:997:9f43:7f6:e31/64 scope global dynamic noprefixroute     valid_lft 258851sec preferred_lft 172451sec  inet6 fe80::997:9f43:7f6:e31/128 scope link     valid_lft forever preferred_lft forever26: netmaker: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1420 qdisc fq_codel state UNKNOWN qlen 500  link/[65534]  inet 192.168.10.3/24 brd 192.168.10.255 scope global netmaker     valid_lft forever preferred_lft forever

• 防火墙配置：

  config defaults  option input 'ACCEPT'  option output 'ACCEPT'  option forward 'ACCEPT'  option fullcone '1'  option synflood_protect '1'  option drop_invalid '1'config zone  option name 'lan'  option input 'ACCEPT'  option output 'ACCEPT'  option forward 'ACCEPT'  list network 'lan'config zone  option name 'dmz'  option input 'REJECT'  option output 'ACCEPT'  option forward 'ACCEPT'  option auto_helper '0'  option log '1'  list network 'netmaker'config zone  option name 'wan'  option input 'REJECT'  option output 'ACCEPT'  option forward 'REJECT'  option masq '1'  option mtu_fix '1'  option log '1'  list network 'wan'  list network 'wan6'config rule  option name 'Allow-DHCP-Renew'  option src 'wan'  option proto 'udp'  option dest_port '68'  option target 'ACCEPT'  option family 'ipv4'config rule  option name 'Allow-Ping'  option src 'wan'  option proto 'icmp'  option icmp_type 'echo-request'  option family 'ipv4'  option target 'ACCEPT'config rule  option name 'Allow-IGMP'  option src 'wan'  option proto 'igmp'  option family 'ipv4'  option target 'ACCEPT'config rule  option name 'Allow-DHCPv6'  option src 'wan'  option proto 'udp'  option dest_port '546'  option family 'ipv6'  option target 'ACCEPT'config rule  option name 'Allow-MLD'  option src 'wan'  option proto 'icmp'  option src_ip 'fe80::/10'  list icmp_type '130/0'  list icmp_type '131/0'  list icmp_type '132/0'  list icmp_type '143/0'  option family 'ipv6'  option target 'ACCEPT'config rule  option name 'Allow-ICMPv6-Input'  option src 'wan'  option proto 'icmp'  option family 'ipv6'  option target 'ACCEPT'  list icmp_type 'bad-header'  list icmp_type 'destination-unreachable'  list icmp_type 'echo-reply'  list icmp_type 'echo-request'  list icmp_type 'neighbour-advertisement'  list icmp_type 'neighbour-solicitation'  list icmp_type 'packet-too-big'  list icmp_type 'router-advertisement'  list icmp_type 'router-solicitation'  list icmp_type 'time-exceeded'  list icmp_type 'unknown-header-type'config rule  option name 'Allow-ICMPv6-Forward'  option src 'wan'  option dest '*'  option proto 'icmp'  option family 'ipv6'  option target 'ACCEPT'  list icmp_type 'bad-header'  list icmp_type 'destination-unreachable'  list icmp_type 'echo-reply'  list icmp_type 'echo-request'  list icmp_type 'packet-too-big'  list icmp_type 'time-exceeded'  list icmp_type 'unknown-header-type'config forwarding  option src 'lan'  option dest 'wan'config forwarding  option src 'wan'  option dest 'lan'config forwarding  option src 'dmz'  option dest 'lan'config forwarding  option src 'lan'  option dest 'dmz'

• 路由信息：

  default via 100.64.128.1 dev pppoe-wan100.64.128.1 dev pppoe-wan scope link  src 100.64.215.177172.19.0.0/16 via 192.168.10.3 dev netmaker192.168.10.0/24 dev netmaker scope link  src 192.168.10.3192.168.31.0/24 dev br-lan scope link  src 192.168.31.1

  问题

  在阿里云主机 172.19.227.25 上安装 mysql 服务，端口为 30000。

通过以下几种方式执行 mysql -uroot -p'xxxx' -P30000 -h172.19.227.25 命令都能正
常的连接并返回“Server version: 5.7.43-log MySQL Community Server (GPL)”响应。

但是继续执行 use mysql; 命令时：

主机	结果
192.168.31.1（openwrt主机）	成功
192.168.31.160（局域网的另一台主机）	无响应，命令行 hang 住了
192.168.31.222（安装了 netclient 的主机）	成功

抓包日志

• 192.168.31.160 丢包日志：https://tmp-repo.oss-cn-shanghai.aliyuncs.com/mysql.cap

• 192.168.31.222 正常日志：https://tmp-repo.oss-cn-shanghai.aliyuncs.com/mysql3.cap

  其他

  我也试过在 192.168.31.160 上部署 mysql 服务，能从阿里云的 172.19.227.25 上正常访问。

测试下来只有 yintan 内网主机 -> 192.168.31.1（br-lan） -> 192.168.10.3(netmaker/yintan-gw) -> 192.168.10.1(netmaker/aliyun-gw) -> 172.19.227.25 这条链路会丢包

Answer 1

看上去都没啥问题，openwrt能成功连接，说明 openwrt->阿里云vpc-> mysql 链路是通的。

然后只要保证在openwrt上 192.168.10.x 和 192.168.31.x 之间能有 ip_forward 的的权限。不过从抓包的结果看，31.160已经握手成功，说明中间的链路其实也是通的。

装了netclient的主机能成功，应该是是走的客户端到netmark节点的隧道，没有走openwrt建立的隧道，所以能连接成功。

我感觉可以在openwrt上抓一下netmark接口和lan口，看看包发到openwrt后，有没有正常转发到netmark，对比一下看下是在那个环节丢的包。

Answer 2

我把 br-lan 跟 eth0 的 mtu 一起改为 1420 就可以了。跟 https://www.cnblogs.com/Jcloud/p/16931753.html 很类似，我的 mysql 是安装在k8s里面的，网络插件用的 cilium