mysql - MGR 节点之间通信失败,你们遇到过吗?

Question

有一套INNODB CLUSTER 基于社区版MYSQL8.0.33 搭建 在辅助节点上备份
使用xtrabackup8.0.33每周进行全备，每天两点进行差异备份。
但是在日志中每天进行差异备份的时候节点总是报一些警告。

2024-07-21T00:00:03.026463+08:00 15730415 [Warning] [MY-013712] [Server] No suitable 'keyring_component_metadata_query' service implementation found to fulfill the request.
2024-07-21T02:00:03.680055+08:00 15730430 [Warning] [MY-013712] [Server] No suitable 'keyring_component_metadata_query' service implementation found to fulfill the request.
2024-07-21T04:00:02.249723+08:00 15730432 [Warning] [MY-013712] [Server] No suitable 'keyring_component_metadata_query' service implementation found to fulfill the request.
2024-07-21T06:00:02.597706+08:00 15730434 [Warning] [MY-013712] [Server] No suitable 'keyring_component_metadata_query' service implementation found to fulfill the request.
2024-07-21T08:00:02.621314+08:00 15730436 [Warning] [MY-013712] [Server] No suitable 'keyring_component_metadata_query' service implementation found to fulfill the request.
2024-07-21T08:00:10.510218+08:00 0 [Warning] [MY-011493] [Repl] Plugin group_replication reported: 'Member with address 10.27.13.42:3306 has become unreachable.'
2024-07-21T08:00:10.514438+08:00 0 [Warning] [MY-011493] [Repl] Plugin group_replication reported: 'Member with address 10.27.13.43:3306 has become unreachable.'
2024-07-21T08:00:10.514460+08:00 0 [ERROR] [MY-011495] [Repl] Plugin group_replication reported: 'This server is not able to reach a majority of members in the group. This server will now block all updates. The server will remain blocked until contact with the majority is restored. It is possible to use group_replication_force_members to force a new group membership.'
2024-07-21T08:00:10.518201+08:00 0 [Warning] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] Shutting down an outgoing connection. This happens because something might be wrong on a bi-directional connection to node 10.27.13.43:33061. Please check the connection status to this member'
2024-07-21T08:00:10.521480+08:00 0 [Warning] [MY-011494] [Repl] Plugin group_replication reported: 'Member with address 10.27.13.43:3306 is reachable again.'
2024-07-21T08:00:10.521515+08:00 0 [Warning] [MY-011498] [Repl] Plugin group_replication reported: 'The member has resumed contact with a majority of the members in the group. Regular operation is restored and transactions are unblocked.'
2024-07-21T08:00:10.798208+08:00 0 [Warning] [MY-011494] [Repl] Plugin group_replication reported: 'Member with address 10.27.13.42:3306 is reachable again.'
2024-07-21T10:00:02.631572+08:00 15730438 [Warning] [MY-013712] [Server] No suitable 'keyring_component_metadata_query' service implementation found to fulfill the request.
2024-07-21T12:00:02.274789+08:00 15730440 [Warning] [MY-013712] [Server] No suitable 'keyring_component_metadata_query' service implementation found to fulfill the request.
2024-07-21T14:00:02.009040+08:00 15730442 [Warning] [MY-013712] [Server] No suitable 'keyring_component_metadata_query' service implementation found to fulfill the request.
2024-07-21T14:00:09.272643+08:00 0 [Warning] [MY-011493] [Repl] Plugin group_replication reported: 'Member with address 10.27.13.42:3306 has become unreachable.'
2024-07-21T14:00:09.281750+08:00 0 [Warning] [MY-011493] [Repl] Plugin group_replication reported: 'Member with address 10.27.13.43:3306 has become unreachable.'
2024-07-21T14:00:09.281772+08:00 0 [ERROR] [MY-011495] [Repl] Plugin group_replication reported: 'This server is not able to reach a majority of members in the group. This server will now block all updates. The server will remain blocked until contact with the majority is restored. It is possible to use group_replication_force_members to force a new group membership.'
2024-07-21T14:00:09.288686+08:00 0 [Warning] [MY-011494] [Repl] Plugin group_replication reported: 'Member with address 10.27.13.43:3306 is reachable again.'
2024-07-21T14:00:09.288735+08:00 0 [Warning] [MY-011498] [Repl] Plugin group_replication reported: 'The member has resumed contact with a majority of the members in the group. Regular operation is restored and transactions are unblocked.'
2024-07-21T14:00:09.900326+08:00 0 [Warning] [MY-011494] [Repl] Plugin group_replication reported: 'Member with address 10.27.13.42:3306 is reachable again.'
2024-07-21T16:00:03.018447+08:00 15730444 [Warning] [MY-013712] [Server] No suitable 'keyring_component_metadata_query' service implementation found to fulfill the request.
2024-07-21T18:00:02.425924+08:00 15730446 [Warning] [MY-013712] [Server] No suitable 'keyring_component_metadata_query' service implementation found to fulfill the request.
2024-07-21T20:00:01.697066+08:00 15730448 [Warning] [MY-013712] [Server] No suitable 'keyring_component_metadata_query' service implementation found to fulfill the request.
2024-07-21T20:00:09.949877+08:00 0 [Warning] [MY-011493] [Repl] Plugin group_replication reported: 'Member with address 10.27.13.42:3306 has become unreachable.'
2024-07-21T20:00:09.961070+08:00 0 [Warning] [MY-011493] [Repl] Plugin group_replication reported: 'Member with address 10.27.13.43:3306 has become unreachable.'
2024-07-21T20:00:09.966909+08:00 0 [ERROR] [MY-011495] [Repl] Plugin group_replication reported: 'This server is not able to reach a majority of members in the group. This server will now block all updates. The server will remain blocked until contact with the majority is restored. It is possible to use group_replication_force_members to force a new group membership.'
2024-07-21T20:00:09.981279+08:00 0 [Warning] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] Shutting down an outgoing connection. This happens because something might be wrong on a bi-directional connection to node 10.27.13.43:33061. Please check the connection status to this member'
2024-07-21T20:00:10.026796+08:00 0 [Warning] [MY-011494] [Repl] Plugin group_replication reported: 'Member with address 10.27.13.43:3306 is reachable again.'
2024-07-21T20:00:10.026875+08:00 0 [Warning] [MY-011498] [Repl] Plugin group_replication reported: 'The member has resumed contact with a majority of the members in the group. Regular operation is restored and transactions are unblocked.'
2024-07-21T20:00:10.442596+08:00 0 [Warning] [MY-011494] [Repl] Plugin group_replication reported: 'Member with address 10.27.13.42:3306 is reachable again.'
2024-07-21T22:00:01.852544+08:00 15730450 [Warning] [MY-013712] [Server] No suitable 'keyring_component_metadata_query' service implementation found to fulfill the request.

查看过 网卡日志发现是存在掉包情况

Answer 1

关于 keyring_component_metadata_query 警告

问题解释:
警告 No suitable 'keyring_component_metadata_query' service implementation found to fulfill the request 通常与 MySQL 的密钥管理服务相关。在 MySQL 8.0 中，密钥管理是一个可选功能，用于加密敏感数据（如密码、密钥等）。如果配置了密钥管理服务但相关组件未正确启动或配置，就会出现此类警告。

解决方案:

1. 检查配置：确认 MySQL 配置文件（如 my.cnf 或 my.ini）中是否启用了密钥管理服务，并检查相关组件（如 keyring_file 或 keyring_encrypted_file）的配置。
2. 禁用密钥管理：如果不需要密钥管理服务，可以在配置文件中移除或注释掉相关设置。
3. 确保密钥管理组件正确安装和配置：如果需要使用密钥管理服务，确保所有必要的组件都已正确安装、配置和启动。

关于 MGR 节点通信问题

问题解释:
日志中的 MGR（MySQL Group Replication）相关警告和错误表明集群中的节点之间存在通信问题。特别是 Member with address ... has become unreachable 和 This server is not able to reach a majority of members in the group 错误表明节点之间的网络连接不稳定或存在故障。

解决方案:

1. 检查网络连接：由于已经确认网卡存在掉包情况，这是导致问题的主要原因。应首先解决网络问题，包括检查网络硬件（如交换机、路由器）、网络配置（如 IP 地址、子网掩码、网关设置）以及网络线缆等。
2. 监控网络状态：使用网络监控工具（如 ping、traceroute、tcpdump 等）持续监控节点之间的网络状态，确保网络连接稳定。
3. 调整 MGR 配置：考虑调整 MGR 的相关配置，如 group_replication_local_address、group_replication_group_seeds 等，以确保节点能够正确识别并连接到其他成员。
4. 升级和修复：确保 MySQL 和所有相关组件（如网络驱动、操作系统等）都已更新到最新版本，并修复所有已知的网络和性能问题。
5. 硬件检查：如果问题持续存在，可能需要检查服务器的硬件（如网卡、CPU、内存等）是否存在故障或性能瓶颈。

总结

解决这类问题通常需要综合考虑网络、配置、软件和硬件等多个方面。首先解决网络问题，然后检查并调整相关配置，最后确保所有组件都处于良好状态。