Redis trouble15 -- unable to failover:check ‘cluster-replica-validity-factor‘ configuration option
一、问题描述
标题太短了放不下,完整的异常提示为:
Currently unable to failover: Disconnected from master for longer than allowed. Please check the 'cluster-replica-validity-factor' configuration option.
原为三主三从集群,同时宕掉一对主从,此时集群状态fail,启动从节点,会发现从节点并没有如我们所想那样成为主节点,集群状态依旧fail,这个是个正常的现象并没有解决方案,这篇文章是为了分析在什么情况下会出现主从不能切换的问题
1:S 18 Aug 2021 17:00:07.622 # Error condition on socket for SYNC: Connection refused
1:S 18 Aug 2021 17:00:08.626 * Connecting to MASTER 127.0.0.1:8518
1:S 18 Aug 2021 17:00:08.626 * MASTER <-> REPLICA sync started
1:S 18 Aug 2021 17:00:08.626 # Error condition on socket for SYNC: Connection refused
1:S 18 Aug 2021 17:00:09.528 # Currently unable to failover: Disconnected from master for longer than allowed. Please check the 'cluster-replica-validity-factor' configuration option.
1:S 18 Aug 2021 17:00:09.628 * Connecting to MASTER 127.0.0.1:8518
1:S 18 Aug 2021 17:00:09.628 * MASTER <-> REPLICA sync started
1:S 18 Aug 2021 17:00:09.628 # Error condition on socket for SYNC: Connection refused
1:S 18 Aug 2021 17:00:10.631 * Connecting to MASTER 127.0.0.1:8518
1:S 18 Aug 2021 17:00:10.631 * MASTER <-> REPLICA sync started
1:S 18 Aug 2021 17:00:10.631 # Error condition on socket for SYNC: Connection refused
1:S 18 Aug 2021 17:00:11.636 * Connecting to MASTER 127.0.0.1:8518
1:S 18 Aug 2021 17:00:11.636 * MASTER <-> REPLICA sync started
1:S 18 Aug 2021 17:00:11.636 # Error condition on socket for SYNC: Connection refused
1:S 18 Aug 2021 17:00:12.640 * Connecting to MASTER 127.0.0.1:8518
二、什么情况下主从会不切换
通过以下的源码分析,主从断开达到一定的时间(计算时长主从断开超过160s)时,会存在从不能切主的情况,需要手动failover才可以切换
slave和master的复制连接断开时间不超过给定的值（值可配置，目的是确保slave上的数据足够完整，所以运维时不能任由一个slave长时间不可用，需要通过监控将异常的slave及时恢复）。
计算failover的超时时间
超时时间为 max(cluster_node_timeout*2,2000),
集群默认超时时间为15000*2,所以默认的 auth_timeout 时间为30s 默认 auth_retry_time 时间为60s
/* Compute the failover timeout (the max time we have to send votes
* and wait for replies), and the failover retry time (the time to wait
* before trying to get voted again).
*
* Timeout is MAX(NODE_TIMEOUT*2,2000) milliseconds.
* Retry is two times the Timeout.
*/
auth_timeout = server.cluster_node_timeout*2;
if (auth_timeout < 2000) auth_timeout = 2000;
auth_retry_time = auth_timeout*2;
'以下情况不能执行failover'
1).当前节点不是slave节点
2).主节点被标记为fail,或者是手动故障切换
3).没有设置failover的配置(cluster-slave-no-failover yes) 并且不是手动故障切换
4).上边有slots使用
/* Pre conditions to run the function, that must be met both in case
* of an automatic or manual failover:
* 1) We are a slave.
* 2) Our master is flagged as FAIL, or this is a manual failover.
* 3) We don't have the no failover configuration set, and this is
* not a manual failover.
* 4) It is serving slots. */
if (nodeIsMaster(myself) ||
myself->slaveof == NULL ||
(!nodeFailed(myself->slaveof) && !manual_failover) ||
(server.cluster_slave_no_failover && !manual_failover) ||
myself->slaveof->numslots == 0)
{
/* There are no reasons to failover, so we set the reason why we
* are returning without failing over to NONE. */
server.cluster->cant_failover_reason = CLUSTER_CANT_FAILOVER_NONE;
return;
}// 主节点断开连接的秒数
/* Set data_age to the number of seconds we are disconnected from
* the master. */
if (server.repl_state == REPL_STATE_CONNECTED) {
data_age = (mstime_t)(server.unixtime - server.master->lastinteraction)
* 1000;
} else {
data_age = (mstime_t)(server.unixtime - server.repl_down_since) * 1000;
}
//data_age中要除去集群超时时间
/* Remove the node timeout from the data age as it is fine that we are
* disconnected from our master at least for the time it was down to be
* flagged as FAIL, that's the baseline. */
if (data_age > server.cluster_node_timeout)
data_age -= server.cluster_node_timeout;根据用户配置的cluster_slave_validity_factor检查我们的数据是否足够新,并检查是否有手动failover'
'默认参数配置:'
cluster-node-timeout 15000
cluster-replica-validity-factor 10
repl-ping-replica-period 10
repl-timeout 60
'同时满足以下条件不能主从切换'
1.配置了cluster_slave_validity_factor
2.data_age > 10*1000 + 15000*10 = 160s
3.不是手动failover
/* Check if our data is recent enough according to the slave validity
* factor configured by the user.
* Check bypassed for manual failovers. */
if (server.cluster_slave_validity_factor &&
data_age >
(((mstime_t)server.repl_ping_slave_period * 1000) +
(server.cluster_node_timeout * server.cluster_slave_validity_factor)))
{
if (!manual_failover) {
clusterLogCantFailover(CLUSTER_CANT_FAILOVER_DATA_AGE);
return;
}
}排除以上不能failover的情况以外,如果可以failover下边提供了具体的failover的规则
如果上一次故障切换尝试时间已过，且重试时间已过，则可以设置新的故障切换尝试时间
只有master为fail状态，slave才会发起选举。但并不是master为fail时立即发起选举，而是延迟下列随机时长，以避免多个slaves同时发起选举（至少延迟0.5秒后才会发起选举）：
500 milliseconds + random delay between 0 and 500 milliseconds + SLAVE_RANK * 1000 milliseconds
/* If the previous failover attempt timedout and the retry time has
* elapsed, we can setup a new one. */
if (auth_age > auth_retry_time) {
server.cluster->failover_auth_time = mstime() +
500 + /* Fixed delay of 500 milliseconds, let FAIL msg propagate. */
random() % 500; /* Random delay between 0 and 500 milliseconds. */
server.cluster->failover_auth_count = 0;
server.cluster->failover_auth_sent = 0;
server.cluster->failover_auth_rank = clusterGetSlaveRank();
/* We add another delay that is proportional to the slave rank.
* Specifically 1 second * rank. This way slaves that have a probably
* less updated replication offset, are penalized. */
server.cluster->failover_auth_time +=
server.cluster->failover_auth_rank * 1000;
/* However if this is a manual failover, no delay is needed. */
if (server.cluster->mf_end) {
server.cluster->failover_auth_time = mstime();
server.cluster->failover_auth_rank = 0;
clusterDoBeforeSleep(CLUSTER_TODO_HANDLE_FAILOVER);
}
serverLog(LL_WARNING,
"Start of election delayed for %lld milliseconds "
"(rank #%d, offset %lld).",
server.cluster->failover_auth_time - mstime(),
server.cluster->failover_auth_rank,
replicationGetSlaveOffset());
/* Now that we have a scheduled election, broadcast our offset
* to all the other slaves so that they'll updated their offsets
* if our offset is better. */
clusterBroadcastPong(CLUSTER_BROADCAST_LOCAL_SLAVES);
return;
}