解决Splunk kvstore “starting“ 问题

最近kvstore 很不稳定，一次三个同组的searh head 出现"starting" 的状态，当我用splunk show kvstore-status 检查的时候。

检查log:

1: /opt/spunk/var/log/splunkd.log

当时： https://abc.com:8089是captain (splunk show shcluster-status)

05-25-2022 18:31:36.907 +0800 INFO  SHCRaftConsensus [845326 TcpChannelThread] - Rejecting AppendEntries RPC from https://abc.com:8089

 

05-25-2022 18:31:41.241 +0800 ERROR KVStorageProvider [848995 TcpChannelThread] - Failed to create client connection (Local)

05-25-2022 18:31:41.241 +0800 ERROR KVStoreIntrospection [848995