cdh6.2仍然无法解决zookeeper注册信息过期造成yarn的主备脑裂问题。
摘录别人的文章:https://sukbeta.github.io/zookeeper-delete-yarn-rmstore/
错误日志(日志可能会有不同,现象就是无法手动切换主备)
2019-06-16 21:07:39,030 WARN org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=yarn OPERATION=transitionToActive TARGET=RMHAProtocolService RESULT=FAILURE DESCRIPTION=Exception transitioning to active PERMISSIONS=Users [yarn] and members of the groups [<users>] are allowed
2019-06-16 21:07:39,030 WARN org.apache.hadoop.ha.ActiveStandbyElector: Exception handling the winning of election
org.apache.hadoop.ha.ServiceFailedException: RM could not transition to Active
at org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService.becomeActive(EmbeddedElectorService.java:124)
at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:812)
at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:417)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:599)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
Caused by: org.apache.hadoop.ha.ServiceFailedException: Error when transitioning to Active mode
at org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:304)
at org.apache.hadoop.yarn.server.resourcemanager.EmbeddedElectorService.becomeActive(EmbeddedElectorService.java:122)
... 4 more
Caused by: org.apache.hadoop.service.ServiceStateException: org.apache.hadoop.yarn.server.resourcemanager.recovery.StoreFencedException: RMStateStore has been fenced
at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:204)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:577)
at org.apache.hadoop.service.Abstr
解决办法:
我们需要删除yarn在ZK上的 rmstore 信息, 之后重启yarn,就可以了。
但是在删除zk上 rmstore 信息的时候, 遇到了问题, yarn在注册时候的时候自己添加上ACL。所以我们直接删除是不行的。
但我们可以可以重新设置一个ACL,就可以了, 如下:
cd $ZOOKEEPER_HOME/bin
./zkCli.sh # 连接hadoop配置的zk,如果是客户端需添加 -server IP:port
[zk: localhost:2181(CONNECTED) 0] ls /
[zookeeper, hadoop-ha, hbase, rmstore]
[zk: localhost:2181(CONNECTED) 1] rmr /rmstore
Authentication is not valid : /rmstore/ZKRMStateRoot/RMVersionNode
我们可以看一下这个目录的ACL
[zk: localhost:2181(CONNECTED) 2] getAcl /rmstore/ZKRMStateRoot
'world,'anyone
: rwa
'digest,'shining-namenode01.host.com:yelhKlz39YVCV9p4NTModoBq9fw=
: cd
我们重新设置ACL,并删除目录
[zk: localhost:2181(CONNECTED) 3] setAcl /rmstore/ZKRMStateRoot world:anyone:rwcda
cZxid = 0x10000001b
ctime = Mon May 27 14:58:45 CST 2019
mZxid = 0x10000001b
mtime = Mon May 27 14:58:45 CST 2019
pZxid = 0x10016efd3
cversion = 380191
dataVersion = 0
aclVersion = 5
ephemeralOwner = 0x0
dataLength = 0
numChildren = 5
[zk: localhost:2181(CONNECTED) 4] getAcl /rmstore/ZKRMStateRoot
'world,'anyone
: cdrwa
[zk: localhost:2181(CONNECTED) 6] rmr /rmstore/ZKRMStateRoot
[zk: localhost:2181(CONNECTED) 7] ls /
[zookeeper, hadoop-ha, hbase, rmstore]
[zk: localhost:2181(CONNECTED) 8] rmr /rmstore
[zk: localhost:2181(CONNECTED) 9] ls /
[zookeeper, hadoop-ha, hbase]
之后重新启动yarn,让yarn重新在zk上注册就可以了。