resourcemanager UI界面无法访问报错KeeperErrorCode = ConnectionLoss for /rmstore
任伟
之前有4个Zookeeper节点，删除一个节点后，过段时间重启了resourcemanager服务，服务启动了，但是UI界面看不了，报错如下：
2016-05-05 16:13:22,683 INFO recovery.ZKRMStateStore (ZKRMStateStore.java:runWithRetries(1145)) - Retrying operation on ZK. Retry no. 283
2016-05-05 16:13:23,423 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(975)) - Opening socket connection to server testserver4.bj/10.111.32.53:2181. Will not attempt to authenticate using SASL (unknown error)
2016-05-05 16:13:23,424 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(852)) - Socket connection established to testserver4.bj/10.111.32.53:2181, initiating session
2016-05-05 16:13:23,425 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(1098)) - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2016-05-05 16:13:23,874 INFO zookeeper.ZooKeeper (ZooKeeper.java:close(684)) - Session: 0x0 closed
2016-05-05 16:13:23,874 INFO zookeeper.ZooKeeper (ZooKeeper.java:<init>(438)) - Initiating client connection, connectString=testserver1.bj:2181,testserver2.bj:2181,testserver4.bj:2181 sessionTimeout=10000 watcher=null
2016-05-05 16:13:23,874 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(512)) - EventThread shut down
2016-05-05 16:13:23,881 INFO recovery.ZKRMStateStore (ZKRMStateStore.java:createConnection(1184)) - Created new ZK connection
2016-05-05 16:13:23,883 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(975)) - Opening socket connection to server testserver4.bj/10.111.32.53:2181. Will not attempt to authenticate using SASL (unknown error)
2016-05-05 16:13:23,884 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(852)) - Socket connection established to testserver4.bj/10.111.32.53:2181, initiating session
2016-05-05 16:13:23,884 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(1098)) - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2016-05-05 16:13:23,985 INFO recovery.ZKRMStateStore (ZKRMStateStore.java:runWithRetries(1143)) - Exception while executing a ZK operation.
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /rmstore
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$1.run(ZKRMStateStore.java:309)
at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$1.run(ZKRMStateStore.java:305)
at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithCheck(ZKRMStateStore.java:1104)
at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithRetries(ZKRMStateStore.java:1125)
at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.createRootDir(ZKRMStateStore.java:305)
at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.createRootDirRecursively(ZKRMStateStore.java:1219)
at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.startInternal(ZKRMStateStore.java:288)
at org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore.serviceStart(RMStateStore.java:498)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:580)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:982)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1023)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1019)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1019)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1059)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1203)
2016-05-05 16:13:23,985 INFO recovery.ZKRMStateStore (ZKRMStateStore.java:runWithRetries(1145)) - Retrying operation on ZK. Retry no. 284
2016-05-05 16:13:24,136 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(975)) - Opening socket connection to server testserver1.bj/10.111.32.50:2181. Will not attempt to authenticate using SASL (unknown error)
2016-05-05 16:13:24,136 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(852)) - Socket connection established to testserver1.bj/10.111.32.50:2181, initiating session
2016-05-05 16:13:24,137 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(1098)) - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2016-05-05 16:13:24,315 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(975)) - Opening socket connection to server testserver2.bj/10.111.32.51:2181. Will not attempt to authenticate using SASL (unknown error)
2016-05-05 16:13:24,315 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(852)) - Socket connection established to testserver2.bj/10.111.32.51:2181, initiating session
2016-05-05 16:13:24,316 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(1098)) - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
重启所有Zookeeper问题解决。
2016-05-05 16:13:22,683 INFO recovery.ZKRMStateStore (ZKRMStateStore.java:runWithRetries(1145)) - Retrying operation on ZK. Retry no. 283
2016-05-05 16:13:23,423 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(975)) - Opening socket connection to server testserver4.bj/10.111.32.53:2181. Will not attempt to authenticate using SASL (unknown error)
2016-05-05 16:13:23,424 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(852)) - Socket connection established to testserver4.bj/10.111.32.53:2181, initiating session
2016-05-05 16:13:23,425 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(1098)) - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2016-05-05 16:13:23,874 INFO zookeeper.ZooKeeper (ZooKeeper.java:close(684)) - Session: 0x0 closed
2016-05-05 16:13:23,874 INFO zookeeper.ZooKeeper (ZooKeeper.java:<init>(438)) - Initiating client connection, connectString=testserver1.bj:2181,testserver2.bj:2181,testserver4.bj:2181 sessionTimeout=10000 watcher=null
2016-05-05 16:13:23,874 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(512)) - EventThread shut down
2016-05-05 16:13:23,881 INFO recovery.ZKRMStateStore (ZKRMStateStore.java:createConnection(1184)) - Created new ZK connection
2016-05-05 16:13:23,883 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(975)) - Opening socket connection to server testserver4.bj/10.111.32.53:2181. Will not attempt to authenticate using SASL (unknown error)
2016-05-05 16:13:23,884 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(852)) - Socket connection established to testserver4.bj/10.111.32.53:2181, initiating session
2016-05-05 16:13:23,884 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(1098)) - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2016-05-05 16:13:23,985 INFO recovery.ZKRMStateStore (ZKRMStateStore.java:runWithRetries(1143)) - Exception while executing a ZK operation.
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /rmstore
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$1.run(ZKRMStateStore.java:309)
at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$1.run(ZKRMStateStore.java:305)
at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithCheck(ZKRMStateStore.java:1104)
at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore$ZKAction.runWithRetries(ZKRMStateStore.java:1125)
at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.createRootDir(ZKRMStateStore.java:305)
at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.createRootDirRecursively(ZKRMStateStore.java:1219)
at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.startInternal(ZKRMStateStore.java:288)
at org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore.serviceStart(RMStateStore.java:498)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:580)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:982)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1023)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1019)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1019)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1059)
at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1203)
2016-05-05 16:13:23,985 INFO recovery.ZKRMStateStore (ZKRMStateStore.java:runWithRetries(1145)) - Retrying operation on ZK. Retry no. 284
2016-05-05 16:13:24,136 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(975)) - Opening socket connection to server testserver1.bj/10.111.32.50:2181. Will not attempt to authenticate using SASL (unknown error)
2016-05-05 16:13:24,136 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(852)) - Socket connection established to testserver1.bj/10.111.32.50:2181, initiating session
2016-05-05 16:13:24,137 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(1098)) - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
2016-05-05 16:13:24,315 INFO zookeeper.ClientCnxn (ClientCnxn.java:logStartConnect(975)) - Opening socket connection to server testserver2.bj/10.111.32.51:2181. Will not attempt to authenticate using SASL (unknown error)
2016-05-05 16:13:24,315 INFO zookeeper.ClientCnxn (ClientCnxn.java:primeConnection(852)) - Socket connection established to testserver2.bj/10.111.32.51:2181, initiating session
2016-05-05 16:13:24,316 INFO zookeeper.ClientCnxn (ClientCnxn.java:run(1098)) - Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
重启所有Zookeeper问题解决。
类似资料：