Cassandra驱动程序异常“所有尝试查询的主机失败”每隔几个小时出现一次，没有任何解释

Question

我的卡桑德拉集群（4 节点集群）有问题。卡桑德拉版本是2.2.9，驱动程序版本是3.0.3。
几个小时（〜3小时）后，我在驱动程序日志中看到以下问题：

1. 传出直接记忆错误（零星发生，大部分时间不受影响）
2. 没有与整数版本匹配的协议版本
3. 未知响应操作码
4. 检测信号查询超时
5. 所有尝试查询的主机都失败 --

Cassandra群集运行正常，当我重新启动应用程序时，一切都可以再次工作几个小时。

日志片段:

First Time                       Count  Message
2017-11-11 19:03:03 +0100            51  [/??.???.??.??:????] preparing to open ? new connections, total = ???
2017-11-11 19:03:03 +0100            49  [/??.???.??.??:????] Connection[/??.???.??.??:????-???, inFlight=?, closed=false] Transport initialized, connection ready
2017-11-11 19:03:03 +0100            24  [/??.???.??.??:????] Connection[/??.???.??.??:????-???, inFlight=?, closed=true] closed, remaining = ???
2017-11-11 19:03:29 +0100             1  Defuncting Connection[/??.???.??.??:????-???, inFlight=???, closed=false] because: [/??.???.??.??:????] Unexpected exception triggered (io.netty.handler.codec.DecoderException: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate ??????? byte(s) of direct memory (used: ???????, max: ????????))
2017-11-11 19:03:29 +0100            14  [/??.???.??.??:????] Connection[/??.???.??.??:????-???, inFlight=???, closed=false] failed, remaining = ???
2017-11-11 19:03:29 +0100             7  [/??.???.??.??:????] Connection[/??.???.??.??:????-???, inFlight=??, closed=false] failed, remaining = ???
2017-11-11 19:03:29 +0100             1  Defuncting Connection[/??.???.??.??:????-???, inFlight=??, closed=false] because: [/??.???.??.??:????] Unexpected exception triggered (io.netty.handler.codec.DecoderException: java.lang.IllegalArgumentException: No protocol version matching integer version ?)
2017-11-11 19:03:29 +0100             5  Defuncting Connection[/??.???.??.??:????-???, inFlight=??, closed=false] because: [/??.???.??.??:????] Unexpected exception triggered (io.netty.handler.codec.DecoderException: com.datastax.driver.core.exceptions.DriverInternalError: Unknown response opcode ??)
2017-11-11 19:03:29 +0100             4  Defuncting Connection[/??.???.??.??:????-???, inFlight=???, closed=false] because: [/??.???.??.??:????] Unexpected exception triggered (io.netty.handler.codec.DecoderException: com.datastax.driver.core.exceptions.DriverInternalError: Unknown response opcode ?)
2017-11-11 19:03:29 +0100             3  Defuncting Connection[/??.???.??.??:????-???, inFlight=???, closed=false] because: [/??.???.??.??:????] Unexpected exception triggered (io.netty.handler.codec.DecoderException: com.datastax.driver.core.exceptions.DriverInternalError: Unknown response opcode -???)
2017-11-11 19:03:30 +0100             3  Defuncting Connection[/??.???.??.??:????-???, inFlight=???, closed=false] because: [/??.???.??.??:????] Unexpected exception triggered (io.netty.handler.codec.DecoderException: com.datastax.driver.core.exceptions.DriverInternalError: Unknown response opcode ??)
2017-11-11 19:03:30 +0100             2  Defuncting Connection[/??.???.??.??:????-???, inFlight=?, closed=false] because: [/??.???.??.??:????] Unexpected exception triggered (io.netty.handler.codec.DecoderException: com.datastax.driver.core.exceptions.DriverInternalError: Unknown response opcode ??)
2017-11-11 19:03:30 +0100           401  [/??.???.??.??:????] Connection[/??.???.??.??:????-???, inFlight=?, closed=false] failed, remaining = ???
2017-11-11 19:03:33 +0100             1  Defuncting Connection[/??.???.??.??:????-???, inFlight=???, closed=false] because: [/??.???.??.??:????] Unexpected exception triggered (io.netty.handler.codec.DecoderException: com.datastax.driver.core.exceptions.DriverInternalError: Unknown response opcode ???)
2017-11-11 19:03:41 +0100           722  Defuncting Connection[/??.???.??.??:????-???, inFlight=?, closed=false] because: [/??.???.??.??:????] Heartbeat query timed out
2017-11-11 19:03:41 +0100             8  [/??.???.??.??:????] Connection[/??.???.??.??:????-?, inFlight=?, closed=false] failed, remaining = ???
2017-11-11 19:03:41 +0100            11  Defuncting Connection[/??.???.??.??:????-?, inFlight=?, closed=false] because: [/??.???.??.??:????] Heartbeat query timed out
2017-11-11 19:03:41 +0100            67  [/??.???.??.??:????] Connection[/??.???.??.??:????-??, inFlight=?, closed=false] failed, remaining = ???
2017-11-11 19:03:41 +0100           115  Defuncting Connection[/??.???.??.??:????-??, inFlight=?, closed=false] because: [/??.???.??.??:????] Heartbeat query timed out
2017-11-11 19:03:44 +0100             2  Defuncting Connection[/??.???.??.??:????-???, inFlight=??, closed=false] because: [/??.???.??.??:????] Unexpected exception triggered (io.netty.handler.codec.DecoderException: com.datastax.driver.core.exceptions.DriverInternalError: Unknown response opcode ?)
2017-11-11 19:03:51 +0100             2  Defuncting Connection[/??.???.??.??:????-???, inFlight=???, closed=false] because: [/??.???.??.??:????] Heartbeat query timed out
2017-11-11 19:03:51 +0100           265  Failed to post timeseries data Error Returned - 
2017-11-11 19:03:57 +0100             3  Defuncting Connection[/??.???.??.??:????-???, inFlight=??, closed=false] because: [/??.???.??.??:????] Heartbeat query timed out
2017-11-11 19:04:01 +0100            39  Defuncting Connection[/??.???.??.??:????-???, inFlight=?, closed=false] because: [/??.???.??.??:????] Operation timed out
2017-11-11 19:04:01 +0100            12  Error processing jobs: execution of statement failed:All host(s) tried for query failed (tried: /??.???.??.??:???? (com.datastax.driver.core.exceptions.DriverException: Timeout while trying to acquire available connection (you may want to increase the driver number of per-host connections)), /??.???.??.??:???? (com.datastax.driver.core.exceptions.DriverException: Timeout while trying to acquire available connection (you may want to increase the driver number of per-host connections)), /??.???.??.??:???? (com.datastax.driver.core.exceptions.DriverException: Timeout while trying to acquire available connection (you may want to increase the driver number of per-host connections)), /??.???.??.??:???? [only showing errors of first ? hosts, use getErrors() for more details])

有人知道什么是根本原因吗？

Answer 1

请检查您为应用程序使用的直接内存设置。确保驱动程序可以分配足够的可用存储器。Cassandra需要分配直接内存。在无法分配内存的情况下，我已经看到类似的问题，即使与内存相关，它也被报告为NoHost可用性异常。

Answer 2

Defuncting Connection[/??.???.??.??:????-???, inFlight=???, closed=false] because: [/??.???.??.??:????] Unexpected exception triggered (io.netty.handler.codec.DecoderException: io.netty.util.internal.OutOfDirectMemoryError: failed to allocate ??????? byte(s) of direct memory (used: ???????, max: ????????))

你的记忆有些问题。只要这些问题存在，就不能指望驱动程序正常工作。您还说您的应用程序在几个小时后停止工作。对我来说，听起来你的应用程序中有内存泄漏。