org.apache.cassandra.exceptions。ReadTimeoutException:操作超时-仅收到0个响应

Question

我的Cassandra Java客户端出现了以下错误。

java.util.concurrent.ExecutionException: org.springframework.cassandra.support.exception.CassandraConnectionFailureException: [DNS/IP:PORT] Connection has been closed; nested exception is com.datastax.driver.core.exceptions.TransportException: [DNS/IP:PORT] Connection has been closed
    at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
    at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)

同时，Cassandra日志中的错误如下。

ERROR [Native-Transport-Requests-37] 2022-07-19 09:11:38,152 ErrorMessage.java:384 - Unexpected exception during request
com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) ~[guava-18.0.jar:na]
        at com.google.common.cache.LocalCache.get(LocalCache.java:3937) ~[guava-18.0.jar:na]
        at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) ~[guava-18.0.jar:na]
        at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) ~[guava-18.0.jar:na]
        at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.service.ClientState.authorize(ClientState.java:439) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:368) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:345) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:332) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:310) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:211) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:221) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:255) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:240) ~[apache-cassandra-3.11.2.jar:3.11.2]
        at org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:116) ~[apache-cassandra-3.11.2.jar:3.11.2]

除了CPU IO等待(下面的红色尖峰)，一切(CPU使用/内存使用/每分钟读取查询计数等)看起来都很正常。当时没有运行压缩任务。

我在网上查看了多篇文章，问题和错误报告，但找不到任何令人信服的解决方案。类似的问题： 卡桑德拉：2.2.8：组织.apache.卡桑德拉.exceptions.读取超时异常：操作超时 - 仅收到 0 个响应

错误时间 - 上午 9：11 GC 活动在当时是正常的。GC活动在上午8：20异常。

这是一个虚拟机，邻居/兄弟姐妹没有得到充分利用。

关于我应该寻找什么有什么建议吗？

Answer 1

堆栈跟踪的这一部分向我指示在尝试检查身份验证缓存中的用户权限时请求失败：

    ...
    at org.apache.cassandra.auth.AuthCache.get(AuthCache.java:108) ~[apache-cassandra-3.11.2.jar:3.11.2]
    at org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:45) ~[apache-cassandra-3.11.2.jar:3.11.2]
    at org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104) ~[apache-cassandra-3.11.2.jar:3.11.2]
    ...

它实际上是在检查本地缓存，所以它应该非常快。我唯一一次看到身份验证缓存查找超时是当节点过载时，因此它们无响应。

检查长GC或多次出现短GC，这会导致长暂停时间。如果节点是虚拟机，请确保没有“噪音邻居”从节点窃取CPU周期。干杯