Cassandra读取延迟问题

Question

我正在开发Azure Managed Cassandra，目前正在观察从一个分区读取数据时的读取性能问题。卡珊德拉密钥空间

在一个RF为3的数据中心中使用SimpleStrategy创建了密钥空间。下面是表格说明:

CREATE TABLE ks1.table1 (
item text,
market text,
location int,
brand text,
channel text,
qty1 int,
locationtype text,
nonsellableqty int,
qty2 int,
qty3 int,
qty3 int,
refitem text,
reflocation int,
rtvqty int,
soh int,
tsfexpectedqty int,
tsfreservedqty int,
PRIMARY KEY (item, market, location) ) WITH CLUSTERING ORDER BY (market ASC, location ASC)
AND bloom_filter_fp_chance = 0.1
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND comment = ''
AND compaction = {'class': 'SizeTieredCompactionStrategy'} ;

  Query is  select * from table1 where item='1001';

对于单分区查询，响应时间大约为300到400毫秒。

我启用了跟踪来检查系统跟踪，下面是观察结果

   session_id                           | event_id                             | activity                                                                  | source        | source_elapsed | source_port | thread
--------------------------------------+--------------------------------------+---------------------------------------------------------------------------+---------------+----------------+-------------+------------------------------------------------
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb20f060-7756-11ec-bf3a-35d288fe166a |                                         reading digest from /host3 | host1 |                  209 |        null |                    Native-Transport-Requests-1
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb20f061-7756-11ec-bf3a-35d288fe166a |                                 Executing single-partition query on roles | host1 |            241 |        null |                                    ReadStage-3
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb20f062-7756-11ec-bf3a-35d288fe166a |                                              Acquiring sstable references | host1 |            269 |        null |                                    ReadStage-3
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb20f063-7756-11ec-bf3a-35d288fe166a | Skipped 0/2 non-slice-intersecting sstables, included 0 due to tombstones | host1 |            294 |        null |                                    ReadStage-3
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb20f064-7756-11ec-bf3a-35d288fe166a |                                               Key cache hit for sstable 1 | host1 |            329 |        null |                                    ReadStage-3
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb20f065-7756-11ec-bf3a-35d288fe166a |                                     Sending READ message to /host3 | host1 |                   330 |        null |  MessagingService-Outgoing-/host3-Small
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb20f066-7756-11ec-bf3a-35d288fe166a |                                               Key cache hit for sstable 2 | host1 |            373 |        null |                                    ReadStage-3
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb20f067-7756-11ec-bf3a-35d288fe166a |                                 Merged data from memtables and 2 sstables | host1 |            425 |        null |                                    ReadStage-3
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb20f068-7756-11ec-bf3a-35d288fe166a |                                    Read 1 live rows and 0 tombstone cells | host1 |            446 |        null |                                    ReadStage-3
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb211770-7756-11ec-a029-857a112314fb |                                 READ message received from /host1 |  host3 |                     2 |        null |       MessagingService-Incoming-/host1
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb211770-7756-11ec-bf3a-35d288fe166a |                      REQUEST_RESPONSE message received from /host3 | host1 |                  1247 |        null |        MessagingService-Incoming-/host3
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb211771-7756-11ec-a029-857a112314fb |                                 Executing single-partition query on roles |  host2 |             71 |        null |                                    ReadStage-3
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb211771-7756-11ec-bf3a-35d288fe166a |                                    Processing response from /host2 | host1 |                  1316 |        null |                         RequestResponseStage-3
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb211772-7756-11ec-a029-857a112314fb |                                              Acquiring sstable references |  host2 |             96 |        null |                                    ReadStage-3
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb211772-7756-11ec-bf3a-35d288fe166a |                                        reading digest from /host4 | host1 |                    1426 |        null |                    Native-Transport-Requests-1
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb211773-7756-11ec-a029-857a112314fb | Skipped 0/2 non-slice-intersecting sstables, included 0 due to tombstones |  host2 |            117 |        null |                                    ReadStage-3
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb211773-7756-11ec-bf3a-35d288fe166a |                    Executing single-partition query on table1 | host1 |           1441 |        null |                                    ReadStage-3
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb211774-7756-11ec-a029-857a112314fb |                                               Key cache hit for sstable 1 |  host2 |            144 |        null |                                    ReadStage-3
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb211774-7756-11ec-bf3a-35d288fe166a |                                              Acquiring sstable references | host1 |           1449 |        null |                                    ReadStage-3
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb211775-7756-11ec-a029-857a112314fb |                                               Key cache hit for sstable 2 |  host2 |            172 |        null |                                    ReadStage-3
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb211775-7756-11ec-bf3a-35d288fe166a |                                   speculating read retry on /host2 | host1 |                   1453 |        null |                    Native-Transport-Requests-1
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb211776-7756-11ec-a029-857a112314fb |                                 Merged data from memtables and 2 sstables |  host2 |            225 |        null |                                    ReadStage-3
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb211776-7756-11ec-bf3a-35d288fe166a |                                               Key cache hit for sstable 2 | host1 |           1467 |        null |                                    ReadStage-3
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb211777-7756-11ec-a029-857a112314fb |                                    Read 1 live rows and 0 tombstone cells |  host2 |            244 |        null |                                    ReadStage-3
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb211777-7756-11ec-bf3a-35d288fe166a |                                               Key cache hit for sstable 1 | host1 |           1481 |        null |                                    ReadStage-3
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb211778-7756-11ec-a029-857a112314fb |                                      Enqueuing response to /host1 |  host2 |                  250 |        null |                                    ReadStage-3
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb211778-7756-11ec-bf3a-35d288fe166a | Skipped 0/2 non-slice-intersecting sstables, included 0 due to tombstones | host1 |           1490 |        null |                                    ReadStage-3
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb211779-7756-11ec-a029-857a112314fb |                        Sending REQUEST_RESPONSE message to /host1 |  host2 |                  375 |        null | MessagingService-Outgoing-/host1-Small
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb211779-7756-11ec-bf3a-35d288fe166a |                                     Sending READ message to /host2 | host1 |                  1495 |        null |  MessagingService-Outgoing-/host2-Small
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb21177a-7756-11ec-bf3a-35d288fe166a |                                    Sending READ message to /host4 | host1 |           1501 |        null | MessagingService-Outgoing-/host4-Small
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb21177b-7756-11ec-bf3a-35d288fe166a |                                 Merged data from memtables and 2 sstables | host1 |           1603 |        null |                                    ReadStage-3
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb21177c-7756-11ec-bf3a-35d288fe166a |                                    Read 2 live rows and 2 tombstone cells | host1 |           1622 |        null |                                    ReadStage-3
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb213e80-7756-11ec-a029-857a112314fb |                                 READ message received from /host1 |  host2 |              1 |        null |       MessagingService-Incoming-/host1
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb213e80-7756-11ec-a9d1-87a6d519c8de |                                 READ message received from /host1 | host4 |              6 |        null |       MessagingService-Incoming-/host1
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb213e80-7756-11ec-bf3a-35d288fe166a |                     REQUEST_RESPONSE message received from /host4 | host1 |           2371 |        null |       MessagingService-Incoming-/host4
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb213e81-7756-11ec-a029-857a112314fb |                    Executing single-partition query on table1 |  host2 |            100 |        null |                                    ReadStage-1
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb213e81-7756-11ec-bf3a-35d288fe166a |                                   Processing response from /host4 | host1 |           2437 |        null |                         RequestResponseStage-2
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb213e82-7756-11ec-a029-857a112314fb |                                              Acquiring sstable references |  host2 |            117 |        null |                                    ReadStage-1
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb216590-7756-11ec-a029-857a112314fb |                        Partition index with 0 entries found for sstable 2 |  host2 |            665 |        null |                                    ReadStage-1
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb216590-7756-11ec-a9d1-87a6d519c8de |                    Executing single-partition query on table1 | host4 |            137 |        null |                                    ReadStage-8
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb216590-7756-11ec-bf3a-35d288fe166a |                      REQUEST_RESPONSE message received from /host2 | host1 |             39 |        null |        MessagingService-Incoming-/host2
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb216591-7756-11ec-a029-857a112314fb |                                               Key cache hit for sstable 1 |  host2 |           1063 |        null |                                    ReadStage-1
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb216591-7756-11ec-a9d1-87a6d519c8de |                                              Acquiring sstable references | host4 |            171 |        null |                                    ReadStage-8
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb216591-7756-11ec-bf3a-35d288fe166a |                                    Processing response from /host2 | host1 |            129 |        null |                         RequestResponseStage-2
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb216592-7756-11ec-a029-857a112314fb | Skipped 0/2 non-slice-intersecting sstables, included 0 due to tombstones |  host2 |           1087 |        null |                                    ReadStage-1
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb216592-7756-11ec-a9d1-87a6d519c8de |                                               Key cache hit for sstable 2 | host4 |            198 |        null |                                    ReadStage-8
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb216592-7756-11ec-bf3a-35d288fe166a |                                                    Initiating read-repair | host1 |            174 |        null |                         RequestResponseStage-2
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb216593-7756-11ec-a029-857a112314fb |                                 Merged data from memtables and 2 sstables |  host2 |           1354 |        null |                                    ReadStage-1
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb216593-7756-11ec-a9d1-87a6d519c8de |                                               Key cache hit for sstable 1 | host4 |            228 |        null |                                    ReadStage-8
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb216594-7756-11ec-a029-857a112314fb |                                    Read 2 live rows and 2 tombstone cells |  host2 |           1386 |        null |                                    ReadStage-1
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb216594-7756-11ec-a9d1-87a6d519c8de | Skipped 0/2 non-slice-intersecting sstables, included 0 due to tombstones | host4 |            238 |        null |                                    ReadStage-8
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb216595-7756-11ec-a029-857a112314fb |                                      Enqueuing response to /host1 |  host2 |           1394 |        null |                                    ReadStage-1
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb216595-7756-11ec-a9d1-87a6d519c8de |                                 Merged data from memtables and 2 sstables | host4 |            391 |        null |                                    ReadStage-8
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb216596-7756-11ec-a029-857a112314fb |                        Sending REQUEST_RESPONSE message to /host1 |  host2 |           1431 |        null | MessagingService-Outgoing-/host1-Small
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb216596-7756-11ec-a9d1-87a6d519c8de |                                    Read 2 live rows and 2 tombstone cells | host4 |            415 |        null |                                    ReadStage-8
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb216597-7756-11ec-a9d1-87a6d519c8de |                                      Enqueuing response to /host1 | host4 |            423 |        null |                                    ReadStage-8
 eb20c950-7756-11ec-bf3a-35d288fe166a | eb216598-7756-11ec-a9d1-87a6d519c8de |                        Sending REQUEST_RESPONSE message to /host1 | host4 |            499 |        null | MessagingService-Outgoing-/host1-Small

对于上述项目 ID（1001），我们有 2000 条记录。

当我们将RF指定为3时，为什么要对超过3台主机执行查询？我们期望在20毫秒内收到查询响应。请告诉我配置有什么问题吗？或者如何调整设置以达到亚毫秒的响应时间。

Answer 1

根据驱动程序设置-如果选择的协调器不是具有副本的节点之一，则至少需要2个其他节点被命中，因此可能导致3个节点在跟踪范围内，1个协调器，2个副本。

但是，还有其他 2 件事导致节点计数不仅仅是 3。

启动读取修复

如果检测到两个副本之间的不匹配，它将同步修复所有副本之间的数据，这本身将导致响应延迟。这将导致您看到的场景中有4个节点（协调器3个副本）被联系。

推测/host2上的读取重试

查询花费的时间足够长，因此已从驱动程序重新发出查询。这再次可能导致它将查询发送到另一个协调器，然后该协调器可以涉及不同的节点来回答查询。

问题的最后一部分是关于性能——这比较棘手。确保数据得到修复并避免摘要不匹配当然会有所帮助，但是在跟踪(可能只是测试数据)中，我们可以看到存在2个活动记录和2个逻辑删除。墓碑的存在对性能没有帮助，而且从表面上看，数据是从多个表中读取的。需要更多关于使用模式的信息来了解要优化的选项，但是如果出现大量的删除/更新，使用STCS将会导致问题。