为什么检查点对延迟的影响如此之大？

Question

我观察到，在使用内存后端时，使用检查点会导致观察到的延迟意外增加。

考虑以下检查点：

2019-02-27 15:35:46,322 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator     - Triggering checkpoint 2 @ 1551281746322 for job a80597b3312f0704beed75397c371bf5.
2019-02-27 15:35:46,326 INFO  org.apache.flink.runtime.state.heap.HeapKeyedStateBackend     - Heap backend snapshot (In-Memory Stream Factory, synchronous part) in thread Thread[KeyedProcess -> Map -> Sink: Unnamed (1/1),5,Flink Task Threads] took 0 ms.
2019-02-27 15:35:46,342 INFO  org.apache.flink.runtime.state.DefaultOperatorStateBackend    - DefaultOperatorStateBackend snapshot (In-Memory Stream Factory, synchronous part) in thread Thread[Async calls on Source: Custom Source -> Map -> Timestamps/Watermarks (1/1),5,Flink Task Threads] took 2 ms.
2019-02-27 15:35:46,346 INFO  org.apache.flink.runtime.state.DefaultOperatorStateBackend    - DefaultOperatorStateBackend snapshot (In-Memory Stream Factory, asynchronous part) in thread Thread[pool-14-thread-2,5,Flink Task Threads] took 3 ms.
2019-02-27 15:35:46,351 INFO  org.apache.flink.runtime.state.heap.HeapKeyedStateBackend     - Heap backend snapshot (In-Memory Stream Factory, asynchronous part) in thread Thread[pool-11-thread-2,5,Flink Task Threads] took 14 ms.
2019-02-27 15:35:46,378 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator     - Completed checkpoint 2 for job a80597b3312f0704beed75397c371bf5 (1157653 bytes in 54 ms).

尽管端到端持续时间仅为50ms，但在15:35:46385注入的事件的响应仅在520ms后到达。在这两个时间戳之间，没有处理任何事件。在没有检查点的情况下，99.99%的延迟约为15ms。

设置：

• 平行度=1

编辑：这是一项线性工作，所以我想检查点屏障并没有对齐。

Answer 1

将时间花在同步确认消息到RabbitMQ上（MessageAcknowledgeingSourceBase\notifyCheckpointComplete

因为我的检查点间隔是3分钟，并且我正在注入200 ev/s，这意味着每个检查点都会触发36k条消息的确认（200*60*3），这大约需要500毫秒。

使用较小的间隔可能有助于获得更可预测的延迟，但代价是更高的中值延迟。