Pulsar: No such ledger exists on Metadata Server

背景

1. 本地的一个Pulsar集群（版本为2.7.0）;
2. 为topic设置了retention为半个小时;
3. 一个flink-pulsar-source的connnector，一个pulsar的producer;

问题

1. 当producer断掉后，过了一段时间，broker报错如下：

06:13:39.598 [bookkeeper-ml-workers-OrderedExecutor-7-0] ERROR org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/default/persistent/stream-identity] Error opening ledger for reading at position 264656:23050 - org.apache.bookkeeper.mledger.ManagedLedgerException$NonRecoverableLedgerException: No such ledger exists on Metadata Server
06:13:39.599 [bookkeeper-ml-workers-OrderedExecutor-7-0] INFO  org.eclipse.jetty.server.RequestLog - 22.21.8.14 - - [04/Nov/2021:06:13:39 +0800] "GET /admin/v2/persistent/public/default/stream-identity/lastMessageId?authoritative=true HTTP/1.1" 500 53 "-" "Pulsar-Java-v2.7.2" 2

原因

当一个topic长时间没有producer写数据时，之前的消息就被删除了，也即消息所在的ledger被删除了。因此不能获取到之前的lastMessageId

It looks like that when a topic remains empty due to message retention, it is not possible to ask for the lastMessageId

When we are trimming the ledgers we are saving the currentLedger but as soon as your restart the broker the currentLedger is not containing the lastMessageId (because it is a fresh new ledger).

解决方案

Pulsar官网的一个issure

问题提出: https://github.com/apache/pulsar/issues/8677
解决方法： https://github.com/apache/pulsar/pull/8725/files#

步骤一

在源码中按照fix中更改的文件对源码进行更改，主要改的文件为：

1. managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java
   该文件对应的lib包为managed-ledger.jar，之后替换集群中的lib包。

2. pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java

   该文件对应的lib包为pulsar-broker.jar，之后替换集群中的lib包。

步骤二

更新lib包后，将集群中的节点按照RoundRobin的方式进行重启。例如集群中有三台节点，分别为broker1、 broker2和broker3。现在假设三台节点都是正常的。则依次执行如下命令：

./bin/pulsar-daemon stop broker1
./bin/pulsar-daemon start broker1

./bin/pulsar-daemon stop broker2
./bin/pulsar-daemon start broker1

./bin/pulsar-daemon stop broker3
./bin/pulsar-daemon start broker1

最后集群正常运行，且原来跑的任务未中断。