pulsar版本升级

濮阳奇逸

2023-12-01

pulsar升级

当前版本

2.7.1

升级到的版本

本次升级至2.9.3，pulsar当前最新版本为2.10.1
https://pulsar.apache.org/release-notes/versioned/pulsar-2.9.3

升级方式

滚动升级

升级顺序

Upgrade ZooKeeper (optional)
Upgrade bookies
Upgrade brokers

升级步骤

一、准备包

1、下载2.9.3版本的tar包

https://archive.apache.org/dist/pulsar/pulsar-2.9.3/

wget https://archive.apache.org/dist/pulsar/pulsar-2.9.3/apache-pulsar-2.9.3-bin.tar.gz

2、下载2.9.3版本的内置连接器

https://archive.apache.org/dist/pulsar/pulsar-2.9.3/connectors/

 #原内置连接器
 pulsar-io-canal-2.7.1.nar
 pulsar-io-elastic-search-2.7.1.nar
 pulsar-io-file-2.7.1.nar
 pulsar-io-hdfs2-2.7.1.nar
 pulsar-io-hdfs3-2.7.1.nar
 pulsar-io-influxdb-2.7.1.nar
 pulsar-io-jdbc-mariadb-2.7.1.nar
 pulsar-io-jdbc-postgres-2.7.1.nar
 pulsar-io-kafka-2.7.1.nar
 pulsar-io-kafka-connect-adaptor-2.7.1.nar
 pulsar-io-netty-2.7.1.nar
 pulsar-io-rabbitmq-2.7.1.nar
 pulsar-io-redis-2.7.1.nar
 
tar xf apache-pulsar-2.9.3-bin.tar.gz -C /opt
cd /opt/apache-pulsar-2.9.3
mkdir connectors
cd connectors
wget https://archive.apache.org/dist/pulsar/pulsar-2.9.3/connectors/pulsar-io-canal-2.9.3.nar
wget https://archive.apache.org/dist/pulsar/pulsar-2.9.3/connectors/pulsar-io-elastic-search-2.9.3.nar
wget https://archive.apache.org/dist/pulsar/pulsar-2.9.3/connectors/pulsar-io-file-2.9.3.nar
wget https://archive.apache.org/dist/pulsar/pulsar-2.9.3/connectors/pulsar-io-hdfs2-2.9.3.nar
wget https://archive.apache.org/dist/pulsar/pulsar-2.9.3/connectors/pulsar-io-hdfs3-2.9.3.nar
wget https://archive.apache.org/dist/pulsar/pulsar-2.9.3/connectors/pulsar-io-influxdb-2.9.3.nar
wget https://archive.apache.org/dist/pulsar/pulsar-2.9.3/connectors/pulsar-io-jdbc-mariadb-2.9.3.nar
wget https://archive.apache.org/dist/pulsar/pulsar-2.9.3/connectors/pulsar-io-jdbc-postgres-2.9.3.nar
wget https://archive.apache.org/dist/pulsar/pulsar-2.9.3/connectors/pulsar-io-kafka-2.9.3.nar
wget https://archive.apache.org/dist/pulsar/pulsar-2.9.3/connectors/pulsar-io-kafka-connect-adaptor-2.9.3.nar
wget https://archive.apache.org/dist/pulsar/pulsar-2.9.3/connectors/pulsar-io-netty-2.9.3.nar
wget https://archive.apache.org/dist/pulsar/pulsar-2.9.3/connectors/pulsar-io-rabbitmq-2.9.3.nar
wget https://archive.apache.org/dist/pulsar/pulsar-2.9.3/connectors/pulsar-io-redis-2.9.3.nar

3、下载kop

https://github.com/streamnative/kop/releases

wget https://github.com/streamnative/kop/releases/download/v2.9.3.4/pulsar-protocol-handler-kafka-2.9.3.4.nar
mkdir protocol-handler
mv pulsar-protocol-handler-kafka-2.9.3.4.nar protocol-handler

二、准备配置文件

1、同步2.7.1版本的配置文件到新版本目录下

cp /opt/pulsar/conf/* /opt/apache-pulsar-2.9.3/conf/

2、修改bookeeper.conf

Zookeeper占用8000端口

ZooKeeper 端口
Pulsar 默认使用的 ZooKeeper 配置文件中 metrics 端口设成了 8000，而非 ZooKeeper 默认的 7000。ZooKeeper 和 BooKkeeper metrics 端口冲突。

这里确实会存在和 BookKeeper metric 端口冲突的问题，如果是在同一台机器上混布 BookKeeper 和 Zookeeper 的话，建议修改其中一个端口。
可以通过 export PULSAR_EXTRA_OPTS=-Dstats_server_port=8002 修改 ZooKeeper metrics 端口号。

PULSAR_EXTRA_OPTS="-Dstats_server_port=8001" pulsar-daemon start zookeeper

如果启动zookeeper时，指定其metrics为非8000端口，则忽略修改bookeeper端口

#按报错提示修改配置
cat bookeeper.conf
prometheusStatsHttpPort=8000
advertisedAddress=

可选：sed -i 's/prometheusStatsHttpPort=8000/prometheusStatsHttpPort=8001/g' /opt/apache-pulsar-2.9.3/conf/bookkeeper.conf
sed -i 's/advertisedAddress=/advertisedAddress=10.11.0.12/g' /opt/apache-pulsar-2.9.3/conf/bookkeeper.conf

三、升级

https://pulsar.apache.org/docs/2.9.3/administration-upgrade

1、升级zookeeper（可选）

本次升级为必选否则启动broker时报错：https://github.com/apache/pulsar/pull/11198

升级 ZooKeeper 服务器的步骤如下：

1. 终止 ZooKeeper服务。
2. 升级二进制文件和配置文件。
3. 用新的二进制文件启动 ZooKeeper 服务。
4. 用 pulsar zookeeper-shell 连接新的 ZooKeeper 服务，并运行命令测试其运行状态。
5. 运行ZooKeeper服务几天，观察 ZooKeeper 集群的运行状态，确保可以正常运行。

1.1、停止旧版本zookeeper

cd /opt/pulsar
./bin/pulsar-daemon stop zookeeper

1.2、升级二进制文件和配置文件。

1.3、用新的二进制文件启动 ZooKeeper 服务

cd /opt/apache-pulsar-2.9.3
./bin/pulsar-daemon start zookeeper

1.4、升级后验证

用 pulsar zookeeper-shell 连接新的 ZooKeeper 服务，并运行命令测试其运行状态。
#输出
WatchedEvent state:SyncConnected type:None path:null
[zk: xxxx:6181(CONNECTED) 0] ls /
[admin, bookies, counters, ledgers, loadbalance, managed-ledgers, namespace, schemas, stream, zookeeper
]

失败回滚

关闭出故障的 ZooKeeper 节点，使用旧版本的二进制文件和配置文件重新启动 ZooKeeper。

2、升级bookeeper

https://bookkeeper.apache.org/docs/admin/upgrade/
https://pulsar.apache.org/docs/2.9.3/administration-upgrade
升级前，需要确定是否一次性升级整个集群，包括停机升级和滚动升级。
在滚动升级中，一次升级一个 bookie。在停机升级中，需先关闭全部集群，升级所有 bookie 后再启动集群。

2.1、禁用 autorecovery

cd /opt/pulsar
./bin/bookkeeper shell autorecovery -disable

2.2、停止旧的bookeeper

cd /opt/pulsar
./bin/pulsar-daemon stop bookie

2.3、新版本只读模式启动 bookie，以验证此新版本 bookie 的读性能

cd /opt/apache-pulsar-2.9.3
./bin/pulsar bookie --readOnly

2.4、报错及解决

报错

15:03:02.872 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server
java.net.UnknownHostException: Trying to listen on loopback address, 127.0.0.1:3181 but this is forbidden by default (see ServerConfiguration#getAllowLoopback()).
If this happen, you can consider specifying the network interface to listen on (e.g. listeningInterface=eth0) or specifying the advertised address (e.g. advertisedAddress=172.x.y.z)
	at org.apache.bookkeeper.bookie.Bookie.getBookieAddress(Bookie.java:608) ~[org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
	at org.apache.bookkeeper.bookie.Bookie.getBookieId(Bookie.java:563) ~[org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
	at org.apache.bookkeeper.proto.BookieNettyServer.<init>(BookieNettyServer.java:166) ~[org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
	at org.apache.bookkeeper.proto.BookieServer.<init>(BookieServer.java:118) ~[org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
	at org.apache.bookkeeper.server.service.BookieService.<init>(BookieService.java:52) ~[org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
	at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:304) ~[org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
	at org.apache.bookkeeper.server.Main.doMain(Main.java:226) [org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
	at org.apache.bookkeeper.server.Main.main(Main.java:208) [org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]

 [main] ERROR org.apache.bookkeeper.common.component.AbstractLifecycleComponent - Failed to start Component: stats-provider
java.lang.RuntimeException: java.io.IOException: Failed to bind to /0.0.0.0:8000

解决

#按报错提示修改配置
#涉及到的配置：
conf/bookeeper.conf
prometheusStatsHttpPort=8000
httpServerPort=8000
advertisedAddress=

报错

./bin/bookkeeper shell decommissionbookie

14:29:58.801 [main] ERROR org.apache.bookkeeper.client.BookKeeperAdmin - Autorecovery is disabled. So giving up!
14:29:58.801 [main] ERROR org.apache.bookkeeper.tools.cli.commands.bookies.DecommissionCommand - Received exception in DecommissionBookieCmd
org.apache.bookkeeper.replication.ReplicationException$UnavailableException: Autorecovery is disabled. So giving up!
	at org.apache.bookkeeper.client.BookKeeperAdmin.triggerAudit(BookKeeperAdmin.java:1523) ~[org.apache.bookkeeper-bookkeeper-server-4.14.5.jar:4.14.5]
	at org.apache.bookkeeper.client.BookKeeperAdmin.decommissionBookie(BookKeeperAdmin.java:1565) ~[org.apache.bookkeeper-bookkeeper-server-4.14.5.jar:4.14.5]
	at org.apache.bookkeeper.tools.cli.commands.bookies.DecommissionCommand.decommission(DecommissionCommand.java:97) [org.apache.bookkeeper-bookkeeper-server-4.14.5.jar:4.14.5]
	at org.apache.bookkeeper.tools.cli.commands.bookies.DecommissionCommand.apply(DecommissionCommand.java:82) [org.apache.bookkeeper-bookkeeper-server-4.14.5.jar:4.14.5]
	at org.apache.bookkeeper.bookie.BookieShell$DecommissionBookieCmd.runCmd(BookieShell.java:1956) [org.apache.bookkeeper-bookkeeper-server-4.14.5.jar:4.14.5]
	at org.apache.bookkeeper.bookie.BookieShell$MyCommand.runCmd(BookieShell.java:238) [org.apache.bookkeeper-bookkeeper-server-4.14.5.jar:4.14.5]
	at org.apache.bookkeeper.bookie.BookieShell.run(BookieShell.java:2278) [org.apache.bookkeeper-bookkeeper-server-4.14.5.jar:4.14.5]

解决

 ./bin/bookkeeper shell autorecovery -enable

报错

13:56:45.817 [main] ERROR org.apache.bookkeeper.bookie.Bookie - There are directories without a cookie, and this is neither a new environment, nor is storage expansion enabled. Empty directories are [/data/pulsar/data/bookkeeper/journal/current, /data/pulsar/data/bookkeeper/ledgers/current]
13:56:45.817 [main] INFO  org.apache.bookkeeper.proto.BookieNettyServer - Shutting down BookieNettyServer
13:56:45.836 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server
org.apache.bookkeeper.bookie.BookieException$InvalidCookieException:
	at org.apache.bookkeeper.bookie.Bookie.checkEnvironmentWithStorageExpansion(Bookie.java:494) ~[org.apache.bookkeeper-bookkeeper-server-4.14.5.jar:4.14.5]
	at org.apache.bookkeeper.bookie.Bookie.checkEnvironment(Bookie.java:273) ~[org.apache.bookkeeper-bookkeeper-server-4.14.5.jar:4.14.5]
	at org.apache.bookkeeper.bookie.Bookie.<init>(Bookie.java:731) ~[org.apache.bookkeeper-bookkeeper-server-4.14.5.jar:4.14.5]
	at org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:152) ~[org.apache.bookkeeper-bookkeeper-server-4.14.5.jar:4.14.5]
	at org.apache.bookkeeper.proto.BookieServer.<init>(BookieServer.java:120) ~[org.apache.bookkeeper-bookkeeper-server-4.14.5.jar:4.14.5]
	at org.apache.bookkeeper.server.service.BookieService.<init>(BookieService.java:52) ~[org.apache.bookkeeper-bookkeeper-server-4.14.5.jar:4.14.5]
	at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:304) ~[org.apache.bookkeeper-bookkeeper-server-4.14.5.jar:4.14.5]
	at org.apache.bookkeeper.server.Main.doMain(Main.java:226) [org.apache.bookkeeper-bookkeeper-server-4.14.5.jar:4.14.5]
	at org.apache.bookkeeper.server.Main.main(Main.java:208) [org.apache.bookkeeper-bookkeeper-server-4.14.5.jar:4.14.5]

解决

https://github.com/apache/pulsar/issues/3121

 run decommissionbookie to decommission the bookie first.
 The decommission command will re-replicate the data and delete the cookie in the zookeeper. 
 So the bookie can restart freshly
 运行decommissionbookie停用 bookie。
 decommission 命令将重新复制数据并删除 zookeeper 中的 cookie。
 所以 bookie 可以重新启动
 
 ./bin/bookkeeper shell autorecovery -enable
 ./bin/bookkeeper shell decommissionbookie
 ./bin/pulsar-daemon start bookie
 ./bin/bookkeeper shell recover xxxx:9181
 验证查看bookie可用节点个数
 ./bin/bookkeeper shell listbookies -rw

2.5、测试通过，后台启动bookie

cd /opt/apache-pulsar-2.9.3
./bin/pulsar-daemon start bookie

#验证是否启动成功
jps -l
#查看端口
ss -anptl |grep 进程号
ps -ef |grep 进程号

./bin/bookkeeper shell simpletest --ensemble 3 --writeQuorum 3 --ackQuorum 3 --numEntries 3

./bin/bookkeeper shell bookiesanity
#启动成功输出
- Bookie sanity test succeeded

2.6、升级所有节点的 bookie 后，重新启用 `autorecovery`

cd /opt/apache-pulsar-2.9.3
./bin/bookkeeper shell autorecovery -enable

失败回滚

https://bookkeeper.apache.org/docs/admin/autorecovery

方式一、启动一个新节点
 ./bin/bookkeeper shell autorecovery -enable
 ./bin/bookkeeper shell decommissionbookie
 ./bin/pulsar-daemon start bookie
 ./bin/bookkeeper shell recover xxxx:9181
 验证查看bookie可用节点个数
 ./bin/bookkeeper shell listbookies -rw
#手动 
e.g:
./bin/bookkeeper shell recover xxxx:3181
#自动：
停止出故障的 bookie 节点，通过 autorecovery 用其他集群中的 bookie 来替换出现异常的 bookie 节点。
也可以手动执行以下命令恢复
./bin/bookkeeper autorecovery

3、升级broker

终止 broker。

cd /opt/pulsar
./bin/pulsar-daemon stop broker

升级二进制文件和配置文件。
启动 broker。

cd /opt/apache-pulsar-2.9.3
./bin/pulsar-daemon start broker

#验证
jps -l
#查看端口
ss -anptl |grep 进程号
ps -ef |grep 进程号

查看集群 brokers 节点是否启动成功
./bin/pulsar-admin brokers list pulsar-cluster-prod

#输出为：
Warning: Nashorn engine is planned to be removed from a future JDK release
"10.11.0.10:8081"
"10.11.0.11:8081"
"10.11.0.12:8081"

3.1、报错

https://github.com/apache/pulsar/pull/11198

18:20:55.248 [main] ERROR org.apache.pulsar.broker.PulsarService - Failed to start Pulsar service: org.apache.zookeeper.KeeperException$UnimplementedException: KeeperErrorCode = Unimplemented for /
org.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$UnimplementedException: KeeperErrorCode = Unimplemented for /
	at org.apache.pulsar.metadata.impl.ZKMetadataStore.<init>(ZKMetadataStore.java:89) ~[org.apache.pulsar-pulsar-metadata-2.9.3.jar:2.9.3]
	at org.apache.pulsar.metadata.impl.MetadataStoreFactoryImpl.newInstance(MetadataStoreFactoryImpl.java:52) ~[org.apache.pulsar-pulsar-metadata-2.9.3.jar:2.9.3]
	at org.apache.pulsar.metadata.impl.MetadataStoreFactoryImpl.createExtended(MetadataStoreFactoryImpl.java:36) ~[org.apache.pulsar-pulsar-metadata-2.9.3.jar:2.9.3]
	at org.apache.pulsar.metadata.api.extended.MetadataStoreExtended.create(MetadataStoreExtended.java:43) ~[org.apache.pulsar-pulsar-metadata-2.9.3.jar:2.9.3]
	at org.apache.pulsar.broker.PulsarService.createLocalMetadataStore(PulsarService.java:928) ~[org.apache.pulsar-pulsar-broker-2.9.3.jar:2.9.3]
	at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:621) [org.apache.pulsar-pulsar-broker-2.9.3.jar:2.9.3]
	at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:273) [org.apache.pulsar-pulsar-broker-2.9.3.jar:2.9.3]
	at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:350) [org.apache.pulsar-pulsar-broker-2.9.3.jar:2.9.3]
Caused by: org.apache.zookeeper.KeeperException$UnimplementedException: KeeperErrorCode = Unimplemented for /
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:106) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
	at org.apache.zookeeper.ZooKeeper.addWatch(ZooKeeper.java:3192) ~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
	at org.apache.pulsar.metadata.impl.PulsarZooKeeperClient.access$3301(PulsarZooKeeperClient.java:74) ~[org.apache.pulsar-pulsar-metadata-2.9.3.jar:2.9.3]
	at org.apache.pulsar.metadata.impl.PulsarZooKeeperClient$23.call(PulsarZooKeeperClient.java:1130) ~[org.apache.pulsar-pulsar-metadata-2.9.3.jar:2.9.3]
	at org.apache.pulsar.metadata.impl.PulsarZooKeeperClient$23.call(PulsarZooKeeperClient.java:1124) ~[org.apache.pulsar-pulsar-metadata-2.9.3.jar:2.9.3]
	at org.apache.pulsar.metadata.impl.PulsarZooKeeperClient$ZooWorker.syncCallWithRetries(PulsarZooKeeperClient.java:1529) ~[org.apache.pulsar-pulsar-metadata-2.9.3.jar:2.9.3]
	at org.apache.pulsar.metadata.impl.PulsarZooKeeperClient.addWatch(PulsarZooKeeperClient.java:1124) ~[org.apache.pulsar-pulsar-metadata-2.9.3.jar:2.9.3]
	at org.apache.pulsar.metadata.impl.ZKMetadataStore.<init>(ZKMetadataStore.java:82) ~[org.apache.pulsar-pulsar-metadata-2.9.3.jar:2.9.3]
	... 7 more

3.2、解决

2.7.1升级至2.9.3需升级zookeeper集群
参照上方步骤三.1升级zookeeper

3.3、解决完报错后再次重启升级broker

cd /opt/pulsar-2.9.3
./bin/pulsar-daemon start broker

#验证
jps -l
#查看端口
ss -anptl |grep 进程号
ps -ef |grep 进程号

#依次升级其他节点的broker
查看集群 brokers 节点是否启动成功
./bin/pulsar-admin brokers list pulsar-cluster-prod

#输出为：
Warning: Nashorn engine is planned to be removed from a future JDK release
"10.11.0.10:8081"
"10.11.0.11:8081"
"10.11.0.12:8081"

失败回滚

关闭有问题的 broker节点， 使用旧版本的二进制文件和配置文件启动 broker。

四、验证集群

通过启动consumer和producer测试消息是否能传输成功。

建议先启动Consumer，再启动Producer。

Consumer

bin/pulsar-client consume \
  persistent://public/default/test \
  -n 100 \
  -s "consumer-test" \
  -t "Exclusive"

Produce

如果你指定了 pulsar-cluster-1 以外的集群名称，你需要在主题中使用不同的集群名称。

bin/pulsar-client produce \
  persistent://public/default/test \
  -n 1 \
  -m "Hello Pulsar"

结果

key:[null], properties:[], content:Hello Pulsar

pulsar版本升级

pulsar升级

当前版本

升级到的版本

升级方式

升级顺序

升级步骤

一、准备包

1、下载2.9.3版本的tar包

2、下载2.9.3版本的内置连接器

3、下载kop

二、准备配置文件

1、同步2.7.1版本的配置文件到新版本目录下

2、修改bookeeper.conf

三、升级

1、升级zookeeper（可选）

1.1、停止旧版本zookeeper

1.2、升级二进制文件和配置文件。

1.3、用新的二进制文件启动 ZooKeeper 服务

1.4、升级后验证

失败回滚

2、升级bookeeper

2.1、禁用 autorecovery

2.2、停止旧的bookeeper

2.3、新版本只读模式启动 bookie，以验证此新版本 bookie 的读性能

2.4、报错及解决

报错

解决

报错

解决

报错

解决

2.5、测试通过，后台启动bookie

2.6、升级所有节点的 bookie 后，重新启用 autorecovery

失败回滚

3、升级broker

3.1、报错

3.2、解决

3.3、解决完报错后再次重启升级broker

失败回滚

四、验证集群

相关阅读

相关文章

相关问答

相关文档

2.6、升级所有节点的 bookie 后，重新启用 `autorecovery`