一、kafka-manager简介
为了简化开发者和服务工程师维护Kafka集群的工作，yahoo构建了一个叫做Kafka管理器的基于Web工具，叫做 Kafka Manager。这个管理工具可以很容易地发现分布在集群中的哪些topic分布不均匀，或者是分区在整个集群分布不均匀的的情况。它支持管理多个集群、选择副本、副本重新分配以及创建Topic。同时，这个管理工具也是一个非常好的可以快速浏览这个集群的工具，有如下功能：
- 管理多个集群
- 轻松检查群集状态（主题，消费者，偏移，代理，副本分发，分区分发）
- 运行首选副本选举
- 使用选项生成分区分配以选择要使用的代理
- 运行分区重新分配（基于生成的分配）
- 使用可选主题配置创建主题（0.8.1.1具有与0.8.2+不同的配置）
- 删除主题（仅支持0.8.2+并记住在代理配置中设置delete.topic.enable = true）
- 主题列表现在指示标记为删除的主题（仅支持0.8.2+）
- 批量生成多个主题的分区分配，并可选择要使用的代理
- 批量运行重新分配多个主题的分区
- 将分区添加到现有主题
- 更新现有主题的配置
- kafka-manager 项目地址：https://github.com/yahoo/kafka-manager/
二、kafka-manager安装
1.下载安装包
使用Git或者直接从Releases中下载，这里下载 1.3.3.18 版本：https://github.com/yahoo/kafka-manager/releases
wget https://github.com/yahoo/kafka-manager/archive/1.3.3.18.zip
2.解压安装包
Last login: Thu Sep 6 20:51:48 2018 from 192.168.0.103 [spark@master ~]$ cd /opt/ [spark@master opt]$ wget https://github.com/yahoo/kafka-manager/archive/1.3.3.18.zip [spark@master opt]$ ll total 1076 drwxr-xr-x. 6 root root 99 Jul 5 07:00 apache-maven-3.5.4 drwxr-xr-x. 9 spark spark 155 Jul 18 04:31 elasticsearch-6.2.2 drwxr-xr-x. 7 spark spark 4096 Jul 20 22:13 elasticsearch-head-master drwxr-xr-x. 11 spark spark 172 Jul 17 06:28 hadoop-2.9.0 drwxr-xr-x. 4 root root 153 Jul 9 05:35 hdfs-over-ftp-master drwxr-xr-x. 8 10 143 255 Mar 29 08:18 jdk1.8.0_171 drwxrwxrwx. 7 root root 101 Jul 17 07:01 kafka_2.11-1.1.0 -rw-r--r--. 1 root root 1075857 Sep 11 21:32 kafka-manager-1.3.3.18.zip drwxrwxr-x. 12 spark spark 232 Feb 17 2018 kibana-6.2.2-linux-x86_64 -rw-r--r--. 1 root root 9116 Apr 11 2016 mysql57-community-release-el7-8.noarch.rpm drwxrwxr-x. 14 spark spark 252 Aug 6 20:23 nifi-1.7.1 drwxr-xr-x. 7 spark spark 67 Jul 17 22:54 node-8.9.1 drwxr-xr-x. 10 spark spark 4096 Jul 17 22:33 node-v8.9.1 drwxrwxr-x. 6 2000 2000 50 Apr 17 2014 scala-2.11.0 drwxrwxrwx. 13 hadoop hadoop 236 Sep 6 22:34 spark-2.2.1-bin-hadoop2.7 drwxrwxrwx. 12 spark spark 4096 Aug 8 00:23 zookeeper-3.4.12 [spark@master opt]$ su root Password: [root@master opt]# unzip kafka-manager-1.3.3.18.zip Archive: kafka-manager-1.3.3.18.zip 8dcdbf8fabb0001691c9b52b447b656f498b4d7b creating: kafka-manager-1.3.3.18/ 。。。。 inflating: kafka-manager-1.3.3.18/test/kafka/test/SeededBroker.scala [root@master opt]# ls apache-maven-3.5.4 jdk1.8.0_171 mysql57-community-release-el7-8.noarch.rpm spark-2.2.1-bin-hadoop2.7 elasticsearch-6.2.2 kafka_2.11-1.1.0 nifi-1.7.1 zookeeper-3.4.12 elasticsearch-head-master kafka-manager-1.3.3.18 node-8.9.1 hadoop-2.9.0 kafka-manager-1.3.3.18.zip node-v8.9.1 hdfs-over-ftp-master kibana-6.2.2-linux-x86_64 scala-2.11.0
3.sbt编译
1）yum安装sbt(因为kafka-manager需要sbt编译)
[root@master opt]# cd kafka-manager-1.3.3.18
[root@master kafka-manager-1.3.3.18]# ls
app build.sbt conf img LICENCE project public README.md sbt src test
[root@master kafka-manager-1.3.3.18]# sbt
bash: sbt: command not found
[root@master kafka-manager-1.3.3.18]# cd ..
[root@master opt]# curl https://bintray.com/sbt/rpm/rpm > bintray-sbt-rpm.repo
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 160 0 160 0 0 21 0 --:--:-- 0:00:07 --:--:-- 40
[root@master opt]# ll
total 1080
drwxr-xr-x. 6 root root 99 Jul 5 07:00 apache-maven-3.5.4
-rw-r--r--. 1 root root 160 Sep 11 21:38 bintray-sbt-rpm.repo
drwxr-xr-x. 9 spark spark 155 Jul 18 04:31 elasticsearch-6.2.2
drwxr-xr-x. 7 spark spark 4096 Jul 20 22:13 elasticsearch-head-master
drwxr-xr-x. 11 spark spark 172 Jul 17 06:28 hadoop-2.9.0
drwxr-xr-x. 4 root root 153 Jul 9 05:35 hdfs-over-ftp-master
drwxr-xr-x. 8 10 143 255 Mar 29 08:18 jdk1.8.0_171
drwxrwxrwx. 7 root root 101 Jul 17 07:01 kafka_2.11-1.1.0
drwxr-xr-x. 9 root root 189 Jul 7 04:44 kafka-manager-1.3.3.18
-rw-r--r--. 1 root root 1075857 Sep 11 21:32 kafka-manager-1.3.3.18.zip
drwxrwxr-x. 12 spark spark 232 Feb 17 2018 kibana-6.2.2-linux-x86_64
-rw-r--r--. 1 root root 9116 Apr 11 2016 mysql57-community-release-el7-8.noarch.rpm
drwxrwxr-x. 14 spark spark 252 Aug 6 20:23 nifi-1.7.1
drwxr-xr-x. 7 spark spark 67 Jul 17 22:54 node-8.9.1
drwxr-xr-x. 10 spark spark 4096 Jul 17 22:33 node-v8.9.1
drwxrwxr-x. 6 2000 2000 50 Apr 17 2014 scala-2.11.0
drwxrwxrwx. 13 hadoop hadoop 236 Sep 6 22:34 spark-2.2.1-bin-hadoop2.7
drwxrwxrwx. 12 spark spark 4096 Aug 8 00:23 zookeeper-3.4.12
[root@master opt]# mv bintray-sbt-rpm.repo /etc/yum.repos.d/
[root@master opt]# yum install sbt
Loaded plugins: fastestmirror
Determining fastest mirrors
* base: mirrors.zju.edu.cn
* extras: mirrors.shu.edu.cn
* updates: mirrors.zju.edu.cn
base | 3.6 kB 00:00:00
bintray--sbt-rpm | 1.3 kB 00:00:00
extras | 3.4 kB 00:00:00
mysql-connectors-community | 2.5 kB 00:00:00
mysql-tools-community | 2.5 kB 00:00:00
mysql57-community | 2.5 kB 00:00:00
updates | 3.4 kB 00:00:00
(1/4): extras/7/x86_64/primary_db | 187 kB 00:00:00
(2/4): mysql-connectors-community/x86_64/primary_db | 26 kB 00:00:00
(3/4): bintray--sbt-rpm/primary | 3.8 kB 00:00:02
(4/4): updates/7/x86_64/primary_db | 5.2 MB 00:00:02
bintray--sbt-rpm 32/32
Resolving Dependencies
--> Running transaction check
---> Package sbt.noarch 0:1.2.1-0 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
====================================================================================================================================
Package Arch Version Repository Size
====================================================================================================================================
Installing:
sbt noarch 1.2.1-0 bintray--sbt-rpm 1.1 M
Transaction Summary
====================================================================================================================================
Install 1 Package
Total download size: 1.1 M
Installed size: 1.2 M
Is this ok [y/d/N]: y
Downloading packages:
sbt-1.2.1.rpm | 1.1 MB 00:00:19
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : sbt-1.2.1-0.noarch 1/1
Verifying : sbt-1.2.1-0.noarch 1/1
Installed:
sbt.noarch 0:1.2.1-0
Complete!
改仓库地址：（sbt 默认下载库文件很慢, 还时不时被打断），我们可以在用户目录下创建 touch ~/.sbt/repositories, 填上阿里云的镜像 # vi ~/.sbt/repositories
[root@master opt]# ls
apache-maven-3.5.4 jdk1.8.0_171 mysql57-community-release-el7-8.noarch.rpm spark-2.2.1-bin-hadoop2.7
elasticsearch-6.2.2 kafka_2.11-1.1.0 nifi-1.7.1 zookeeper-3.4.12
elasticsearch-head-master kafka-manager-1.3.3.18 node-8.9.1
hadoop-2.9.0 kafka-manager-1.3.3.18.zip node-v8.9.1
hdfs-over-ftp-master kibana-6.2.2-linux-x86_64 scala-2.11.0
[root@master opt]# sbt
Getting org.scala-sbt sbt 1.2.1 (this may take some time)...
^C[root@master opt]# sbt -version
Getting org.scala-sbt sbt 1.2.1 (this may take some time)...
^C[root@master opt]# cd kafka-manager-1.3.3.18
[root@master kafka-manager-1.3.3.18]# ./sbt clean dist
Downloading sbt launcher for 0.13.9:
From http://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/0.13.9/sbt-launch.jar
To /root/.sbt/launchers/0.13.9/sbt-launch.jar
Download failed. Obtain the jar manually and place it at /root/.sbt/launchers/0.13.9/sbt-launch.jar
[root@master kafka-manager-1.3.3.18]# vi ~/.sbt/repositories
[repositories]
local
aliyun-nexus: http://maven.aliyun.com/nexus/content/groups/public/
jcenter: https://jcenter.bintray.com/
typesafe-ivy-releases: https://repo.typesafe.com/typesafe/ivy-releases/, [organization]/[module]/[revision]/[type]s/[artifact](-[c
lassifier]).[ext], bootOnly
maven-central
~
~
~
~
~
~
2）编译kafka-manager
[root@master kafka-manager-1.3.3.18]# ./sbt clean dist
Downloading sbt launcher for 0.13.9:
From http://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/0.13.9/sbt-launch.jar
To /root/.sbt/launchers/0.13.9/sbt-launch.jar
[info] Loading project definition from /opt/kafka-manager-1.3.3.18/project
Missing bintray credentials /root/.bintray/.credentials. Some bintray features depend on this.
[info] Set current project to kafka-manager (in build file:/opt/kafka-manager-1.3.3.18/)
Missing bintray credentials /root/.bintray/.credentials. Some bintray features depend on this.
[warn] Credentials file /root/.bintray/.credentials does not exist
[success] Total time: 0 s, completed Sep 11, 2018 10:54:50 PM
。。。。。
[info] [SUCCESSFUL ] jline#jline;2.12.1!jline.jar (340ms)
[info] Done updating.
[warn] Scala version was updated by one of library dependencies:
[warn] * org.scala-lang:scala-library:(2.11.6, 2.11.8, 2.11.11, 2.11.7, 2.11.5, 2.11.0) -> 2.11.12
[warn] To force scalaVersion, add the following:
[warn] ivyScala := ivyScala.value map { _.copy(overrideScalaVersion = true) }
[warn] There may be incompatibilities among your library dependencies.
[warn] Here are some of the libraries that were evicted:
[warn] * org.webjars:jquery:1.11.1 -> 2.1.4
[warn] Run 'evicted' to see detailed eviction warnings
[info] Wrote /opt/kafka-manager-1.3.3.18/target/scala-2.11/kafka-manager_2.11-1.3.3.18.pom
[info] Compiling 127 Scala sources and 2 Java sources to /opt/kafka-manager-1.3.3.18/target/scala-2.11/classes...
[info] 'compiler-interface' not yet compiled for Scala 2.11.8. Compiling...
[info] Main Scala API documentation to /opt/kafka-manager-1.3.3.18/target/scala-2.11/api...
[info] Compilation completed in 8.323 s
model contains 701 documentable templates
[info] Main Scala API documentation successful.
[info] Packaging /opt/kafka-manager-1.3.3.18/target/scala-2.11/kafka-manager_2.11-1.3.3.18-javadoc.jar ...
[info] Done packaging.
[info] LESS compiling on 1 source(s)
[info] Packaging /opt/kafka-manager-1.3.3.18/target/scala-2.11/kafka-manager_2.11-1.3.3.18-web-assets.jar ...
[info] Done packaging.
[info] Packaging /opt/kafka-manager-1.3.3.18/target/scala-2.11/kafka-manager_2.11-1.3.3.18.jar ...
[info] Done packaging.
[info] Packaging /opt/kafka-manager-1.3.3.18/target/scala-2.11/kafka-manager_2.11-1.3.3.18-sans-externalized.jar ...
[info] Done packaging.
[info]
[info] Your package is ready in /opt/kafka-manager-1.3.3.18/target/universal/kafka-manager-1.3.3.18.zip
[info]
[success] Total time: 376 s, completed Sep 11, 2018 11:01:06 PM
[root@master kafka-manager-1.3.3.18]# ls
app build.sbt conf img LICENCE project public README.md sbt src target test
[root@master kafka-manager-1.3.3.18]# cd target/universal
[root@master universal]# ls
kafka-manager-1.3.3.18.zip scripts
[root@master universal]# scp kafka-manager-1.3.3.18.zip /opt/kafka-manager-1.3.3.18.zip
4.安装
重新解压编译好的kafka-manager-1.3.3.18.zip&修改配置文件
[root@master universal]# cd /opt/
[root@master opt]# ls
apache-maven-3.5.4 hadoop-2.9.0 kafka_2.11-1.1.0 kibana-6.2.2-linux-x86_64 node-8.9.1 spark-2.2.1-bin-hadoop2.7
elasticsearch-6.2.2 hdfs-over-ftp-master kafka-manager-1.3.3.18 mysql57-community-release-el7-8.noarch.rpm node-v8.9.1 zookeeper-3.4.12
elasticsearch-head-master jdk1.8.0_171 kafka-manager-1.3.3.18.zip nifi-1.7.1 scala-2.11.0
[root@master opt]# mv kafka-manager-1.3.3.18 kafka-manager-1.3.3.18-source
[root@master opt]# ll
total 77696
drwxr-xr-x. 6 root root 99 Jul 5 07:00 apache-maven-3.5.4
drwxr-xr-x. 9 spark spark 155 Jul 18 04:31 elasticsearch-6.2.2
drwxr-xr-x. 7 spark spark 4096 Jul 20 22:13 elasticsearch-head-master
drwxr-xr-x. 11 spark spark 172 Jul 17 06:28 hadoop-2.9.0
drwxr-xr-x. 4 root root 153 Jul 9 05:35 hdfs-over-ftp-master
drwxr-xr-x. 8 10 143 255 Mar 29 08:18 jdk1.8.0_171
drwxrwxrwx. 7 root root 101 Jul 17 07:01 kafka_2.11-1.1.0
drwxr-xr-x. 10 root root 203 Sep 11 22:54 kafka-manager-1.3.3.18-source
-rw-r--r--. 1 root root 79534088 Sep 11 23:01 kafka-manager-1.3.3.18.zip
drwxrwxr-x. 12 spark spark 232 Feb 17 2018 kibana-6.2.2-linux-x86_64
-rw-r--r--. 1 root root 9116 Apr 11 2016 mysql57-community-release-el7-8.noarch.rpm
drwxrwxr-x. 14 spark spark 252 Aug 6 20:23 nifi-1.7.1
drwxr-xr-x. 7 spark spark 67 Jul 17 22:54 node-8.9.1
drwxr-xr-x. 10 spark spark 4096 Jul 17 22:33 node-v8.9.1
drwxrwxr-x. 6 2000 2000 50 Apr 17 2014 scala-2.11.0
drwxrwxrwx. 13 hadoop hadoop 236 Sep 6 22:34 spark-2.2.1-bin-hadoop2.7
drwxrwxrwx. 12 spark spark 4096 Aug 8 00:23 zookeeper-3.4.12
[root@master opt]# unzip kafka-manager-1.3.3.18.zip
Archive: kafka-manager-1.3.3.18.zip
inflating: kafka-manager-1.3.3.18/lib/kafka-manager.kafka-manager-1.3.3.18-sans-externalized.jar
。。。。
inflating: kafka-manager-1.3.3.18/share/doc/api/index/index-d.html
inflating: kafka-manager-1.3.3.18/README.md
[root@master opt]# ll
total 77696
drwxr-xr-x. 6 root root 99 Jul 5 07:00 apache-maven-3.5.4
drwxr-xr-x. 9 spark spark 155 Jul 18 04:31 elasticsearch-6.2.2
drwxr-xr-x. 7 spark spark 4096 Jul 20 22:13 elasticsearch-head-master
drwxr-xr-x. 11 spark spark 172 Jul 17 06:28 hadoop-2.9.0
drwxr-xr-x. 4 root root 153 Jul 9 05:35 hdfs-over-ftp-master
drwxr-xr-x. 8 10 143 255 Mar 29 08:18 jdk1.8.0_171
drwxrwxrwx. 7 root root 101 Jul 17 07:01 kafka_2.11-1.1.0
drwxr-xr-x. 6 root root 70 Sep 11 23:02 kafka-manager-1.3.3.18
drwxr-xr-x. 10 root root 203 Sep 11 22:54 kafka-manager-1.3.3.18-source
-rw-r--r--. 1 root root 79534088 Sep 11 23:01 kafka-manager-1.3.3.18.zip
drwxrwxr-x. 12 spark spark 232 Feb 17 2018 kibana-6.2.2-linux-x86_64
-rw-r--r--. 1 root root 9116 Apr 11 2016 mysql57-community-release-el7-8.noarch.rpm
drwxrwxr-x. 14 spark spark 252 Aug 6 20:23 nifi-1.7.1
drwxr-xr-x. 7 spark spark 67 Jul 17 22:54 node-8.9.1
drwxr-xr-x. 10 spark spark 4096 Jul 17 22:33 node-v8.9.1
drwxrwxr-x. 6 2000 2000 50 Apr 17 2014 scala-2.11.0
drwxrwxrwx. 13 hadoop hadoop 236 Sep 6 22:34 spark-2.2.1-bin-hadoop2.7
drwxrwxrwx. 12 spark spark 4096 Aug 8 00:23 zookeeper-3.4.12
[root@master opt]# cd kafka-manager-1.3.3.18
[root@master kafka-manager-1.3.3.18]# ls
bin conf lib README.md share
[root@master kafka-manager-1.3.3.18]# cd conf/
[root@master conf]# ls
application.conf consumer.properties logback.xml logger.xml routes
[root@master conf]# vim application.conf
# Secret key
# ~~~~~
# The secret key is used to secure cryptographics functions.
# If you deploy your application to several instances be sure to use the same key!
play.crypto.secret="^<csmm5Fx4d=r2HEX8pelM3iBkFVv?k[mc;IZE<_Qoq8EkX_/7@Zt6dP05Pzea3U"
play.crypto.secret=${?APPLICATION_SECRET}
# The application languages
# ~~~~~
play.i18n.langs=["en"]
play.http.requestHandler = "play.http.DefaultHttpRequestHandler"
play.http.context = "/"
play.application.loader=loader.KafkaManagerLoader
#kafka-manager.zkhosts="kafka-manager-zookeeper:2181"
kafka-manager.zkhosts="192.168.0.120:2181,192.168.0.121:2181,192.168.0.122:2181"
kafka-manager.zkhosts=${?ZK_HOSTS}
pinned-dispatcher.type="PinnedDispatcher"
pinned-dispatcher.executor="thread-pool-executor"
application.features=["KMClusterManagerFeature","KMTopicManagerFeature","KMPreferredReplicaElectionFeature","KMReassignPartitionsFea
ture"]
akka {
loggers = ["akka.event.slf4j.Slf4jLogger"]
loglevel = "INFO"
}
akka.logger-startup-timeout = 60s
basicAuthentication.enabled=false
basicAuthentication.enabled=${?KAFKA_MANAGER_AUTH_ENABLED}
basicAuthentication.username="admin"
basicAuthentication.username=${?KAFKA_MANAGER_USERNAME}
basicAuthentication.password="password"
basicAuthentication.password=${?KAFKA_MANAGER_PASSWORD}
basicAuthentication.realm="Kafka-Manager"
basicAuthentication.excluded=["/api/health"] # ping the health of your instance without authentification
kafka-manager.consumer.properties.file=${?CONSUMER_PROPERTIES_FILE}
~
~
~
~
~
~
~
~
~
"application.conf" 46L, 1682C written
5.启动服务
启动zk集群，kafka集群，再启动kafka-manager服务。
bin/kafka-manager 默认的端口是9000，可通过 -Dhttp.port，指定端口; -Dconfig.file=conf/application.conf指定配置文件:
[root@master conf]# cd .. [root@master kafka-manager-1.3.3.18]# ls bin conf lib README.md share [root@master kafka-manager-1.3.3.18]# [root@master kafka-manager-1.3.3.18]# nohup bin/kafka-manager -Dconfig.file=conf/application.conf -Dhttp.port=19093 &
jps查看：
[root@master spark]# jps 4596 QuorumPeerMain 4956 Kafka 4476 ProdServerStart 5116 Jps [root@master spark]#
WebUI查看：http://192.168.0.120:19093/ 出现如下界面则启动成功。
其他就是在ui上新建cluster，新建topic，查看topic等操作。
参考：
http://www.cnblogs.com/frankdeng/p/9584870.html
https://www.cnblogs.com/dadonggg/p/8205302.html