为什么kafka producer（perf测试）会有如此低的吞吐量/高的延迟？

Question

我是Kafka的新手，正在运行一些性能测试。我正在运行一个由我的笔记本电脑和一个raspberry pi zero W（1 GHz、单核CPU、512 MB RAM、802.11n无线局域网）组成的2台机器集群。最终，pi将运行一个单独的生成器(java)，该生成器将二进制传感器数据（理想情况下是更小的记录，例如，10 kb，以最快的速度）发送给kafka，然后由消费者在我的笔记本电脑或另一个pi上读取这些数据。一个单独的应用程序稍后脱机处理数据。我只有一个主题，有一个分区，没有复制。

我目前正在尝试对producer性能进行基准测试/了解它的功能。这是来自kafka-producer-perf-test.sh的输出，显示有相当大的延迟，而吞吐量不是很大。我已经玩过这些设置（像linger.ms、batch.size、acks、吞吐量和记录大小，看起来记录大小越大，吞吐量就越高（直到750 kb，然后下降），但是延迟要么是差不多的，要么实际上是下降的。

我只能达到0.6MB/秒，这对于我正在开发的应用程序来说太慢了。当我降低记录大小时，吞吐量更差，为~0.10Mb/秒。因为我是新的Kafka，我想知道这是我应该期待的，当使用pi零w或是一些次优的设置/生产者属性，等等？

pi@raspberrypi:~/Kafka/kafka_2.12-2.4.0 $ bin/kafka-producer-perf-test.sh --topic perf_test --num-records 100 --thughput 1000000 --producer-props bootstrap.servers=192.168.0.170:9092,192.168.0.172:9092 --record-size 750000
OpenJDK Zero VM warning: G1 GC is disabled in this release.
2 records sent, 0.3 records/sec (0.25 MB/sec), 4439.5 ms avg latency, 4607.0 ms max latency.
5 records sent, 0.9 records/sec (0.62 MB/sec), 6909.0 ms avg latency, 9180.0 ms max latency.
5 records sent, 0.8 records/sec (0.58 MB/sec), 12783.4 ms avg latency, 15075.0 ms max latency.
5 records sent, 0.8 records/sec (0.59 MB/sec), 18587.0 ms avg latency, 21165.0 ms max latency.
5 records sent, 0.9 records/sec (0.62 MB/sec), 24551.2 ms avg latency, 26851.0 ms max latency.
4 records sent, 0.8 records/sec (0.55 MB/sec), 30048.0 ms avg latency, 31850.0 ms max latency.
5 records sent, 0.9 records/sec (0.62 MB/sec), 35323.4 ms avg latency, 37657.0 ms max latency.
5 records sent, 0.9 records/sec (0.63 MB/sec), 41030.4 ms avg latency, 43266.0 ms max latency.
5 records sent, 0.8 records/sec (0.59 MB/sec), 46736.6 ms avg latency, 49240.0 ms max latency.
5 records sent, 0.9 records/sec (0.62 MB/sec), 51716.2 ms avg latency, 53756.0 ms max latency.
5 records sent, 0.9 records/sec (0.63 MB/sec), 53765.8 ms avg latency, 53848.0 ms max latency.
5 records sent, 0.9 records/sec (0.61 MB/sec), 53506.2 ms avg latency, 53731.0 ms max latency.
4 records sent, 0.8 records/sec (0.55 MB/sec), 53638.3 ms avg latency, 54029.0 ms max latency.
5 records sent, 0.9 records/sec (0.62 MB/sec), 53690.4 ms avg latency, 54003.0 ms max latency.
5 records sent, 0.9 records/sec (0.64 MB/sec), 53371.6 ms avg latency, 53703.0 ms max latency.
5 records sent, 0.9 records/sec (0.62 MB/sec), 52922.0 ms avg latency, 53008.0 ms max latency.
4 records sent, 0.7 records/sec (0.53 MB/sec), 53774.3 ms avg latency, 53805.0 ms max latency.
5 records sent, 0.9 records/sec (0.63 MB/sec), 53788.6 ms avg latency, 53827.0 ms max latency.
5 records sent, 0.9 records/sec (0.64 MB/sec), 53317.2 ms avg latency, 53698.0 ms max latency.
5 records sent, 0.8 records/sec (0.60 MB/sec), 53373.2 ms avg latency, 53406.0 ms max latency.
5 records sent, 0.9 records/sec (0.63 MB/sec), 53441.8 ms avg latency, 53476.0 ms max latency.
100 records sent, 0.816993 records/sec (0.58 MB/sec), 42056.90 ms avg latency, 54029.00 ms max latency, 52990 ms 50th, 53805 ms 95th, 54029 ms 99th, 54029 ms 99.9th.

Answer 1

消息在发送给Kafaka之前是分批处理的。批处理中的消息数量越多，单次发送的消息数量就越多，吞吐量就越高。

如果要增加消息大小，则还必须增加batch.size，以便每个批分配更多的消息。

从您粘贴的代码中，没有设置Batch.size的属性，因此默认为16384字节(16kb)，这远远小于您的消息大小(750kb)，所以您看到0.3条记录/秒，以此类推。

在batch.size文档中

不会尝试批处理大于此大小的记录。

因此，您可以尝试增加batch.size以分配更多的消息，例如，增加到10MB左右，并检查。