Kafka消费者不接收消息

Question

我是Kafka的新手。我在网上读了很多关于Kafka制作人和Kafka消费者的说明。我成功地实现了前者，它可以向Kafka集群发送消息。然而，我没有完成后一个。请帮我解决这个问题。我看到我的问题像StackOverflow上的一些帖子，但我想更清楚地描述一下。我在虚拟盒子的Ubuntu服务器上运行Kafka和Zookeeper。使用1个Kafka集群和1个Zookeeper集群的最简单配置（几乎是默认配置）。

1.当我为生产者和消费者使用Kafka的命令行时，比如：

* Case 1: It works. I can see the word: Hello, World on the screen

$~/kafka/bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic helloKafka --partitions 1 --replication-factor 1.
$echo "Hello, World" | ~/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic TutorialTopic > /dev/null.
$~/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic TutorialTopic --from-beginning.

2、当我为消费者使用Kafka的制作人和命令行时，比如：

* Case 2: It works. I can see the messages which sent from the Producer on the screen

$~/kafka/bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic helloKafka --partitions 1 --replication-factor 1.
$~/kafka/bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic TutorialTopic --from-beginning.
$java -cp target/Kafka_Producer_Program-0.0.1-SNAPSHOT.jar AddLab_Producer

3、当我使用生产者和消费者时，比如：

* Case 3: Only Producer works perfectly. The Consumer runs but does not shows any messages. 

$~/kafka/bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic helloKafka --partitions 1 --replication-factor 1.
$java -cp target/Kafka_Producer_Program-0.0.1-SNAPSHOT.jar AddLab_Producer
$java -cp target/Kafka_Consumer_Program-0.0.1-SNAPSHOT.jar AddLab_Consumer

这是我的生产者和消费者代码。实际上，我是从Kafka的网站上抄的。

*制片人计划

import java.util.Properties;
import java.util.concurrent.ExecutionException;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.serialization.StringSerializer;
public class AddLab_Producer {
    public static void main(String args[]) throws InterruptedException, ExecutionException {
        Properties props = new Properties();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());

        KafkaProducer<String, String> producer = new KafkaProducer<String, String>(props);

        boolean sync = false;
        String topic = args[0];
        String key = "mykey";

        for (int i = 1; i <= 3; i++) {
            String value = args[1] + " " + i;
            ProducerRecord<String, String> producerRecord = new ProducerRecord<String, String>(topic, value);
            if (sync) {
                producer.send(producerRecord).get();
            } else {
                producer.send(producerRecord);
            }
        }
        producer.close();
    }
}

*消费者计划

import kafka.consumer.Consumer;
import kafka.consumer.ConsumerConfig;
import kafka.consumer.ConsumerIterator;
import kafka.consumer.KafkaStream;
import kafka.javaapi.consumer.ConsumerConnector;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Properties;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class AddLab_Consumer {

    public static class KafkaPartitionConsumer implements Runnable {

        private int tnum ;
        private KafkaStream kfs ;

        public KafkaPartitionConsumer(int id, KafkaStream ks) {
            tnum = id ;
            kfs = ks ;
        }   
        public void run() {
            // TODO Auto-generated method stub
            System.out.println("This is thread " + tnum) ;

            ConsumerIterator<byte[], byte[]> it = kfs.iterator();
                int i = 1 ;
                while (it.hasNext()) {
                    System.out.println(tnum + " " + i + ": " + new String(it.next().message()));
                    ++i ;
                }       
        }
    }

    public static class MultiKafka {    
        public void run() {
        }   
    }

    public static void main(String[] args) {

        Properties props = new Properties();
        props.put("zookeeper.connect", "localhost:2181");
        props.put("group.id", "mygroupid2");
        props.put("zookeeper.session.timeout.ms", "413");
        props.put("zookeeper.sync.time.ms", "203");
        props.put("auto.commit.interval.ms", "1000");

        ConsumerConfig cf = new ConsumerConfig(props) ;    
        ConsumerConnector consumer = Consumer.createJavaConsumerConnector(cf) ;      
        String topic = "mytopic" ;     
        Map<String, Integer> topicCountMap = new HashMap<String, Integer>();
        topicCountMap.put(topic, new Integer(1));
        Map<String, List<KafkaStream<byte[], byte[]>>> consumerMap = consumer.createMessageStreams(topicCountMap);
        List<KafkaStream<byte[], byte[]>> streams = consumerMap.get(topic);

        ExecutorService executor = Executors.newFixedThreadPool(1); ;

        int threadnum = 0 ;     
        for(KafkaStream<byte[],byte[]> stream  : streams) { 
            executor.execute(new KafkaPartitionConsumer(threadnum,stream));
            ++threadnum ;
        }
    }
}

*我的POM. xml文件

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>HelloJava</groupId>
    <artifactId>HelloJava</artifactId>
    <version>0.0.1-SNAPSHOT</version>

    <dependencies>
        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>kafka_2.10</artifactId>
            <version>0.9.0.0</version>
        </dependency>

        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>3.8.1</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <sourceDirectory>src</sourceDirectory>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>1.4</version>
                <configuration>
                    <createDependencyReducedPom>true</createDependencyReducedPom>
                    <filters>
                        <filter>
                            <artifact>*:*</artifact>
                            <excludes>
                                <exclude>META-INF/*.SF</exclude>
                                <exclude>META-INF/*.DSA</exclude>
                                <exclude>META-INF/*.RSA</exclude>
                            </excludes>
                        </filter>
                    </filters>
                </configuration>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

我非常感谢你的帮助。非常感谢你。

消费者屏幕。它似乎在运行，但无法接收来自生产者的任何消息

Answer 1

我也遇到过和你一样的问题。经过长时间的尝试，这里有答案。

您可以选择两种类型的Kafka新消费者api。

库苏默。分配（…）

消费者订阅（…）

使用如下：

    // set these properites or you should run consumer first than run producer
    props.put("enable.auto.commit", "false");
    props.put("auto.offset.reset", "earliest");

    KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);

    boolean assign = false;
    if(assign) {
        TopicPartition tp = new TopicPartition(topic, 0);
        List<TopicPartition> tps = Arrays.asList(tp);
        consumer.assign(tps);
        consumer.seekToBeginning(tps);
    }else {
        consumer.subscribe(Arrays.asList(topic));
    }

http://kafka.apache.org/documentation.html#newconsumerconfigs

如果您使用旧的消费者api，则属性配置几乎是相同的。如果您想查看在消费者消费之前生成的消息，请记住添加以下两个代码：

props.put("enable.auto.commit", "false");
props.put("auto.offset.reset", "earliest");

希望这能帮助其他人。