在本地以Docker方式安装和运行Kafka

前言

最流行的两个Kafka的Docker镜像为：

• Bitmami/kafka (Github)
• wurstmeister/kafka (Github)

以下docker-compose.yaml 配置只适用于不需要安全控制和持久化数据的本地开发环境。

用Bitnami/kafka的镜像

version: "3"
services:
  zookeeper:
    image: 'bitnami/zookeeper:latest'
    ports:
      - '2181:2181'
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  kafka:
    image: 'bitnami/kafka:latest'
    ports:
      - '9092:9092'
    environment:
      - KAFKA_BROKER_ID=1
      - KAFKA_LISTENERS=PLAINTEXT://:9092
      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
    depends_on:
      - zookeeper

参见：

• https://github.com/bitnami/bitnami-docker-kafka

用wurstmeister/kafka的镜像

version: "3"
services:
  zookeeper:
    image: 'wurstmeister/zookeeper:latest'
    ports:
      - '2181:2181'
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  kafka:
    image: 'wurstmeister/kafka:latest'
    ports:
      - '9092:9092'
    environment:
      - KAFKA_BROKER_ID=1
      - KAFKA_LISTENERS=PLAINTEXT://:9092
      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
    depends_on:
      - zookeeper

参见：

• https://github.com/wurstmeister/kafka-docker

启动Kafka

# 创建一个新目录名为`kafka`
# 选用上面某一个`docker-compose.yaml`
mkdir -p kafka
cd kafka
vim docker-compose.yaml

# start
docker-compose up -d

# stop
docker-compose stop

测试创建主题并读/写消息

# check kafka containers
docker ps | grep kafka

# Create topic into `my-topic` topic
docker exec -it kafka_kafka_1 \
  kafka-topics.sh --create --bootstrap-server kafka:9092 --topic my-topic


# Write events
docker exec -it kafka_kafka_1 \
  kafka-console-producer.sh --bootstrap-server kafka:9092 --topic my-topic

# input message per each line, and press <enter> key
# press <ctrl> + <d> to complete writing messages and quit


# In a new terminal
# Read events from `my-topic` topic, and offset is from beginning
docker exec -it kafka_kafka_1 \
  kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic my-topic --from-beginning
  
# watch consumer logs to check if receive messages
# press <ctrl> + <c> to quit

参考文档

• Running Kafka locally with Docker

后记

如果是想在Kubernetes或OpenShfit上运行Kafka，则建议用Kafka Operator方式。

参见：

• Strimzi - http://strimzi.io/ - Apache Kafka Operator for Kubernetes and Openshift. Downloads and Helm Chart - https://github.com/strimzi/strimzi-kafka-operator/releases/latest