从Kafka消费，使用Kafka方法和火花流给出了不同的结果

Question

我正在尝试使用spark Streaming从Kafka中消耗一些数据。

我创造了2个工作岗位,

1. 一个简单的Kafka作业，它使用:

consumeFirstStringMessageFrom(topic)

{
  "data": {
    "type": "SA_LIST",
    "login": "[e-mail]",
    "updateDate": "2020-09-09T14:58:39.775Z",
    "content": [
      {
        "sku": "800633955",
        "status": "ACTIVE",
        "quantity": 1
      }
    ],
    "saCode": "E40056",
    "clientId": "30179801688090",
    "$setOnInsert": {
      "__v": 0
    }
  },
  "operation": "UPDATE",
  "type": "List"
}

val df = spark.readStream
      .format("kafka")
      .option("kafka.bootstrap.servers", kafkaConfig.broker)
      .option("subscribe", kafkaConfig.topic)
      .option("startingOffsets", kafkaConfig.startingOffsets)
      .load()

 df.writeStream
      .outputMode(OutputMode.Append())
      .format("console")
      .option("truncate", false)
      .trigger(Trigger.ProcessingTime("2 seconds"))
      .start().awaitTermination()

{
  "key": "I4NTY4NV9MSVNUX1dJU0hMSVNUIg==",
  "value": "eyJkYXRhIjp7InR5cGUiOiJXSVNITElTVCIsImxvZ2luIjoiZHJlYW1lcjJAeW9wbWFpbC5jb20iLCJ1cGRhdGVEYXRZSI6Ikxpc3QifQ==",
  "topic": "PLP_GLOBAL_QA",
  "partition": 0,
  "offset": 1826,
  "timestamp": "2020-09-10T16:09:08.606Z",
  "timestampType": 0
}

Answer 1

Spark流作业以序列化的形式向您显示数据，而您的Kafka使用者已经对其进行了反序列化。

根据Spark Structured Kafka integration guide，您不仅可以获得Kafka消息的关键和价值，还可以获得其他（元）信息。下面是您从Kafka获得的每条消息的模式：

Column      Type
key         binary
value       binary
topic       string
partition   int
offset      long
timestamp   timestamp
timestampType   int

如果您只想选择键和值，甚至只想选择值，您可以选择它们并将它们转换为人类可读的字符串：

[...]
  .load()
  .selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")
  .as[(String, String)]