Kafka结构化流媒体java.lang.NoClassDefFoundError

Question

我以前能够运行Kafka结构流编程。但是突然间，我所有的结构流python程序都失败了，出现了一个错误。我从Spark网站上拿了基本的Kafka结构流式编程，也以同样的错误失败。

spark-submit--packages org.apache.spark:spark-sql-kafka-0-102.11:2.2.0c:\users\ranjith.gangam\pycharmprojects\sparktest\structured_streaming.py

这是我从Spark github上取的代码

spark = SparkSession\
      .builder\
      .appName("StructuredKafkaWordCount")\
      .getOrCreate()

# Create DataSet representing the stream of input lines from kafka
lines = spark\
    .readStream\
    .format("kafka")\
    .option("kafka.bootstrap.servers", bootstrapServers)\
    .option(subscribeType, topics)\
    .load()\
    .selectExpr("CAST(value AS STRING)")

words = lines.select(
    # explode turns each item in an array into a separate row
    explode(
        split(lines.value, ' ')
    ).alias('word')
)

# Generate running word count
wordCounts = words.groupBy('word').count()
# Start running the query that prints the running counts to the console
query = wordCounts\
    .writeStream\
    .outputMode('complete')\
    .format('console')\
    .start()

query.awaitTermination()

Answer 1

你的方式是正确的，但不幸的是，Kafka0.10还不支持PySpark。正如你在火星-16534中所看到的。

到目前为止，对pySpark的唯一支持是Kafka0.8。因此，您可以迁移到Spark0.8或将代码更改为Scala。