登录
当前位置： 首页 > 知识库问答 >
问题：

星火从未停止第一批加工

索瀚海
2023-03-14

我正在尝试运行我在github上找到的一个应用程序，这个应用程序是：https://github.com/csirt-mu/aida-framework

callForeachRDD at NativeMethodAccessorImpl.java:0
org.apache.spark.streaming.api.python.PythonDStream.callForeachRDD(PythonDStream.scala)
    sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    java.lang.reflect.Method.invoke(Method.java:498)
    py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    py4j.Gateway.invoke(Gateway.java:282)
    py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    py4j.commands.CallCommand.execute(CallCommand.java:79)
    py4j.GatewayConnection.run(GatewayConnection.java:238)
    java.lang.Thread.run(Thread.java:748)

编辑：我注意到当进程启动时会记录错误。我现在才意识到这一点，因为这个过程不会停止。错误有：

May 11 18:13:04 aida bash[5323]: 20/05/11 18:13:04 ERROR Utils: Uncaught exception in thread stdout writer for python3
May 11 18:13:04 aida bash[5323]: java.lang.NoSuchMethodError: net.jpountz.lz4.LZ4BlockInputStream.<init>(Ljava/io/InputStream;Z)V
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.io.LZ4CompressionCodec.compressedInputStream(CompressionCodec.scala:122)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.serializer.SerializerManager.wrapForCompression(SerializerManager.scala:163)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.serializer.SerializerManager.dataDeserializeStream(SerializerManager.scala:209)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.storage.BlockManager.getLocalValues(BlockManager.scala:596)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:886)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.RDD.iterator(RDD.scala:286)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute$1.apply(PartitionerAwareUnionRDD.scala:100)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute$1.apply(PartitionerAwareUnionRDD.scala:99)
May 11 18:13:04 aida bash[5323]:         at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:435)
May 11 18:13:04 aida bash[5323]:         at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:441)
May 11 18:13:04 aida bash[5323]:         at scala.collection.Iterator$class.foreach(Iterator.scala:891)
May 11 18:13:04 aida bash[5323]:         at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:224)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.PythonRunner$$anon$2.writeIteratorToStream(PythonRunner.scala:557)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.BasePythonRunner$WriterThread$$anonfun$run$1.apply(PythonRunner.scala:345)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1945)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:194)
May 11 18:13:04 aida bash[5323]: Exception in thread "stdout writer for python3" java.lang.NoSuchMethodError: net.jpountz.lz4.LZ4BlockInputStream.<init>(Ljava/io/InputStream;Z)V
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.io.LZ4CompressionCodec.compressedInputStream(CompressionCodec.scala:122)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.serializer.SerializerManager.wrapForCompression(SerializerManager.scala:163)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.serializer.SerializerManager.dataDeserializeStream(SerializerManager.scala:209)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.storage.BlockManager.getLocalValues(BlockManager.scala:596)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:886)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.RDD.iterator(RDD.scala:286)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute$1.apply(PartitionerAwareUnionRDD.scala:100)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute$1.apply(PartitionerAwareUnionRDD.scala:99)
May 11 18:13:04 aida bash[5323]:         at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:435)
May 11 18:13:04 aida bash[5323]:         at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:441)
May 11 18:13:04 aida bash[5323]:         at scala.collection.Iterator$class.foreach(Iterator.scala:891)
May 11 18:13:04 aida bash[5323]:         at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:224)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.PythonRunner$$anon$2.writeIteratorToStream(PythonRunner.scala:557)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.BasePythonRunner$WriterThread$$anonfun$run$1.apply(PythonRunner.scala:345)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1945)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:194)
May 11 18:13:04 aida bash[5323]: 20/05/11 18:13:04 ERROR Utils: Uncaught exception in thread stdout writer for python3
May 11 18:13:04 aida bash[5323]: java.lang.NoSuchMethodError: net.jpountz.lz4.LZ4BlockInputStream.<init>(Ljava/io/InputStream;Z)V
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.io.LZ4CompressionCodec.compressedInputStream(CompressionCodec.scala:122)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.serializer.SerializerManager.wrapForCompression(SerializerManager.scala:163)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.serializer.SerializerManager.dataDeserializeStream(SerializerManager.scala:209)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.storage.BlockManager.getLocalValues(BlockManager.scala:596)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:886)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.RDD.iterator(RDD.scala:286)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute$1.apply(PartitionerAwareUnionRDD.scala:100)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute$1.apply(PartitionerAwareUnionRDD.scala:99)
May 11 18:13:04 aida bash[5323]:         at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:435)
May 11 18:13:04 aida bash[5323]:         at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:441)
May 11 18:13:04 aida bash[5323]:         at scala.collection.Iterator$class.foreach(Iterator.scala:891)
May 11 18:13:04 aida bash[5323]:         at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:224)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.PythonRunner$$anon$2.writeIteratorToStream(PythonRunner.scala:557)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.BasePythonRunner$WriterThread$$anonfun$run$1.apply(PythonRunner.scala:345)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1945)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:194)
May 11 18:13:04 aida bash[5323]: Exception in thread "stdout writer for python3" java.lang.NoSuchMethodError: net.jpountz.lz4.LZ4BlockInputStream.<init>(Ljava/io/InputStream;Z)V
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.io.LZ4CompressionCodec.compressedInputStream(CompressionCodec.scala:122)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.serializer.SerializerManager.wrapForCompression(SerializerManager.scala:163)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.serializer.SerializerManager.dataDeserializeStream(SerializerManager.scala:209)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.storage.BlockManager.getLocalValues(BlockManager.scala:596)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:886)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.RDD.iterator(RDD.scala:286)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute$1.apply(PartitionerAwareUnionRDD.scala:100)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute$1.apply(PartitionerAwareUnionRDD.scala:99)
May 11 18:13:04 aida bash[5323]:         at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:435)
May 11 18:13:04 aida bash[5323]:         at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:441)
May 11 18:13:04 aida bash[5323]:         at scala.collection.Iterator$class.foreach(Iterator.scala:891)
May 11 18:13:04 aida bash[5323]:         at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:224)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.PythonRunner$$anon$2.writeIteratorToStream(PythonRunner.scala:557)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.BasePythonRunner$WriterThread$$anonfun$run$1.apply(PythonRunner.scala:345)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1945)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:194)
May 11 18:13:04 aida bash[5323]: 20/05/11 18:13:04 ERROR Utils: Uncaught exception in thread stdout writer for python3
May 11 18:13:04 aida bash[5323]: java.lang.NoSuchMethodError: net.jpountz.lz4.LZ4BlockInputStream.<init>(Ljava/io/InputStream;Z)V
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.io.LZ4CompressionCodec.compressedInputStream(CompressionCodec.scala:122)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.serializer.SerializerManager.wrapForCompression(SerializerManager.scala:163)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.serializer.SerializerManager.dataDeserializeStream(SerializerManager.scala:209)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.storage.BlockManager.getLocalValues(BlockManager.scala:596)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:886)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.RDD.iterator(RDD.scala:286)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute$1.apply(PartitionerAwareUnionRDD.scala:100)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute$1.apply(PartitionerAwareUnionRDD.scala:99)
May 11 18:13:04 aida bash[5323]:         at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:435)
May 11 18:13:04 aida bash[5323]:         at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:441)
May 11 18:13:04 aida bash[5323]:         at scala.collection.Iterator$class.foreach(Iterator.scala:891)
May 11 18:13:04 aida bash[5323]:         at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:224)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.PythonRunner$$anon$2.writeIteratorToStream(PythonRunner.scala:557)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.BasePythonRunner$WriterThread$$anonfun$run$1.apply(PythonRunner.scala:345)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1945)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:194)
May 11 18:13:04 aida bash[5323]: Exception in thread "stdout writer for python3" java.lang.NoSuchMethodError: net.jpountz.lz4.LZ4BlockInputStream.<init>(Ljava/io/InputStream;Z)V
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.io.LZ4CompressionCodec.compressedInputStream(CompressionCodec.scala:122)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.serializer.SerializerManager.wrapForCompression(SerializerManager.scala:163)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.serializer.SerializerManager.dataDeserializeStream(SerializerManager.scala:209)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.storage.BlockManager.getLocalValues(BlockManager.scala:596)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:886)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.RDD.iterator(RDD.scala:286)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute$1.apply(PartitionerAwareUnionRDD.scala:100)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.rdd.PartitionerAwareUnionRDD$$anonfun$compute$1.apply(PartitionerAwareUnionRDD.scala:99)
May 11 18:13:04 aida bash[5323]:         at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:435)
May 11 18:13:04 aida bash[5323]:         at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:441)
May 11 18:13:04 aida bash[5323]:         at scala.collection.Iterator$class.foreach(Iterator.scala:891)
May 11 18:13:04 aida bash[5323]:         at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.PythonRDD$.writeIteratorToStream(PythonRDD.scala:224)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.PythonRunner$$anon$2.writeIteratorToStream(PythonRunner.scala:557)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.BasePythonRunner$WriterThread$$anonfun$run$1.apply(PythonRunner.scala:345)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1945)
May 11 18:13:04 aida bash[5323]:         at org.apache.spark.api.python.BasePythonRunner$WriterThread.run(PythonRunner.scala:194)

有人能帮我解决这些错误吗？

共有1个答案

谢昊乾
2023-03-14

Kafka有一个矛盾的依赖与火花罐子...所以要么不使用lz压缩，使用快速压缩，它将工作

或者按照这里的答案来解决冲突的jar。

 类似资料：
  • 1.12.1 第一版(停止支持)

    功能和命令 创建应用 - create 更新应用 - update 删除应用 - delete 修改应用的实例数 - replica 帮助 - help 帮助 输入错误或命令中使用 -h或help，会出现帮助 创建应用 自动部署工具中，并且强制要求在每个操作后第一个参数是区域名称 ../deploy create cnbj ..... 创建应用的关键字是 -c(或-create),后面可以直接跟创

  • 烽火星空一面

    发发面经攒攒人品😄 电话面 闲聊： 自我介绍 怎么学习前端的？学习的路线是什么 未来的发展方向 1.HTML语义化 2.盒模型 3.元素隐藏的方法 4.flex 5.promise 6.react中key的作用 7.react中如何创建元素 8.react通信方式 9.闭包?场景 10.forEach和map的区别 11.for循环和forEach的区别，性能方面 12.http状态码 13.怎

  • Cosmos Changefeed火花流随机停止

    我有一个 Spark 流式处理作业，它读取 Cosmos 更改源数据，如下所示，在具有 DBR 8.2 的数据砖集群中运行。 虽然作业正常工作，但偶尔，流会突然停止，并且在log4j输出中出现以下循环。重新启动作业将处理“待办事项”中的所有数据。以前有人经历过这样的事情吗？我不确定是什么原因造成的。有什么想法吗？

  • 烽火星空 Java 一面

    #秋招# #面经# #烽火星空# 时间：9.22 14：00 1、自我介绍 2、Java访问权限 3、Java数据类型 4、说一下HashMap 5、说一下线程安全的集合 6、说一下hashMap扩容机制 7、创建线程的几种方式 8、说一下sql注入，怎么解决 9、说一下mysql的外连接 10、介绍一下SpringMVC的执行过程 ……还有几个忘记了 11、项目相关 redis、nginx、li

  • 停止从Selenium Webdriver加载浏览器

    问题内容： 我的Selenium Webdriver转到页面并等待该页面完成加载。如果30秒过去，它将超时并且脚本失败。 无论如何，是否要让网络驱动程序在30秒后停止页面加载（例如按浏览器中的“ x”）？这样可以防止驱动程序超时。 我正在使用Chromedriver。 问题答案： 这就是我遇到此问题的方式。在chrome支持之前，我将使用这种方式。 我在Chrome中安装了一个名为“停止加载”的扩

  • 停止从selenium webdrive加载浏览器

    我的selenium webdriver转到一个页面，等待该页面完成加载。如果超过30秒，则超时，脚本失败。 是否需要webdriver在30秒后停止页面加载（如按浏览器上的“x”）？这将防止我的司机超时。 我正在使用ChromeDrive。

相关问答

星星之火中的排序Spring Boot Spring批处理分区作业完成后未停止如何停止和恢复一个Spring批量作业如何从命令行停止Spring-boot批处理作业？Hadoop后台进程未停止

相关文章

加载彗星/服务器推送iframe时，停止浏览器的“厄运之战”从命令行停止node.js程序获取一个月的第一个星期一MySQL DAYOFWEEK（）-我的星期从星期一开始如何停止同一goroutine的multilpe之一

相关阅读

南京烽火星空测开岗一面诺瓦星云提前批一面西安烽火星空 C++诺瓦星云提前批Java一面诺瓦星云-正式批测试一面

相关工具

GPS火星坐标转换异星工厂NG 星系摘星船NG星系

相关文档

第一性原理我的第一本 Laravel 学习集星图开放平台 API 文档编程小白的第一本 Python 入门书具有高级安全性的 Windows 防火墙
快捷导航： 新手教程 算法原理 架构设计 Java进阶 数据库进阶 大厂专栏 面试经验 编程笔记 编程问答 所有专题 文档资料 工具软件 电子书籍 小牛导航
在线工具： 房贷计算器 个税计算器 Linux命令查询 Json格式化 正则表达式 颜色转换 AES加解密 SHA1加密 MD5加密 毒鸡汤 字数统计 随机密码生成 进制转换 Base64编解码 励志句子
Copyright © 2019-2024 小牛知识库@xnip.cn. All Rights Reserved.