登录
当前位置： 首页 > 知识库问答 >
问题：

Intellij IDEA：火花代码运行导致java.lang.VerifyError

龙逸清
2023-03-14

在IntelliJ IDEA中，我试图用spark代码执行一个Java文件，这将产生Java。验证错误。

StackTrace如下所示：

错误执行器：java阶段1.0（TID 2）中任务0.0中出现异常。lang.VerifyError：（class:org/apache/spark/sql/catalyst/expressions/GeneratedClass$SpecificOrdering，方法：tryCompare签名：（Ljava/lang/Object；Ljava/lang/Object；）Lscala/部分；）组织的函数返回类型错误。阿帕奇。火花sql。催化剂表达。GeneratedClass。在组织中生成（未知源）。阿帕奇。火花sql。催化剂表达。GeneratedClass。在组织中生成（未知源）。阿帕奇。火花sql。催化剂表达。codegen。GenerateOrdering美元。在组织中创建（GenerateOrdering.scala:138）。阿帕奇。火花sql。催化剂表达。codegen。GenerateOrdering美元。在组织中创建（GenerateOrdering.scala:37）。阿帕奇。火花sql。催化剂表达。codegen。代码生成器。在org上生成（CodeGenerator.scala:588）。阿帕奇。火花sql。催化剂表达。codegen。代码生成器。生成（CodeGenerator。scala:585）

我对错误的理解-客户端Scala（IntelliJ）和Spark 1.6.1的Scala版本中存在一些版本冲突。

前一天，同样的代码工作正常，但现在出现了上述错误。

令人惊讶的是，相同的代码在Eclipse上执行良好。

完整堆栈跟踪：

16/08/20 22:25:39 ERROR Executor: Exception in task 0.0 in stage 1.0 (TID 2)
java.lang.VerifyError: (class: org/apache/spark/sql/catalyst/expressions/GeneratedClass$SpecificOrdering, method: tryCompare signature: (Ljava/lang/Object;Ljava/lang/Object;)Lscala/Some;) Wrong return type in function
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass.generate(Unknown Source)
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass.generate(Unknown Source)
    at org.apache.spark.sql.catalyst.expressions.codegen.GenerateOrdering$.create(GenerateOrdering.scala:138)
    at org.apache.spark.sql.catalyst.expressions.codegen.GenerateOrdering$.create(GenerateOrdering.scala:37)
    at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator.generate(CodeGenerator.scala:588)
    at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator.generate(CodeGenerator.scala:585)
    at org.apache.spark.sql.execution.SparkPlan.newOrdering(SparkPlan.scala:257)
    at org.apache.spark.sql.execution.Sort$$anonfun$1.apply(Sort.scala:65)
    at org.apache.spark.sql.execution.Sort$$anonfun$1.apply(Sort.scala:64)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$21.apply(RDD.scala:728)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$21.apply(RDD.scala:728)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
    at org.apache.spark.scheduler.Task.run(Task.scala:89)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
16/08/20 22:25:39 ERROR Executor: Exception in task 1.0 in stage 1.0 (TID 3)
java.lang.VerifyError: (class: org/apache/spark/sql/catalyst/expressions/GeneratedClass$SpecificOrdering, method: tryCompare signature: (Ljava/lang/Object;Ljava/lang/Object;)Lscala/Some;) Wrong return type in function
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass.generate(Unknown Source)
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass.generate(Unknown Source)
    at org.apache.spark.sql.catalyst.expressions.codegen.GenerateOrdering$.create(GenerateOrdering.scala:138)
    at org.apache.spark.sql.catalyst.expressions.codegen.GenerateOrdering$.create(GenerateOrdering.scala:37)
    at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator.generate(CodeGenerator.scala:588)
    at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator.generate(CodeGenerator.scala:585)
    at org.apache.spark.sql.execution.SparkPlan.newOrdering(SparkPlan.scala:257)
    at org.apache.spark.sql.execution.Sort$$anonfun$1.apply(Sort.scala:65)
    at org.apache.spark.sql.execution.Sort$$anonfun$1.apply(Sort.scala:64)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$21.apply(RDD.scala:728)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$21.apply(RDD.scala:728)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
    at org.apache.spark.scheduler.Task.run(Task.scala:89)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
16/08/20 22:25:39 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[Executor task launch worker-0,5,main]
java.lang.VerifyError: (class: org/apache/spark/sql/catalyst/expressions/GeneratedClass$SpecificOrdering, method: tryCompare signature: (Ljava/lang/Object;Ljava/lang/Object;)Lscala/Some;) Wrong return type in function
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass.generate(Unknown Source)
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass.generate(Unknown Source)
    at org.apache.spark.sql.catalyst.expressions.codegen.GenerateOrdering$.create(GenerateOrdering.scala:138)
    at org.apache.spark.sql.catalyst.expressions.codegen.GenerateOrdering$.create(GenerateOrdering.scala:37)
    at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator.generate(CodeGenerator.scala:588)
    at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator.generate(CodeGenerator.scala:585)
    at org.apache.spark.sql.execution.SparkPlan.newOrdering(SparkPlan.scala:257)
    at org.apache.spark.sql.execution.Sort$$anonfun$1.apply(Sort.scala:65)
    at org.apache.spark.sql.execution.Sort$$anonfun$1.apply(Sort.scala:64)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$21.apply(RDD.scala:728)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$21.apply(RDD.scala:728)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
    at org.apache.spark.scheduler.Task.run(Task.scala:89)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
16/08/20 22:25:39 ERROR SparkUncaughtExceptionHandler: [Container in shutdown] Uncaught exception in thread Thread[Executor task launch worker-1,5,main]
java.lang.VerifyError: (class: org/apache/spark/sql/catalyst/expressions/GeneratedClass$SpecificOrdering, method: tryCompare signature: (Ljava/lang/Object;Ljava/lang/Object;)Lscala/Some;) Wrong return type in function
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass.generate(Unknown Source)
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass.generate(Unknown Source)
    at org.apache.spark.sql.catalyst.expressions.codegen.GenerateOrdering$.create(GenerateOrdering.scala:138)
    at org.apache.spark.sql.catalyst.expressions.codegen.GenerateOrdering$.create(GenerateOrdering.scala:37)
    at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator.generate(CodeGenerator.scala:588)
    at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator.generate(CodeGenerator.scala:585)
    at org.apache.spark.sql.execution.SparkPlan.newOrdering(SparkPlan.scala:257)
    at org.apache.spark.sql.execution.Sort$$anonfun$1.apply(Sort.scala:65)
    at org.apache.spark.sql.execution.Sort$$anonfun$1.apply(Sort.scala:64)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$21.apply(RDD.scala:728)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$21.apply(RDD.scala:728)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
    at org.apache.spark.scheduler.Task.run(Task.scala:89)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
16/08/20 22:25:39 INFO SparkContext: Invoking stop() from shutdown hook
16/08/20 22:25:39 WARN TaskSetManager: Lost task 1.0 in stage 1.0 (TID 3, localhost): java.lang.VerifyError: (class: org/apache/spark/sql/catalyst/expressions/GeneratedClass$SpecificOrdering, method: tryCompare signature: (Ljava/lang/Object;Ljava/lang/Object;)Lscala/Some;) Wrong return type in function
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass.generate(Unknown Source)
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass.generate(Unknown Source)
    at org.apache.spark.sql.catalyst.expressions.codegen.GenerateOrdering$.create(GenerateOrdering.scala:138)
    at org.apache.spark.sql.catalyst.expressions.codegen.GenerateOrdering$.create(GenerateOrdering.scala:37)
    at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator.generate(CodeGenerator.scala:588)
    at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator.generate(CodeGenerator.scala:585)
    at org.apache.spark.sql.execution.SparkPlan.newOrdering(SparkPlan.scala:257)
    at org.apache.spark.sql.execution.Sort$$anonfun$1.apply(Sort.scala:65)
    at org.apache.spark.sql.execution.Sort$$anonfun$1.apply(Sort.scala:64)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$21.apply(RDD.scala:728)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$21.apply(RDD.scala:728)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
    at org.apache.spark.scheduler.Task.run(Task.scala:89)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

16/08/20 22:25:39 ERROR TaskSetManager: Task 1 in stage 1.0 failed 1 times; aborting job
16/08/20 22:25:39 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 
16/08/20 22:25:39 INFO TaskSetManager: Lost task 0.0 in stage 1.0 (TID 2) on executor localhost: java.lang.VerifyError ((class: org/apache/spark/sql/catalyst/expressions/GeneratedClass$SpecificOrdering, method: tryCompare signature: (Ljava/lang/Object;Ljava/lang/Object;)Lscala/Some;) Wrong return type in function) [duplicate 1]
16/08/20 22:25:39 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool 
16/08/20 22:25:39 INFO TaskSchedulerImpl: Cancelling stage 1
16/08/20 22:25:39 INFO DAGScheduler: ShuffleMapStage 1 (map at LinearRegression.scala:161) failed in 0.406 s
16/08/20 22:25:39 INFO DAGScheduler: Job 1 failed: first at LinearRegression.scala:163, took 0.437687 s

org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 1.0 failed 1 times, most recent failure: Lost task 1.0 in stage 1.0 (TID 3, localhost): java.lang.VerifyError: (class: org/apache/spark/sql/catalyst/expressions/GeneratedClass$SpecificOrdering, method: tryCompare signature: (Ljava/lang/Object;Ljava/lang/Object;)Lscala/Some;) Wrong return type in function
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass.generate(Unknown Source)
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass.generate(Unknown Source)
    at org.apache.spark.sql.catalyst.expressions.codegen.GenerateOrdering$.create(GenerateOrdering.scala:138)
    at org.apache.spark.sql.catalyst.expressions.codegen.GenerateOrdering$.create(GenerateOrdering.scala:37)
    at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator.generate(CodeGenerator.scala:588)
    at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator.generate(CodeGenerator.scala:585)
    at org.apache.spark.sql.execution.SparkPlan.newOrdering(SparkPlan.scala:257)
    at org.apache.spark.sql.execution.Sort$$anonfun$1.apply(Sort.scala:65)
    at org.apache.spark.sql.execution.Sort$$anonfun$1.apply(Sort.scala:64)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$21.apply(RDD.scala:728)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$21.apply(RDD.scala:728)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
    at org.apache.spark.scheduler.Task.run(Task.scala:89)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

Driver stacktrace:

    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1431)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1419)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1418)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1418)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)
    at scala.Option.foreach(Option.scala:236)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1640)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1599)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1588)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1832)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1845)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1858)
    at org.apache.spark.rdd.RDD$$anonfun$take$1.apply(RDD.scala:1328)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
    at org.apache.spark.rdd.RDD.take(RDD.scala:1302)
    at org.apache.spark.rdd.RDD$$anonfun$first$1.apply(RDD.scala:1342)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
    at org.apache.spark.rdd.RDD.first(RDD.scala:1341)
    at org.apache.spark.ml.regression.LinearRegression.train(LinearRegression.scala:163)
    at org.apache.spark.ml.regression.LinearRegression.train(LinearRegression.scala:67)
    at org.apache.spark.ml.Predictor.fit(Predictor.scala:90)
    at com.datarpm.insights.ensemble.scorer.PipelineEnsembleScorerTest.regressionTests(PipelineEnsembleScorerTest.java:73)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
    at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:69)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:48)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:292)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
    at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:262)
    at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:84)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
Caused by: java.lang.VerifyError: (class: org/apache/spark/sql/catalyst/expressions/GeneratedClass$SpecificOrdering, method: tryCompare signature: (Ljava/lang/Object;Ljava/lang/Object;)Lscala/Some;) Wrong return type in function
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass.generate(Unknown Source)
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass.generate(Unknown Source)
    at org.apache.spark.sql.catalyst.expressions.codegen.GenerateOrdering$.create(GenerateOrdering.scala:138)
    at org.apache.spark.sql.catalyst.expressions.codegen.GenerateOrdering$.create(GenerateOrdering.scala:37)
    at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator.generate(CodeGenerator.scala:588)
    at org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator.generate(CodeGenerator.scala:585)
    at org.apache.spark.sql.execution.SparkPlan.newOrdering(SparkPlan.scala:257)
    at org.apache.spark.sql.execution.Sort$$anonfun$1.apply(Sort.scala:65)
    at org.apache.spark.sql.execution.Sort$$anonfun$1.apply(Sort.scala:64)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$21.apply(RDD.scala:728)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$21.apply(RDD.scala:728)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
    at org.apache.spark.scheduler.Task.run(Task.scala:89)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

共有2个答案

姬振濂
2023-03-14

在我的例子中，问题是冲突的jar。
我使用较旧的Scala版本2.11.4编译了一个内部项目的依赖jar。但是Spark是用2.11.8编译的。

通过重新编译所有依赖JAR以使用相同的Java和Scala目标版本，解决了我的问题。

苏硕
2023-03-14

我也有同样的问题。根本原因是cascading和spark catalyst都有组织。科德豪斯。janino依赖性。因此，spark使用了意外的janino lib版本请参见mvn依赖项：tree-Dverbose命令输出

[INFO] +- org.apache.spark:spark-sql_2.11:jar:2.0.0:compile
[INFO] |  +- org.apache.spark:spark-catalyst_2.11:jar:2.0.0:compile
[INFO] |  |  +- (org.codehaus.janino:janino:jar:2.7.8:compile - omitted for conflict with 2.7.5)    
[INFO] +- cascading:cascading-core:jar:2.6.1:compile
[INFO] |  \- org.codehaus.janino:janino:jar:2.7.5:compile
[INFO] |     \- org.codehaus.janino:commons-compiler:jar:2.7.5:compile
[INFO] +- cascading:cascading-core:jar:tests:2.6.1:compile
[INFO] |  \- (org.codehaus.janino:janino:jar:2.7.5:compile - omitted for duplicate)
 类似资料：
  • 火花：多个火花-并行提交

    一些脚本在工作时什么也不做，当我手动运行它们时，其中一个失败了，出现了以下消息： 错误SparkUI：未能绑定SparkUI java.net.bindexception：地址已在使用：服务“SparkUI”在重试16次后失败！ 所以我想知道是否有一种特定的方法来并行运行脚本？

  • 如何优化火花sql并行运行

    我是spark新手，有一个简单的spark应用程序，使用spark SQL/hiveContext： 从hive表中选择数据（10亿行） 做一些过滤，聚合，包括row_number窗口函数来选择第一行，分组，计数（）和最大（）等。 将结果写入HBase（数亿行） 我提交的作业运行它在纱线集群（100个执行者），它很慢，当我在火花UI中查看DAG可视化时，似乎只有蜂巢表扫描任务并行运行，其余的步骤#

  • 火花行到JSON

    我想从Spark v.1.6（使用scala）数据帧创建一个JSON。我知道有一个简单的解决方案，就是做。 但是，我的问题看起来有点不同。例如，考虑具有以下列的数据帧： 我想在最后有一个数据帧 其中C是包含、、的JSON。不幸的是，我在编译时不知道数据框是什么样子的（除了始终“固定”的列和）。 至于我需要这个的原因：我使用Protobuf发送结果。不幸的是，我的数据帧有时有比预期更多的列，我仍然会

  • 火花码给出错误

    我在运行下面我写的SPARK代码时出错了。我试图根据键找到所有向量的总和。每个输入行以键（整数）开始，然后是127个浮点数，这是一个具有127个维度的单个向量，即每一行以键和向量开始。

  • 火花运行错误java.lang.NoClassDefFoundError： org/codehaus/jackson/annotate/JsonClass

    自从有人提到Spark-jackson冲突问题以来，我使用mvn版本重建了Spark：使用最新版本-Dincludes=org。科德豪斯。jackson:jackson core asl mvn版本：使用最新版本-Dincludes=org。科德豪斯。jackson：jackson mapper asl 因此，JAR已更新为1.9。但我仍然有错误

  • 如何运行导致框架运行时错误的iOS应用“代码签名无效”

    问题内容： 每次我尝试运行此应用程序时，它都会生成OK并安装，但是一旦运行，该应用程序就会中断并显示以下错误消息： 我在这里和App Developer论坛上都看到过许多类似的帖子，但都没有人提供解释或有效的修复方法。 我试过了： 清洁和建筑 重新启动XCode 重新启动Mac 完全重置钥匙串 验证所有钥匙串证书信任设置都在“使用系统默认值”上 恢复出厂设置我的Mac 我的iOS设备似乎有问题。该

相关问答

火花运行错误java.lang.NoClassDefFoundError： org/codehaus/jackson/annotate/JsonClass火花工和执行器芯火花访问行对象值joda DateTime格式导致火花RDD函数中的空指针错误Switch语句导致重复的代码

相关文章

如何使用Java代码运行Java代码？在Java代码中运行liquibase在Vim中运行Python代码什么情况会导致 kafka 运行变慢？PHPUnit代码覆盖率生成导致内存耗尽

相关阅读

致远互联低代码外包千人厂前端面经？8.20雷火游戏客户端笔试 过了2.9个题 贴下代码贝壳花桥 培训运营面经字节-火上视频运营面经Bootstrap 代码

相关工具

妙码生花 - BuildAdmin莲花野火IMPHP代码生成器Google Glass 模板代码

相关文档

代码整洁的 JavaScript追求代码质量JavaScript 秘密花园Airbnb 的 Ruby 代码风格指南uCore OS 实验指导书和源码网址
快捷导航： 新手教程 算法原理 架构设计 Java进阶 数据库进阶 大厂专栏 面试经验 编程笔记 编程问答 所有专题 文档资料 工具软件 电子书籍 小牛导航
在线工具： 房贷计算器 个税计算器 Linux命令查询 Json格式化 正则表达式 颜色转换 AES加解密 SHA1加密 MD5加密 毒鸡汤 字数统计 随机密码生成 进制转换 Base64编解码 励志句子
Copyright © 2019-2024 小牛知识库@xnip.cn. All Rights Reserved.