Spark java。木卫一。InvalidClassException:org。阿帕奇。火花不安全的类型。UTF8字符串；本地类不兼容

Question

我试图在远程群集上运行Spark应用程序，但遇到序列化错误。Scala和Spark版本是相同的。我被困在这一点上了。

spark shell-群集上的版本：

root@a913008dd071:/usr/local/spark-2.1.1# ./bin/spark-submit --version
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.1.1
      /_/

Using Scala version 2.11.8, OpenJDK 64-Bit Server VM, 1.8.0_131
Branch
Compiled by user jenkins on 2017-04-25T23:51:10Z
Revision
Url
Type --help for more information.

建筑sbt

import sbt.ExclusionRule

name := "hxfa"
version := "1.0"
scalaVersion := "2.11.8"

val elasticVersion = "5.4.1"

resolvers += "Spark Packages" at "https://dl.bintray.com/spark-packages/maven/"
resolvers += "Additional spark packages" at "https://dl.bintray.com/sbcd90/org.apache.spark"
resolvers += "Apache HBase" at "https://repository.apache.org/content/repositories/releases"
resolvers += "Thrift" at "http://people.apache.org/~rawson/repo/"
resolvers += "Spring Plugins" at "http://repo.spring.io/plugins-release/"

/* Dependencies */
libraryDependencies ++= Seq(
  // Framework and configuration
  "org.springframework.boot" % "spring-boot-starter-web" % "1.5.4.RELEASE",
  "org.hibernate" % "hibernate-validator" % "5.2.4.Final",

  /* Serializations */
  "com.fasterxml.jackson.core" % "jackson-core" % "2.8.7",
  "com.fasterxml.jackson.core" % "jackson-databind" % "2.8.7",
  "com.fasterxml.jackson.module" % "jackson-module-scala_2.11" % "2.8.7",
  "com.esotericsoftware" % "kryo" % "4.0.0",


  // Spark and utilities
  "org.apache.spark" %% "spark-core" % "2.1.0",
  "org.apache.spark" %% "spark-sql" % "2.1.0" ,
  "org.apache.spark" %% "spark-mllib" % "2.1.0" ,
  "graphframes" % "graphframes" % "0.5.0-spark2.1-s_2.11",


  // Spark connectors
  "org.elasticsearch" % "elasticsearch-spark-20_2.11" % elasticVersion,
  "org.mongodb.spark" % "mongo-spark-connector_2.11" % "2.0.0",


  //JDBC
  "mysql" % "mysql-connector-java" % "5.1.35",

  // HBase
  "org.apache.hbase" % "hbase" % "1.2.4",
  "org.apache.hbase" % "hbase-client" % "1.2.4",
  "org.apache.hbase" % "hbase-common" % "1.2.4",

  // OrientDB
  "com.orientechnologies" % "orientdb-graphdb" % "2.2.20"

).map(_.excludeAll(ExclusionRule("org.slf4j", "slf4j-log4j12"), ExclusionRule("log4j", "log4j")))

libraryDependencies ++= Seq(
  "org.apache.hbase" % "hbase-server" % "1.2.4"
).map(_.excludeAll(
  ExclusionRule("com.sun.jersey", "jersey-server"),
  ExclusionRule("tomcat"),
  ExclusionRule("log4j", "log4j")
))


/* Assembly     */

mainClass in assembly := Some("com.x.x.hello.app.HX")
assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false, includeDependency = false)

assemblyMergeStrategy in assembly := {
  case PathList("META-INF", xs@_*) => MergeStrategy.discard
  case x => MergeStrategy.first
}

堆栈跟踪：

Answer 1

spark submit——version显示其软件包的spark和scala版本，而不是您的系统的版本，而sbt使用的是您系统的scala版本。所以

请换衣服

  "org.apache.spark" %% "spark-core" % "2.1.0",
  "org.apache.spark" %% "spark-sql" % "2.1.0" ,
  "org.apache.spark" %% "spark-mllib" % "2.1.0" ,

到

  "org.apache.spark" % "spark-core_2.11" % "2.1.1",
  "org.apache.spark" % "spark-sql_2.11" % "2.1.1" ,
  "org.apache.spark" % "spark-mllib_2.11" % "2.1.1" ,

如果它没有帮助，请更新您的问题与您的系统scala版本，您是如何提交应用程序和您的远程机器的scala和火花版本。