使用Apache火花1.6使用共享配置2和共享beanutils-1.9

Question

我的应用程序正在使用commons-configuration2和commons-beanutils1。9，但当我尝试将我的应用程序JAR用于spark流媒体作业时，它引发了以下异常。

java.lang.NoSuchmethod odError：org.apache.commons.beanutils.Properties tyUtilsBean.addBean内向者（Lorg/apache/comons/beanutils/Bean内向者；）V在org.apache.commons.configuration2.beanutils.BeanHelper.initBeanUtilsBean（BeanHelper.java:631）在org.apache.commons.configuration2.beanutils.BeanHelper。（BeanHelper.java:89）java.lang.Class.forName0（本地方法）java.lang.Class.for名称（Class.java:264）com.sun.proxy.$Proxy23。（未知源）在sun.reflect.NativeConstructorAccessorImpl.newInstance0（本地方法）在sun.reflect.NativeConstructorAccessorImpl.new实例（NativeConstructorAccessorImpl.java:62）在sun.reflect.委托构建器AccessorImp. newInstance（委托构建器AccessorInp. java： 45）在java. lang.反射。[CNNVD]Constructor. newInstance Constructor. java： 423]Proxy. newProxyInstance（Proxy. java： 739）at org. apache. ProxyInstance（Proxy. java： 739）Parameters. createParametersProxy（Parameters. java： 294），网址为org. apache.联合体。参数. java： 185

这是我的身材。sbt

    libraryDependencies ++= Seq(
      "org.apache.commons" % "commons-configuration2" % "2.0",
      "commons-beanutils" % "commons-beanutils" % "1.9.2",
      "com.databricks" % "spark-avro_2.10" % "2.0.1",
      "com.databricks" % "spark-csv_2.10" % "1.4.0",
      "org.apache.spark" % "spark-sql_2.10" % "1.5.0" % "provided",
      "org.apache.spark" % "spark-hive_2.10" % "1.4.1" % "provided",
      "org.apache.spark" % "spark-core_2.10" % "1.4.1" % "provided",
      "com.amazonaws" % "aws-java-sdk" % "1.10.61",
      "org.apache.logging.log4j" % "log4j-api" % "2.6.2",
      "org.jasypt" % "jasypt" % "1.9.2",
      "commons-codec" % "commons-codec" % "1.8",
      "org.apache.kafka" % "kafka-clients" % "0.10.0.0",
      "org.apache.spark" % "spark-streaming-kafka_2.10" % "1.6.3",
      "org.apache.spark" % "spark-streaming_2.10" % "1.6.3" excludeAll(ExclusionRule(organization = "commons-beanutils"))

    )

    dependencyOverrides ++= Set(
      "com.fasterxml.jackson.core" % "jackson-databind" % "2.4.4",
      "org.apache.logging.log4j" % "log4j-api" % "2.6.2",
      "org.apache.logging.log4j" % "log4j-core" % "2.6.2",
      "org.apache.commons" % "commons-configuration2" % "2.0",
      "commons-beanutils" % "commons-beanutils" % "1.9.2"
    )

assemblyMergeStrategy in assembly := {
  case PathList("META-INF", xs @ _*) => MergeStrategy.discard
  case x => MergeStrategy.first
}

如何确保它正在使用commons-beanutils-1.9。2而不是commons-beanutils-1.7或commons-beanutils-core-1.8，它们是hadoop common的一部分？

Answer 1

在项目设置中排除不需要的jar对我来说很有用：

...
.settings(assemblyExcludedJars in assembly := {
    val cp = (fullClasspath in assembly).value

    val excludes = Set(
      "commons-beanutils-core-1.8.0.jar",
      "commons-beanutils-1.7.0.jar",
      "commons-beanutils-1.8.0.jar"
    )
    cp.filter{jar => excludes.contains(jar.data.getName)}
  })