登录
当前位置： 首页 > 知识库问答 >
问题：

如何修复“ClassCastException:无法分配”的实例-在本地工作，但不在集群上独立工作

周子平
2023-03-14

我有一个Spring Web应用程序（内置在maven中），我用它连接到我的火花集群（4个工作人员和1个主机）和我的cassandra集群（4个节点）。应用程序启动，工作人员与主机通信，cassandra集群也在运行。然而，当我通过我的web应用程序的界面进行PCA（火花mllib）或任何其他计算（集群、皮尔逊、斯皮尔曼）时，我得到以下错误：

java.lang.ClassCastException:无法分配scala.collection.immutable的实例。将$SerializationProxy列为字段org.apache.spark.rdd.rdd。org$apache$spark$rdd$rdd$$dependencies_类型为scala.collection。org.apache.spark.rdd.MapPartitionsRDD实例中的序列

出现在此命令上：

df1=df2.as(Encoders.STRING()).collectAsList();

  • spark-cassandra-connector_2.11-2.4.0.jar
  • guava-19.0.jar
  • cassandra-driver-core-3.4.0.jar

这似乎是一个很难追踪的错误，有相当多不同的方法和解释(这里1这里2这里3还有更多)。在过去的几天里，我尝试了我找到的每一种解决方案(甚至更多),但似乎都没有用。我的应用程序的依赖项和spark(可能是guava或scala)的依赖项之间似乎有冲突，但我不知道具体是什么。我试过了:

    < li >将我的application-jar-with-dependencies放在spark.jars类路径中。 < li >将application-jar-with-dependencies复制到每个worker上。 < li >用我的应用程序jar替换集群的主服务器和每个工作服务器上的所有spark-jar。 < li >我的应用程序和spark-jars文件夹中没有相同的jars。 < li >使用不同版本的不同依赖项。 < li >尝试使用阴影插件(不成功)或提供的作用域。 < li >使用spark.jars和spark . executor . extra class path。 < li >在节点上升级/降级scala版本(最终从workers和master上完全卸载scala，我仍然得到相同的错误) < li >使用spark-版本2.4.0 < li >以及许多其他方法...

火花配置为

    public SparkSession sparksession() {
         SparkSession sp = SparkSession
       .builder()
       .master("spark://(ip):7077")
       .appName("biomet")
       .config("spark.cassandra.connection.host","192.168.100.30,192.168.100.6,192.168.100.13,192.168.100.16")
       .config("spark.executor.extraClassPath","/home/ubuntu/spark-2.4.3-bin-hadoop2.7/jars/guava-19.0.jar")
       .config("spark.sql.shuffle.partitions",2) 
       .config("spark.driver.memory","4g")
       .config("spark.driver.maxResultSize", "2g")
       .config("spark.executor.memory","2g") 
       .config("spark.sql.pivotMaxValues","35000") 
       .config("spark.sql.caseSensitive",true)
       .config("spark.hadoop.fs.s3a.access.key",".....")
       .config("spark.hadoop.fs.s3a.secret.key",".......")
                 .getOrCreate();
         return sp;
     }

    @Bean
    public JavaSparkContext sc(){
        JavaSparkContext sc = new JavaSparkContext(sparksession().sparkContext());
        return sc;
    }

而pom.xml是：

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.example</groupId>
    <artifactId>app</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <packaging>jar</packaging>

    <name>springapp</name>
    <description>Demo project for Spring Boot</description>

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.0.9.RELEASE</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <java.version>1.8</java.version>
    </properties>

        <build>
            <plugins>
                <plugin>
                    <groupId>org.springframework.boot</groupId>
                    <artifactId>spring-boot-maven-plugin</artifactId>
                </plugin>
            </plugins>
    </build>


    <dependencies>
            <dependency>
                    <groupId>org.springframework.boot</groupId>
                    <artifactId>spring-boot-starter-thymeleaf</artifactId>
            </dependency>
            <dependency>
                    <groupId>org.springframework.boot</groupId>
                    <artifactId>spring-boot-starter-web</artifactId>
            </dependency>
            <dependency>
                    <groupId>org.springframework.boot</groupId>
                    <artifactId>spring-boot-starter-test</artifactId>
                    <scope>test</scope>
            </dependency>
            <dependency>
                <groupId>org.apache.spark</groupId>
                <artifactId>spark-mllib_2.11</artifactId>
                <version>2.4.3</version>
            </dependency>
            <dependency>
                <groupId>org.springframework.data</groupId>
                <artifactId>spring-data-cassandra</artifactId>
            </dependency>
            <dependency>
                <groupId>com.datastax.spark</groupId>
                <artifactId>spark-cassandra-connector-unshaded_2.11</artifactId>
                <version>2.4.0</version>
            </dependency>
      </dependencies>
</project>

如果需要，这是mvn依赖关系:树

[INFO] com.example:app:jar:0.0.1-SNAPSHOT
[INFO] +- org.springframework.boot:spring-boot-starter-thymeleaf:jar:2.0.9.RELEASE:compile
[INFO] |  +- org.springframework.boot:spring-boot-starter:jar:2.0.9.RELEASE:compile
[INFO] |  |  +- org.springframework.boot:spring-boot:jar:2.0.9.RELEASE:compile
[INFO] |  |  +- org.springframework.boot:spring-boot-autoconfigure:jar:2.0.9.RELEASE:compile
[INFO] |  |  +- org.springframework.boot:spring-boot-starter-logging:jar:2.0.9.RELEASE:compile
[INFO] |  |  |  +- ch.qos.logback:logback-classic:jar:1.2.3:compile
[INFO] |  |  |  |  \- ch.qos.logback:logback-core:jar:1.2.3:compile
[INFO] |  |  |  \- org.apache.logging.log4j:log4j-to-slf4j:jar:2.10.0:compile
[INFO] |  |  |     \- org.apache.logging.log4j:log4j-api:jar:2.10.0:compile
[INFO] |  |  +- javax.annotation:javax.annotation-api:jar:1.3.2:compile
[INFO] |  |  \- org.yaml:snakeyaml:jar:1.19:runtime
[INFO] |  +- org.thymeleaf:thymeleaf-spring5:jar:3.0.11.RELEASE:compile
[INFO] |  |  \- org.thymeleaf:thymeleaf:jar:3.0.11.RELEASE:compile
[INFO] |  |     +- org.attoparser:attoparser:jar:2.0.5.RELEASE:compile
[INFO] |  |     \- org.unbescape:unbescape:jar:1.1.6.RELEASE:compile
[INFO] |  \- org.thymeleaf.extras:thymeleaf-extras-java8time:jar:3.0.4.RELEASE:compile
[INFO] +- org.springframework.boot:spring-boot-starter-web:jar:2.0.9.RELEASE:compile
[INFO] |  +- org.springframework.boot:spring-boot-starter-json:jar:2.0.9.RELEASE:compile
[INFO] |  |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.9.8:compile
[INFO] |  |  |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.9.0:compile
[INFO] |  |  |  \- com.fasterxml.jackson.core:jackson-core:jar:2.9.8:compile
[INFO] |  |  +- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:jar:2.9.8:compile
[INFO] |  |  +- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.9.8:compile
[INFO] |  |  \- com.fasterxml.jackson.module:jackson-module-parameter-names:jar:2.9.8:compile
[INFO] |  +- org.springframework.boot:spring-boot-starter-tomcat:jar:2.0.9.RELEASE:compile
[INFO] |  |  +- org.apache.tomcat.embed:tomcat-embed-core:jar:8.5.39:compile
[INFO] |  |  +- org.apache.tomcat.embed:tomcat-embed-el:jar:8.5.39:compile
[INFO] |  |  \- org.apache.tomcat.embed:tomcat-embed-websocket:jar:8.5.39:compile
[INFO] |  +- org.hibernate.validator:hibernate-validator:jar:6.0.16.Final:compile
[INFO] |  |  +- javax.validation:validation-api:jar:2.0.1.Final:compile
[INFO] |  |  +- org.jboss.logging:jboss-logging:jar:3.3.2.Final:compile
[INFO] |  |  \- com.fasterxml:classmate:jar:1.3.4:compile
[INFO] |  +- org.springframework:spring-web:jar:5.0.13.RELEASE:compile
[INFO] |  \- org.springframework:spring-webmvc:jar:5.0.13.RELEASE:compile
[INFO] |     \- org.springframework:spring-aop:jar:5.0.13.RELEASE:compile
[INFO] +- org.springframework.boot:spring-boot-starter-test:jar:2.0.9.RELEASE:test
[INFO] |  +- org.springframework.boot:spring-boot-test:jar:2.0.9.RELEASE:test
[INFO] |  +- org.springframework.boot:spring-boot-test-autoconfigure:jar:2.0.9.RELEASE:test
[INFO] |  +- com.jayway.jsonpath:json-path:jar:2.4.0:test
[INFO] |  |  \- net.minidev:json-smart:jar:2.3:test
[INFO] |  |     \- net.minidev:accessors-smart:jar:1.2:test
[INFO] |  +- junit:junit:jar:4.12:test
[INFO] |  +- org.assertj:assertj-core:jar:3.9.1:test
[INFO] |  +- org.mockito:mockito-core:jar:2.15.0:test
[INFO] |  |  +- net.bytebuddy:byte-buddy:jar:1.7.11:test
[INFO] |  |  +- net.bytebuddy:byte-buddy-agent:jar:1.7.11:test
[INFO] |  |  \- org.objenesis:objenesis:jar:2.6:compile
[INFO] |  +- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] |  +- org.hamcrest:hamcrest-library:jar:1.3:test
[INFO] |  +- org.skyscreamer:jsonassert:jar:1.5.0:test
[INFO] |  |  \- com.vaadin.external.google:android-json:jar:0.0.20131108.vaadin1:test
[INFO] |  +- org.springframework:spring-core:jar:5.0.13.RELEASE:compile
[INFO] |  |  \- org.springframework:spring-jcl:jar:5.0.13.RELEASE:compile
[INFO] |  +- org.springframework:spring-test:jar:5.0.13.RELEASE:test
[INFO] |  \- org.xmlunit:xmlunit-core:jar:2.5.1:test
[INFO] +- org.apache.spark:spark-mllib_2.11:jar:2.4.3:compile
[INFO] |  +- org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.1.0:compile
[INFO] |  +- org.apache.spark:spark-core_2.11:jar:2.4.3:compile
[INFO] |  |  +- com.thoughtworks.paranamer:paranamer:jar:2.8:compile
[INFO] |  |  +- org.apache.avro:avro:jar:1.8.2:compile
[INFO] |  |  |  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] |  |  |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] |  |  |  +- org.apache.commons:commons-compress:jar:1.8.1:compile
[INFO] |  |  |  \- org.tukaani:xz:jar:1.5:compile
[INFO] |  |  +- org.apache.avro:avro-mapred:jar:hadoop2:1.8.2:compile
[INFO] |  |  |  +- org.apache.avro:avro-ipc:jar:1.8.2:compile
[INFO] |  |  |  \- commons-codec:commons-codec:jar:1.11:compile
[INFO] |  |  +- com.twitter:chill_2.11:jar:0.9.3:compile
[INFO] |  |  |  \- com.esotericsoftware:kryo-shaded:jar:4.0.2:compile
[INFO] |  |  |     \- com.esotericsoftware:minlog:jar:1.3.0:compile
[INFO] |  |  +- com.twitter:chill-java:jar:0.9.3:compile
[INFO] |  |  +- org.apache.xbean:xbean-asm6-shaded:jar:4.8:compile
[INFO] |  |  +- org.apache.hadoop:hadoop-client:jar:2.6.5:compile
[INFO] |  |  |  +- org.apache.hadoop:hadoop-common:jar:2.6.5:compile
[INFO] |  |  |  |  +- commons-cli:commons-cli:jar:1.2:compile
[INFO] |  |  |  |  +- xmlenc:xmlenc:jar:0.52:compile
[INFO] |  |  |  |  +- commons-httpclient:commons-httpclient:jar:3.1:compile
[INFO] |  |  |  |  +- commons-io:commons-io:jar:2.4:compile
[INFO] |  |  |  |  +- commons-configuration:commons-configuration:jar:1.6:compile
[INFO] |  |  |  |  |  \- commons-digester:commons-digester:jar:1.8:compile
[INFO] |  |  |  |  +- com.google.code.gson:gson:jar:2.8.5:compile
[INFO] |  |  |  |  +- org.apache.hadoop:hadoop-auth:jar:2.6.5:compile
[INFO] |  |  |  |  |  +- org.apache.httpcomponents:httpclient:jar:4.5.8:compile
[INFO] |  |  |  |  |  |  \- org.apache.httpcomponents:httpcore:jar:4.4.11:compile
[INFO] |  |  |  |  |  \- org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:compile
[INFO] |  |  |  |  |     +- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:compile
[INFO] |  |  |  |  |     +- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:compile
[INFO] |  |  |  |  |     \- org.apache.directory.api:api-util:jar:1.0.0-M20:compile
[INFO] |  |  |  |  +- org.apache.curator:curator-client:jar:2.6.0:compile
[INFO] |  |  |  |  \- org.htrace:htrace-core:jar:3.0.4:compile
[INFO] |  |  |  +- org.apache.hadoop:hadoop-hdfs:jar:2.6.5:compile
[INFO] |  |  |  |  +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
[INFO] |  |  |  |  \- xerces:xercesImpl:jar:2.9.1:compile
[INFO] |  |  |  |     \- xml-apis:xml-apis:jar:1.4.01:compile
[INFO] |  |  |  +- org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.6.5:compile
[INFO] |  |  |  |  +- org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.6.5:compile
[INFO] |  |  |  |  |  +- org.apache.hadoop:hadoop-yarn-client:jar:2.6.5:compile
[INFO] |  |  |  |  |  \- org.apache.hadoop:hadoop-yarn-server-common:jar:2.6.5:compile
[INFO] |  |  |  |  \- org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.6.5:compile
[INFO] |  |  |  +- org.apache.hadoop:hadoop-yarn-api:jar:2.6.5:compile
[INFO] |  |  |  +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.6.5:compile
[INFO] |  |  |  |  \- org.apache.hadoop:hadoop-yarn-common:jar:2.6.5:compile
[INFO] |  |  |  |     +- javax.xml.bind:jaxb-api:jar:2.3.1:compile
[INFO] |  |  |  |     |  \- javax.activation:javax.activation-api:jar:1.2.0:compile
[INFO] |  |  |  |     +- org.codehaus.jackson:jackson-jaxrs:jar:1.9.13:compile
[INFO] |  |  |  |     \- org.codehaus.jackson:jackson-xc:jar:1.9.13:compile
[INFO] |  |  |  +- org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.6.5:compile
[INFO] |  |  |  \- org.apache.hadoop:hadoop-annotations:jar:2.6.5:compile
[INFO] |  |  +- org.apache.spark:spark-launcher_2.11:jar:2.4.3:compile
[INFO] |  |  +- org.apache.spark:spark-kvstore_2.11:jar:2.4.3:compile
[INFO] |  |  |  \- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:compile
[INFO] |  |  +- org.apache.spark:spark-network-common_2.11:jar:2.4.3:compile
[INFO] |  |  +- org.apache.spark:spark-network-shuffle_2.11:jar:2.4.3:compile
[INFO] |  |  +- org.apache.spark:spark-unsafe_2.11:jar:2.4.3:compile
[INFO] |  |  +- javax.activation:activation:jar:1.1.1:compile
[INFO] |  |  +- org.apache.curator:curator-recipes:jar:2.6.0:compile
[INFO] |  |  |  \- org.apache.curator:curator-framework:jar:2.6.0:compile
[INFO] |  |  +- org.apache.zookeeper:zookeeper:jar:3.4.6:compile
[INFO] |  |  +- javax.servlet:javax.servlet-api:jar:3.1.0:compile
[INFO] |  |  +- org.apache.commons:commons-lang3:jar:3.7:compile
[INFO] |  |  +- com.google.code.findbugs:jsr305:jar:1.3.9:compile
[INFO] |  |  +- org.slf4j:jul-to-slf4j:jar:1.7.26:compile
[INFO] |  |  +- org.slf4j:jcl-over-slf4j:jar:1.7.26:compile
[INFO] |  |  +- log4j:log4j:jar:1.2.17:compile
[INFO] |  |  +- org.slf4j:slf4j-log4j12:jar:1.7.26:compile
[INFO] |  |  +- com.ning:compress-lzf:jar:1.0.3:compile
[INFO] |  |  +- org.xerial.snappy:snappy-java:jar:1.1.7.3:compile
[INFO] |  |  +- org.lz4:lz4-java:jar:1.4.0:compile
[INFO] |  |  +- com.github.luben:zstd-jni:jar:1.3.2-2:compile
[INFO] |  |  +- org.roaringbitmap:RoaringBitmap:jar:0.7.45:compile
[INFO] |  |  |  \- org.roaringbitmap:shims:jar:0.7.45:compile
[INFO] |  |  +- commons-net:commons-net:jar:3.1:compile
[INFO] |  |  +- org.json4s:json4s-jackson_2.11:jar:3.5.3:compile
[INFO] |  |  |  \- org.json4s:json4s-core_2.11:jar:3.5.3:compile
[INFO] |  |  |     +- org.json4s:json4s-ast_2.11:jar:3.5.3:compile
[INFO] |  |  |     +- org.json4s:json4s-scalap_2.11:jar:3.5.3:compile
[INFO] |  |  |     \- org.scala-lang.modules:scala-xml_2.11:jar:1.0.6:compile
[INFO] |  |  +- org.glassfish.jersey.core:jersey-client:jar:2.26:compile
[INFO] |  |  +- org.glassfish.jersey.core:jersey-server:jar:2.26:compile
[INFO] |  |  |  \- org.glassfish.jersey.media:jersey-media-jaxb:jar:2.26:compile
[INFO] |  |  +- org.glassfish.jersey.containers:jersey-container-servlet:jar:2.26:compile
[INFO] |  |  +- org.glassfish.jersey.containers:jersey-container-servlet-core:jar:2.26:compile
[INFO] |  |  +- io.netty:netty:jar:3.9.9.Final:compile
[INFO] |  |  +- com.clearspring.analytics:stream:jar:2.7.0:compile
[INFO] |  |  +- io.dropwizard.metrics:metrics-core:jar:3.2.6:compile
[INFO] |  |  +- io.dropwizard.metrics:metrics-jvm:jar:3.2.6:compile
[INFO] |  |  +- io.dropwizard.metrics:metrics-json:jar:3.2.6:compile
[INFO] |  |  +- io.dropwizard.metrics:metrics-graphite:jar:3.2.6:compile
[INFO] |  |  +- com.fasterxml.jackson.module:jackson-module-scala_2.11:jar:2.9.8:compile
[INFO] |  |  |  \- com.fasterxml.jackson.module:jackson-module-paranamer:jar:2.9.8:compile
[INFO] |  |  +- org.apache.ivy:ivy:jar:2.4.0:compile
[INFO] |  |  +- oro:oro:jar:2.0.8:compile
[INFO] |  |  +- net.razorvine:pyrolite:jar:4.13:compile
[INFO] |  |  +- net.sf.py4j:py4j:jar:0.10.7:compile
[INFO] |  |  \- org.apache.commons:commons-crypto:jar:1.0.0:compile
[INFO] |  +- org.apache.spark:spark-streaming_2.11:jar:2.4.3:compile
[INFO] |  +- org.apache.spark:spark-sql_2.11:jar:2.4.3:compile
[INFO] |  |  +- com.univocity:univocity-parsers:jar:2.7.3:compile
[INFO] |  |  +- org.apache.spark:spark-sketch_2.11:jar:2.4.3:compile
[INFO] |  |  +- org.apache.spark:spark-catalyst_2.11:jar:2.4.3:compile
[INFO] |  |  |  +- org.codehaus.janino:janino:jar:3.0.12:compile
[INFO] |  |  |  +- org.codehaus.janino:commons-compiler:jar:3.0.9:compile
[INFO] |  |  |  \- org.antlr:antlr4-runtime:jar:4.7:compile
[INFO] |  |  +- org.apache.orc:orc-core:jar:nohive:1.5.5:compile
[INFO] |  |  |  +- org.apache.orc:orc-shims:jar:1.5.5:compile
[INFO] |  |  |  +- com.google.protobuf:protobuf-java:jar:2.5.0:compile
[INFO] |  |  |  +- commons-lang:commons-lang:jar:2.6:compile
[INFO] |  |  |  \- io.airlift:aircompressor:jar:0.10:compile
[INFO] |  |  +- org.apache.orc:orc-mapreduce:jar:nohive:1.5.5:compile
[INFO] |  |  +- org.apache.parquet:parquet-column:jar:1.10.1:compile
[INFO] |  |  |  +- org.apache.parquet:parquet-common:jar:1.10.1:compile
[INFO] |  |  |  \- org.apache.parquet:parquet-encoding:jar:1.10.1:compile
[INFO] |  |  +- org.apache.parquet:parquet-hadoop:jar:1.10.1:compile
[INFO] |  |  |  +- org.apache.parquet:parquet-format:jar:2.4.0:compile
[INFO] |  |  |  \- org.apache.parquet:parquet-jackson:jar:1.10.1:compile
[INFO] |  |  \- org.apache.arrow:arrow-vector:jar:0.10.0:compile
[INFO] |  |     +- org.apache.arrow:arrow-format:jar:0.10.0:compile
[INFO] |  |     +- org.apache.arrow:arrow-memory:jar:0.10.0:compile
[INFO] |  |     +- com.carrotsearch:hppc:jar:0.7.2:compile
[INFO] |  |     \- com.vlkan:flatbuffers:jar:1.2.0-3f79e055:compile
[INFO] |  +- org.apache.spark:spark-graphx_2.11:jar:2.4.3:compile
[INFO] |  |  +- com.github.fommil.netlib:core:jar:1.1.2:compile
[INFO] |  |  \- net.sourceforge.f2j:arpack_combined_all:jar:0.1:compile
[INFO] |  +- org.apache.spark:spark-mllib-local_2.11:jar:2.4.3:compile
[INFO] |  +- org.scalanlp:breeze_2.11:jar:0.13.2:compile
[INFO] |  |  +- org.scalanlp:breeze-macros_2.11:jar:0.13.2:compile
[INFO] |  |  +- net.sf.opencsv:opencsv:jar:2.3:compile
[INFO] |  |  +- com.github.rwl:jtransforms:jar:2.4.0:compile
[INFO] |  |  +- org.spire-math:spire_2.11:jar:0.13.0:compile
[INFO] |  |  |  +- org.spire-math:spire-macros_2.11:jar:0.13.0:compile
[INFO] |  |  |  \- org.typelevel:machinist_2.11:jar:0.6.1:compile
[INFO] |  |  \- com.chuusai:shapeless_2.11:jar:2.3.2:compile
[INFO] |  |     \- org.typelevel:macro-compat_2.11:jar:1.1.1:compile
[INFO] |  +- org.apache.commons:commons-math3:jar:3.4.1:compile
[INFO] |  +- org.apache.spark:spark-tags_2.11:jar:2.4.3:compile
[INFO] |  \- org.spark-project.spark:unused:jar:1.0.0:compile
[INFO] +- org.glassfish.jersey.inject:jersey-hk2:jar:2.26:compile
[INFO] |  +- org.glassfish.jersey.core:jersey-common:jar:2.26:compile
[INFO] |  |  +- javax.ws.rs:javax.ws.rs-api:jar:2.1:compile
[INFO] |  |  +- org.glassfish.hk2.external:javax.inject:jar:2.5.0-b42:compile
[INFO] |  |  \- org.glassfish.hk2:osgi-resource-locator:jar:1.0.1:compile
[INFO] |  \- org.glassfish.hk2:hk2-locator:jar:2.5.0-b42:compile
[INFO] |     +- org.glassfish.hk2.external:aopalliance-repackaged:jar:2.5.0-b42:compile
[INFO] |     +- org.glassfish.hk2:hk2-api:jar:2.5.0-b42:compile
[INFO] |     |  \- javax.inject:javax.inject:jar:1:compile
[INFO] |     +- org.glassfish.hk2:hk2-utils:jar:2.5.0-b42:compile
[INFO] |     \- org.javassist:javassist:jar:3.22.0-CR2:compile
[INFO] +- org.springframework.data:spring-data-cassandra:jar:2.0.14.RELEASE:compile
[INFO] |  +- org.springframework:spring-context:jar:5.0.13.RELEASE:compile
[INFO] |  +- org.springframework:spring-beans:jar:5.0.13.RELEASE:compile
[INFO] |  +- org.springframework:spring-tx:jar:5.0.13.RELEASE:compile
[INFO] |  +- org.springframework:spring-expression:jar:5.0.13.RELEASE:compile
[INFO] |  +- org.springframework.data:spring-data-commons:jar:2.0.14.RELEASE:compile
[INFO] |  +- com.datastax.cassandra:cassandra-driver-core:jar:3.4.0:compile
[INFO] |  |  +- io.netty:netty-handler:jar:4.1.34.Final:compile
[INFO] |  |  |  +- io.netty:netty-common:jar:4.1.34.Final:compile
[INFO] |  |  |  +- io.netty:netty-buffer:jar:4.1.34.Final:compile
[INFO] |  |  |  +- io.netty:netty-transport:jar:4.1.34.Final:compile
[INFO] |  |  |  |  \- io.netty:netty-resolver:jar:4.1.34.Final:compile
[INFO] |  |  |  \- io.netty:netty-codec:jar:4.1.34.Final:compile
[INFO] |  |  +- com.github.jnr:jnr-ffi:jar:2.0.7:compile
[INFO] |  |  |  +- com.github.jnr:jffi:jar:1.2.10:compile
[INFO] |  |  |  +- com.github.jnr:jffi:jar:native:1.2.10:runtime
[INFO] |  |  |  +- org.ow2.asm:asm:jar:5.0.3:compile
[INFO] |  |  |  +- org.ow2.asm:asm-commons:jar:5.0.3:compile
[INFO] |  |  |  +- org.ow2.asm:asm-analysis:jar:5.0.3:compile
[INFO] |  |  |  +- org.ow2.asm:asm-tree:jar:5.0.3:compile
[INFO] |  |  |  +- org.ow2.asm:asm-util:jar:5.0.3:compile
[INFO] |  |  |  \- com.github.jnr:jnr-x86asm:jar:1.0.2:compile
[INFO] |  |  \- com.github.jnr:jnr-posix:jar:3.0.27:compile
[INFO] |  |     \- com.github.jnr:jnr-constants:jar:0.9.0:compile
[INFO] |  +- com.google.guava:guava:jar:19.0:compile
[INFO] |  \- org.slf4j:slf4j-api:jar:1.7.26:compile
[INFO] \- com.datastax.spark:spark-cassandra-connector-unshaded_2.11:jar:2.4.0:compile
[INFO]    +- org.scala-lang:scala-library:jar:2.11.12:compile
[INFO]    +- com.datastax.cassandra:cassandra-driver-mapping:jar:3.4.0:compile
[INFO]    +- commons-beanutils:commons-beanutils:jar:1.9.3:compile
[INFO]    |  \- commons-collections:commons-collections:jar:3.2.2:compile
[INFO]    +- org.joda:joda-convert:jar:1.2:compile
[INFO]    +- joda-time:joda-time:jar:2.9.9:compile
[INFO]    +- io.netty:netty-all:jar:4.1.34.Final:compile
[INFO]    +- com.twitter:jsr166e:jar:1.1.0:compile
[INFO]    \- org.scala-lang:scala-reflect:jar:2.11.12:compile
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS

共有2个答案

邰宇
2023-03-14

尝试用log4j替换logback（删除logback依赖项），至少在我们的类似情况下它有所帮助。

袁骏祥
2023-03-14

依赖关系之间似乎存在冲突，但不知道到底是什么！这就是我所做的：Spring-Spark：罐子/依赖关系之间的冲突

编辑：

正如它在上面的链接中所解释的，我只需要改变一下：

<plugin>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-maven-plugin</artifactId>
            <version>2.0.9.RELEASE</version>
            <executions>
            <execution>
                <goals>
                    <goal>repackage</goal>
                </goals>
            </execution>
        </executions>
            <configuration>
                <requiresUnpack>
                    <dependency>
                        <groupId>org.apache.spark</groupId>
                        <artifactId>spark-core_2.11</artifactId>
                    </dependency>
                    <dependency>
                        <groupId>org.apache.spark</groupId>
                        <artifactId>spark-streaming_2.11</artifactId>
                    </dependency>

                </requiresUnpack>
            </configuration>
        </plugin>

有了这个:

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-compiler-plugin</artifactId>
    <version>3.8.0</version>
    <configuration>
        <source>1.8</source>
        <target>1.8</target>
    </configuration>
</plugin>
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>2.4.1</version>
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-maven-plugin</artifactId>
            <version>2.0.9.RELEASE</version>
        </dependency>
    </dependencies>
    <configuration>
        <keepDependenciesWithProvidedScope>true</keepDependenciesWithProvidedScope>
        <createDependencyReducedPom>true</createDependencyReducedPom>
        <filters>
            <filter>
                <artifact>*:*</artifact>
                <excludes>
                    <exclude>META-INF/*.SF</exclude>
                    <exclude>META-INF/*.DSA</exclude>
                    <exclude>META-INF/*.RSA</exclude>
                </excludes>
            </filter>
        </filters>
    </configuration>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>shade</goal>
            </goals>
            <configuration>
                <transformers>
                    <transformer
                        implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                        <resource>META-INF/spring.handlers</resource>
                    </transformer>
                    <transformer
                        implementation="org.springframework.boot.maven.PropertiesMergingResourceTransformer">
                        <resource>META-INF/spring.factories</resource>
                    </transformer>
                    <transformer
                        implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                        <resource>META-INF/spring.schemas</resource>
                    </transformer>
                    <transformer
                        implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
                    <transformer
                        implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                        <mainClass>com.example.app.Mtbapp1</mainClass>
                    </transformer>
                </transformers>
            </configuration>
        </execution>
    </executions>
</plugin>
 类似资料：
  • Jenkins jnlp从机不在本地Kubernetes群集上工作

    我尝试了本地minikube和kubeadm Kubernetes集群，并通过https://github.com/jenkinsci/kubernetes-plugin运行Jenkins服务器，并使用。 以下是了解这个问题的更多细节。

  • socket.io房间不在Azure上工作，但在本地工作

    IIS可能会搞砸它？我倾向于认为IIS对于和)有不同的进程，因此，我在app.post上引用的套接字与我在()中加入用户是不同的。 在Azure/IIS中使用socket.io房间时有什么提示吗？谢了！

  • Puppeteer脚本在本地工作，但不能在EC2 AWS上工作

    我正在使用Amazon Linux AMI在EC2 AWS上运行以下脚本 从今天起，由于EC2上没有任何更改或新安装，脚本在昨天之前一直工作时停止工作。 本地计算机上的相同脚本仍然有效。 相反,在EC2上会出现以下错误： （节点：12636）UnhandledPromiseRejectionWarning：错误：导航失败，因为浏览器已断开连接！在cdpsession.lifecyclewatche

  • Spark cassandra连接器不能在独立的Spark集群中工作

    我有一个向spark独立单节点集群提交spark作业的maven scala应用程序。提交作业时，Spark应用程序尝试使用spark-cassandra-connector访问Amazon EC2实例上托管的cassandra。连接已建立，但不返回结果。一段时间后连接器断开。如果我在本地模式下运行spark，它工作得很好。我试图创建简单的应用程序，代码如下所示： SparkContext.Sca

  • 如何在独立集群模式下为每个工作者分配更多的执行者？

    当Spark应用程序提交到集群时，这些设置在上设置。

  • TCPDF在本地工作，但不在线

    我正在使用PHP*TCPDF/FPDI类动态生成PDF。。 虽然本地发展（WAMP/localhost）... 现在我已经把东西上传到了直播服务器上。。我在同一个文件上只得到一张空白页。。使用相同的目录结构。。 但是，在实时环境中进行测试时。。它不起作用？ 只是一个空白页，没有错误...等等。 这是我目前使用的代码。。 （我试图在构造函数和超文本标记语言输出中添加UTF-8编码...？？ 代码：

相关问答

如何在独立集群模式下为每个工作者分配更多的执行者？TCPDF在本地工作，但不在线spark独立集群：如何限制工作人员的数量？Camel独立-PropertiesComponent的配置不能在@BeanInject的bean上工作设计在本地工作，但在Heroku上不行

相关文章

pySerial在Python解释器中可以正常工作，但不能独立运行numpy的就地操作（例如，“ + =”）如何工作？在类本身内部创建类的实例如何工作？如何在Android上重复警报工作日来自Linux的Python pysftp get_r在Linux上工作正常，但在Windows上工作不正常

相关阅读

Java9集合工厂方法不知道如何分库分表，看完这篇文章，轻松应对工作面试有效地使用工作区控制开立医疗-AI算法工程师开立医疗 操作系统工程师 一面

相关工具

独立于window的按钮容联云通讯API的开发工具集网络矿工采集器（原soukey采摘）小鸡软件工作室应用程序保护系统common工具类

相关文档

工作流程如流智能工作台帮助文档前端工程师自我修养Oracle RAC 集群部署 kubernetes 集群
快捷导航： 新手教程 算法原理 架构设计 Java进阶 数据库进阶 大厂专栏 面试经验 编程笔记 编程问答 所有专题 文档资料 工具软件 电子书籍 小牛导航
在线工具： 房贷计算器 个税计算器 Linux命令查询 Json格式化 正则表达式 颜色转换 AES加解密 SHA1加密 MD5加密 毒鸡汤 字数统计 随机密码生成 进制转换 Base64编解码 励志句子
Copyright © 2019-2024 小牛知识库@xnip.cn. All Rights Reserved.