apache-spark - 如何修复“ClassCastException：无法分配实例”-在本地工作但不能在集群上独立运行

Question

我有一个 Spring Web 应用程序（内置 maven），我用它连接到我的 spark 集群（4 个 worker 和 1 个 master）和我的 cassandra 集群（4 个节点）。应用程序启动，worker 与 master 通信，cassandra 集群也在运行。但是，当我通过我的网络应用程序的界面进行 PCA（spark mllib）或任何其他计算（聚类、pearson、spearman）时，我收到以下错误：

java.lang.ClassCastException：无法将 scala.collection.immutable.List$SerializationProxy 的实例分配给 scala.collection.Seq 类型的字段 org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_在 org.apache.spark.rdd.MapPartitionsRDD 的实例中

出现在此命令上：

df1=df2.as(Encoders.STRING()).collectAsList();

我已经在 master 和每个 worker 上复制了 spark-2.4.3-bin-hadoop2.7 文件夹。我还在 spark-jars 文件夹中添加了 3 个 jar，因为如果我不这样做（火花需要它们）会出现错误：

spark-cassandra-connector_2.11-2.4.0.jar
番石榴19.0.jar
cassandra-driver-core-3.4.0.jar

似乎这是一个难以追踪的错误，并且有大量不同的方法和解释（here1 here2 here3 等等）。我尝试了最近几天找到的所有解决方案（以及更多），但似乎没有任何效果。这似乎是我的应用程序的依赖项和 spark 的依赖项（可能是番石榴或 scala）之间的冲突，但我找不到确切的内容。我试过了：

将我的 application-jar-with-dependencies 放在 spark.jars 类路径中。
将 application-jar-with-dependencies 复制到每个 worker 上。
用我在master和集群的每个worker上的应用程序jar替换所有spark-jar。
我的应用程序和 spark-jars 文件夹中没有相同的 jar。
玩了不同版本的不同依赖项。
尝试使用带阴影的插件（不成功）或提供的范围。
使用 spark.jars 和 spark.executor.extraClassPath。
升级/降级节点上的scala版本（最终从worker和master完全卸载scala，我仍然得到同样的错误）
使用火花版本 2.4.0
和许多其他方法...

火花配置是

    public SparkSession sparksession() {
         SparkSession sp = SparkSession
       .builder()
       .master("spark://(ip):7077")
       .appName("biomet")
       .config("spark.cassandra.connection.host","192.168.100.30,192.168.100.6,192.168.100.13,192.168.100.16")
       .config("spark.executor.extraClassPath","/home/ubuntu/spark-2.4.3-bin-hadoop2.7/jars/guava-19.0.jar")
       .config("spark.sql.shuffle.partitions",2) 
       .config("spark.driver.memory","4g")
       .config("spark.driver.maxResultSize", "2g")
       .config("spark.executor.memory","2g") 
       .config("spark.sql.pivotMaxValues","35000") 
       .config("spark.sql.caseSensitive",true)
       .config("spark.hadoop.fs.s3a.access.key",".....")
       .config("spark.hadoop.fs.s3a.secret.key",".......")
                 .getOrCreate();
         return sp;
     }

    @Bean
    public JavaSparkContext sc(){
        JavaSparkContext sc = new JavaSparkContext(sparksession().sparkContext());
        return sc;
    }

pom.xml 是：

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.example</groupId>
    <artifactId>app</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <packaging>jar</packaging>

    <name>springapp</name>
    <description>Demo project for Spring Boot</description>

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.0.9.RELEASE</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <java.version>1.8</java.version>
    </properties>

        <build>
            <plugins>
                <plugin>
                    <groupId>org.springframework.boot</groupId>
                    <artifactId>spring-boot-maven-plugin</artifactId>
                </plugin>
            </plugins>
    </build>


    <dependencies>
            <dependency>
                    <groupId>org.springframework.boot</groupId>
                    <artifactId>spring-boot-starter-thymeleaf</artifactId>
            </dependency>
            <dependency>
                    <groupId>org.springframework.boot</groupId>
                    <artifactId>spring-boot-starter-web</artifactId>
            </dependency>
            <dependency>
                    <groupId>org.springframework.boot</groupId>
                    <artifactId>spring-boot-starter-test</artifactId>
                    <scope>test</scope>
            </dependency>
            <dependency>
                <groupId>org.apache.spark</groupId>
                <artifactId>spark-mllib_2.11</artifactId>
                <version>2.4.3</version>
            </dependency>
            <dependency>
                <groupId>org.springframework.data</groupId>
                <artifactId>spring-data-cassandra</artifactId>
            </dependency>
            <dependency>
                <groupId>com.datastax.spark</groupId>
                <artifactId>spark-cassandra-connector-unshaded_2.11</artifactId>
                <version>2.4.0</version>
            </dependency>
      </dependencies>
</project>

如果需要，这是 mvn 依赖项：tree

[INFO] com.example:app:jar:0.0.1-SNAPSHOT
[INFO] +- org.springframework.boot:spring-boot-starter-thymeleaf:jar:2.0.9.RELEASE:compile
[INFO] |  +- org.springframework.boot:spring-boot-starter:jar:2.0.9.RELEASE:compile
[INFO] |  |  +- org.springframework.boot:spring-boot:jar:2.0.9.RELEASE:compile
[INFO] |  |  +- org.springframework.boot:spring-boot-autoconfigure:jar:2.0.9.RELEASE:compile
[INFO] |  |  +- org.springframework.boot:spring-boot-starter-logging:jar:2.0.9.RELEASE:compile
[INFO] |  |  |  +- ch.qos.logback:logback-classic:jar:1.2.3:compile
[INFO] |  |  |  |  \- ch.qos.logback:logback-core:jar:1.2.3:compile
[INFO] |  |  |  \- org.apache.logging.log4j:log4j-to-slf4j:jar:2.10.0:compile
[INFO] |  |  |     \- org.apache.logging.log4j:log4j-api:jar:2.10.0:compile
[INFO] |  |  +- javax.annotation:javax.annotation-api:jar:1.3.2:compile
[INFO] |  |  \- org.yaml:snakeyaml:jar:1.19:runtime
[INFO] |  +- org.thymeleaf:thymeleaf-spring5:jar:3.0.11.RELEASE:compile
[INFO] |  |  \- org.thymeleaf:thymeleaf:jar:3.0.11.RELEASE:compile
[INFO] |  |     +- org.attoparser:attoparser:jar:2.0.5.RELEASE:compile
[INFO] |  |     \- org.unbescape:unbescape:jar:1.1.6.RELEASE:compile
[INFO] |  \- org.thymeleaf.extras:thymeleaf-extras-java8time:jar:3.0.4.RELEASE:compile
[INFO] +- org.springframework.boot:spring-boot-starter-web:jar:2.0.9.RELEASE:compile
[INFO] |  +- org.springframework.boot:spring-boot-starter-json:jar:2.0.9.RELEASE:compile
[INFO] |  |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.9.8:compile
[INFO] |  |  |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.9.0:compile
[INFO] |  |  |  \- com.fasterxml.jackson.core:jackson-core:jar:2.9.8:compile
[INFO] |  |  +- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:jar:2.9.8:compile
[INFO] |  |  +- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.9.8:compile
[INFO] |  |  \- com.fasterxml.jackson.module:jackson-module-parameter-names:jar:2.9.8:compile
[INFO] |  +- org.springframework.boot:spring-boot-starter-tomcat:jar:2.0.9.RELEASE:compile
[INFO] |  |  +- org.apache.tomcat.embed:tomcat-embed-core:jar:8.5.39:compile
[INFO] |  |  +- org.apache.tomcat.embed:tomcat-embed-el:jar:8.5.39:compile
[INFO] |  |  \- org.apache.tomcat.embed:tomcat-embed-websocket:jar:8.5.39:compile
[INFO] |  +- org.hibernate.validator:hibernate-validator:jar:6.0.16.Final:compile
[INFO] |  |  +- javax.validation:validation-api:jar:2.0.1.Final:compile
[INFO] |  |  +- org.jboss.logging:jboss-logging:jar:3.3.2.Final:compile
[INFO] |  |  \- com.fasterxml:classmate:jar:1.3.4:compile
[INFO] |  +- org.springframework:spring-web:jar:5.0.13.RELEASE:compile
[INFO] |  \- org.springframework:spring-webmvc:jar:5.0.13.RELEASE:compile
[INFO] |     \- org.springframework:spring-aop:jar:5.0.13.RELEASE:compile
[INFO] +- org.springframework.boot:spring-boot-starter-test:jar:2.0.9.RELEASE:test
[INFO] |  +- org.springframework.boot:spring-boot-test:jar:2.0.9.RELEASE:test
[INFO] |  +- org.springframework.boot:spring-boot-test-autoconfigure:jar:2.0.9.RELEASE:test
[INFO] |  +- com.jayway.jsonpath:json-path:jar:2.4.0:test
[INFO] |  |  \- net.minidev:json-smart:jar:2.3:test
[INFO] |  |     \- net.minidev:accessors-smart:jar:1.2:test
[INFO] |  +- junit:junit:jar:4.12:test
[INFO] |  +- org.assertj:assertj-core:jar:3.9.1:test
[INFO] |  +- org.mockito:mockito-core:jar:2.15.0:test
[INFO] |  |  +- net.bytebuddy:byte-buddy:jar:1.7.11:test
[INFO] |  |  +- net.bytebuddy:byte-buddy-agent:jar:1.7.11:test
[INFO] |  |  \- org.objenesis:objenesis:jar:2.6:compile
[INFO] |  +- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] |  +- org.hamcrest:hamcrest-library:jar:1.3:test
[INFO] |  +- org.skyscreamer:jsonassert:jar:1.5.0:test
[INFO] |  |  \- com.vaadin.external.google:android-json:jar:0.0.20131108.vaadin1:test
[INFO] |  +- org.springframework:spring-core:jar:5.0.13.RELEASE:compile
[INFO] |  |  \- org.springframework:spring-jcl:jar:5.0.13.RELEASE:compile
[INFO] |  +- org.springframework:spring-test:jar:5.0.13.RELEASE:test
[INFO] |  \- org.xmlunit:xmlunit-core:jar:2.5.1:test
[INFO] +- org.apache.spark:spark-mllib_2.11:jar:2.4.3:compile
[INFO] |  +- org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.1.0:compile
[INFO] |  +- org.apache.spark:spark-core_2.11:jar:2.4.3:compile
[INFO] |  |  +- com.thoughtworks.paranamer:paranamer:jar:2.8:compile
[INFO] |  |  +- org.apache.avro:avro:jar:1.8.2:compile
[INFO] |  |  |  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] |  |  |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] |  |  |  +- org.apache.commons:commons-compress:jar:1.8.1:compile
[INFO] |  |  |  \- org.tukaani:xz:jar:1.5:compile
[INFO] |  |  +- org.apache.avro:avro-mapred:jar:hadoop2:1.8.2:compile
[INFO] |  |  |  +- org.apache.avro:avro-ipc:jar:1.8.2:compile
[INFO] |  |  |  \- commons-codec:commons-codec:jar:1.11:compile
[INFO] |  |  +- com.twitter:chill_2.11:jar:0.9.3:compile
[INFO] |  |  |  \- com.esotericsoftware:kryo-shaded:jar:4.0.2:compile
[INFO] |  |  |     \- com.esotericsoftware:minlog:jar:1.3.0:compile
[INFO] |  |  +- com.twitter:chill-java:jar:0.9.3:compile
[INFO] |  |  +- org.apache.xbean:xbean-asm6-shaded:jar:4.8:compile
[INFO] |  |  +- org.apache.hadoop:hadoop-client:jar:2.6.5:compile
[INFO] |  |  |  +- org.apache.hadoop:hadoop-common:jar:2.6.5:compile
[INFO] |  |  |  |  +- commons-cli:commons-cli:jar:1.2:compile
[INFO] |  |  |  |  +- xmlenc:xmlenc:jar:0.52:compile
[INFO] |  |  |  |  +- commons-httpclient:commons-httpclient:jar:3.1:compile
[INFO] |  |  |  |  +- commons-io:commons-io:jar:2.4:compile
[INFO] |  |  |  |  +- commons-configuration:commons-configuration:jar:1.6:compile
[INFO] |  |  |  |  |  \- commons-digester:commons-digester:jar:1.8:compile
[INFO] |  |  |  |  +- com.google.code.gson:gson:jar:2.8.5:compile
[INFO] |  |  |  |  +- org.apache.hadoop:hadoop-auth:jar:2.6.5:compile
[INFO] |  |  |  |  |  +- org.apache.httpcomponents:httpclient:jar:4.5.8:compile
[INFO] |  |  |  |  |  |  \- org.apache.httpcomponents:httpcore:jar:4.4.11:compile
[INFO] |  |  |  |  |  \- org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:compile
[INFO] |  |  |  |  |     +- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:compile
[INFO] |  |  |  |  |     +- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:compile
[INFO] |  |  |  |  |     \- org.apache.directory.api:api-util:jar:1.0.0-M20:compile
[INFO] |  |  |  |  +- org.apache.curator:curator-client:jar:2.6.0:compile
[INFO] |  |  |  |  \- org.htrace:htrace-core:jar:3.0.4:compile
[INFO] |  |  |  +- org.apache.hadoop:hadoop-hdfs:jar:2.6.5:compile
[INFO] |  |  |  |  +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
[INFO] |  |  |  |  \- xerces:xercesImpl:jar:2.9.1:compile
[INFO] |  |  |  |     \- xml-apis:xml-apis:jar:1.4.01:compile
[INFO] |  |  |  +- org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.6.5:compile
[INFO] |  |  |  |  +- org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.6.5:compile
[INFO] |  |  |  |  |  +- org.apache.hadoop:hadoop-yarn-client:jar:2.6.5:compile
[INFO] |  |  |  |  |  \- org.apache.hadoop:hadoop-yarn-server-common:jar:2.6.5:compile
[INFO] |  |  |  |  \- org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.6.5:compile
[INFO] |  |  |  +- org.apache.hadoop:hadoop-yarn-api:jar:2.6.5:compile
[INFO] |  |  |  +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.6.5:compile
[INFO] |  |  |  |  \- org.apache.hadoop:hadoop-yarn-common:jar:2.6.5:compile
[INFO] |  |  |  |     +- javax.xml.bind:jaxb-api:jar:2.3.1:compile
[INFO] |  |  |  |     |  \- javax.activation:javax.activation-api:jar:1.2.0:compile
[INFO] |  |  |  |     +- org.codehaus.jackson:jackson-jaxrs:jar:1.9.13:compile
[INFO] |  |  |  |     \- org.codehaus.jackson:jackson-xc:jar:1.9.13:compile
[INFO] |  |  |  +- org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.6.5:compile
[INFO] |  |  |  \- org.apache.hadoop:hadoop-annotations:jar:2.6.5:compile
[INFO] |  |  +- org.apache.spark:spark-launcher_2.11:jar:2.4.3:compile
[INFO] |  |  +- org.apache.spark:spark-kvstore_2.11:jar:2.4.3:compile
[INFO] |  |  |  \- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:compile
[INFO] |  |  +- org.apache.spark:spark-network-common_2.11:jar:2.4.3:compile
[INFO] |  |  +- org.apache.spark:spark-network-shuffle_2.11:jar:2.4.3:compile
[INFO] |  |  +- org.apache.spark:spark-unsafe_2.11:jar:2.4.3:compile
[INFO] |  |  +- javax.activation:activation:jar:1.1.1:compile
[INFO] |  |  +- org.apache.curator:curator-recipes:jar:2.6.0:compile
[INFO] |  |  |  \- org.apache.curator:curator-framework:jar:2.6.0:compile
[INFO] |  |  +- org.apache.zookeeper:zookeeper:jar:3.4.6:compile
[INFO] |  |  +- javax.servlet:javax.servlet-api:jar:3.1.0:compile
[INFO] |  |  +- org.apache.commons:commons-lang3:jar:3.7:compile
[INFO] |  |  +- com.google.code.findbugs:jsr305:jar:1.3.9:compile
[INFO] |  |  +- org.slf4j:jul-to-slf4j:jar:1.7.26:compile
[INFO] |  |  +- org.slf4j:jcl-over-slf4j:jar:1.7.26:compile
[INFO] |  |  +- log4j:log4j:jar:1.2.17:compile
[INFO] |  |  +- org.slf4j:slf4j-log4j12:jar:1.7.26:compile
[INFO] |  |  +- com.ning:compress-lzf:jar:1.0.3:compile
[INFO] |  |  +- org.xerial.snappy:snappy-java:jar:1.1.7.3:compile
[INFO] |  |  +- org.lz4:lz4-java:jar:1.4.0:compile
[INFO] |  |  +- com.github.luben:zstd-jni:jar:1.3.2-2:compile
[INFO] |  |  +- org.roaringbitmap:RoaringBitmap:jar:0.7.45:compile
[INFO] |  |  |  \- org.roaringbitmap:shims:jar:0.7.45:compile
[INFO] |  |  +- commons-net:commons-net:jar:3.1:compile
[INFO] |  |  +- org.json4s:json4s-jackson_2.11:jar:3.5.3:compile
[INFO] |  |  |  \- org.json4s:json4s-core_2.11:jar:3.5.3:compile
[INFO] |  |  |     +- org.json4s:json4s-ast_2.11:jar:3.5.3:compile
[INFO] |  |  |     +- org.json4s:json4s-scalap_2.11:jar:3.5.3:compile
[INFO] |  |  |     \- org.scala-lang.modules:scala-xml_2.11:jar:1.0.6:compile
[INFO] |  |  +- org.glassfish.jersey.core:jersey-client:jar:2.26:compile
[INFO] |  |  +- org.glassfish.jersey.core:jersey-server:jar:2.26:compile
[INFO] |  |  |  \- org.glassfish.jersey.media:jersey-media-jaxb:jar:2.26:compile
[INFO] |  |  +- org.glassfish.jersey.containers:jersey-container-servlet:jar:2.26:compile
[INFO] |  |  +- org.glassfish.jersey.containers:jersey-container-servlet-core:jar:2.26:compile
[INFO] |  |  +- io.netty:netty:jar:3.9.9.Final:compile
[INFO] |  |  +- com.clearspring.analytics:stream:jar:2.7.0:compile
[INFO] |  |  +- io.dropwizard.metrics:metrics-core:jar:3.2.6:compile
[INFO] |  |  +- io.dropwizard.metrics:metrics-jvm:jar:3.2.6:compile
[INFO] |  |  +- io.dropwizard.metrics:metrics-json:jar:3.2.6:compile
[INFO] |  |  +- io.dropwizard.metrics:metrics-graphite:jar:3.2.6:compile
[INFO] |  |  +- com.fasterxml.jackson.module:jackson-module-scala_2.11:jar:2.9.8:compile
[INFO] |  |  |  \- com.fasterxml.jackson.module:jackson-module-paranamer:jar:2.9.8:compile
[INFO] |  |  +- org.apache.ivy:ivy:jar:2.4.0:compile
[INFO] |  |  +- oro:oro:jar:2.0.8:compile
[INFO] |  |  +- net.razorvine:pyrolite:jar:4.13:compile
[INFO] |  |  +- net.sf.py4j:py4j:jar:0.10.7:compile
[INFO] |  |  \- org.apache.commons:commons-crypto:jar:1.0.0:compile
[INFO] |  +- org.apache.spark:spark-streaming_2.11:jar:2.4.3:compile
[INFO] |  +- org.apache.spark:spark-sql_2.11:jar:2.4.3:compile
[INFO] |  |  +- com.univocity:univocity-parsers:jar:2.7.3:compile
[INFO] |  |  +- org.apache.spark:spark-sketch_2.11:jar:2.4.3:compile
[INFO] |  |  +- org.apache.spark:spark-catalyst_2.11:jar:2.4.3:compile
[INFO] |  |  |  +- org.codehaus.janino:janino:jar:3.0.12:compile
[INFO] |  |  |  +- org.codehaus.janino:commons-compiler:jar:3.0.9:compile
[INFO] |  |  |  \- org.antlr:antlr4-runtime:jar:4.7:compile
[INFO] |  |  +- org.apache.orc:orc-core:jar:nohive:1.5.5:compile
[INFO] |  |  |  +- org.apache.orc:orc-shims:jar:1.5.5:compile
[INFO] |  |  |  +- com.google.protobuf:protobuf-java:jar:2.5.0:compile
[INFO] |  |  |  +- commons-lang:commons-lang:jar:2.6:compile
[INFO] |  |  |  \- io.airlift:aircompressor:jar:0.10:compile
[INFO] |  |  +- org.apache.orc:orc-mapreduce:jar:nohive:1.5.5:compile
[INFO] |  |  +- org.apache.parquet:parquet-column:jar:1.10.1:compile
[INFO] |  |  |  +- org.apache.parquet:parquet-common:jar:1.10.1:compile
[INFO] |  |  |  \- org.apache.parquet:parquet-encoding:jar:1.10.1:compile
[INFO] |  |  +- org.apache.parquet:parquet-hadoop:jar:1.10.1:compile
[INFO] |  |  |  +- org.apache.parquet:parquet-format:jar:2.4.0:compile
[INFO] |  |  |  \- org.apache.parquet:parquet-jackson:jar:1.10.1:compile
[INFO] |  |  \- org.apache.arrow:arrow-vector:jar:0.10.0:compile
[INFO] |  |     +- org.apache.arrow:arrow-format:jar:0.10.0:compile
[INFO] |  |     +- org.apache.arrow:arrow-memory:jar:0.10.0:compile
[INFO] |  |     +- com.carrotsearch:hppc:jar:0.7.2:compile
[INFO] |  |     \- com.vlkan:flatbuffers:jar:1.2.0-3f79e055:compile
[INFO] |  +- org.apache.spark:spark-graphx_2.11:jar:2.4.3:compile
[INFO] |  |  +- com.github.fommil.netlib:core:jar:1.1.2:compile
[INFO] |  |  \- net.sourceforge.f2j:arpack_combined_all:jar:0.1:compile
[INFO] |  +- org.apache.spark:spark-mllib-local_2.11:jar:2.4.3:compile
[INFO] |  +- org.scalanlp:breeze_2.11:jar:0.13.2:compile
[INFO] |  |  +- org.scalanlp:breeze-macros_2.11:jar:0.13.2:compile
[INFO] |  |  +- net.sf.opencsv:opencsv:jar:2.3:compile
[INFO] |  |  +- com.github.rwl:jtransforms:jar:2.4.0:compile
[INFO] |  |  +- org.spire-math:spire_2.11:jar:0.13.0:compile
[INFO] |  |  |  +- org.spire-math:spire-macros_2.11:jar:0.13.0:compile
[INFO] |  |  |  \- org.typelevel:machinist_2.11:jar:0.6.1:compile
[INFO] |  |  \- com.chuusai:shapeless_2.11:jar:2.3.2:compile
[INFO] |  |     \- org.typelevel:macro-compat_2.11:jar:1.1.1:compile
[INFO] |  +- org.apache.commons:commons-math3:jar:3.4.1:compile
[INFO] |  +- org.apache.spark:spark-tags_2.11:jar:2.4.3:compile
[INFO] |  \- org.spark-project.spark:unused:jar:1.0.0:compile
[INFO] +- org.glassfish.jersey.inject:jersey-hk2:jar:2.26:compile
[INFO] |  +- org.glassfish.jersey.core:jersey-common:jar:2.26:compile
[INFO] |  |  +- javax.ws.rs:javax.ws.rs-api:jar:2.1:compile
[INFO] |  |  +- org.glassfish.hk2.external:javax.inject:jar:2.5.0-b42:compile
[INFO] |  |  \- org.glassfish.hk2:osgi-resource-locator:jar:1.0.1:compile
[INFO] |  \- org.glassfish.hk2:hk2-locator:jar:2.5.0-b42:compile
[INFO] |     +- org.glassfish.hk2.external:aopalliance-repackaged:jar:2.5.0-b42:compile
[INFO] |     +- org.glassfish.hk2:hk2-api:jar:2.5.0-b42:compile
[INFO] |     |  \- javax.inject:javax.inject:jar:1:compile
[INFO] |     +- org.glassfish.hk2:hk2-utils:jar:2.5.0-b42:compile
[INFO] |     \- org.javassist:javassist:jar:3.22.0-CR2:compile
[INFO] +- org.springframework.data:spring-data-cassandra:jar:2.0.14.RELEASE:compile
[INFO] |  +- org.springframework:spring-context:jar:5.0.13.RELEASE:compile
[INFO] |  +- org.springframework:spring-beans:jar:5.0.13.RELEASE:compile
[INFO] |  +- org.springframework:spring-tx:jar:5.0.13.RELEASE:compile
[INFO] |  +- org.springframework:spring-expression:jar:5.0.13.RELEASE:compile
[INFO] |  +- org.springframework.data:spring-data-commons:jar:2.0.14.RELEASE:compile
[INFO] |  +- com.datastax.cassandra:cassandra-driver-core:jar:3.4.0:compile
[INFO] |  |  +- io.netty:netty-handler:jar:4.1.34.Final:compile
[INFO] |  |  |  +- io.netty:netty-common:jar:4.1.34.Final:compile
[INFO] |  |  |  +- io.netty:netty-buffer:jar:4.1.34.Final:compile
[INFO] |  |  |  +- io.netty:netty-transport:jar:4.1.34.Final:compile
[INFO] |  |  |  |  \- io.netty:netty-resolver:jar:4.1.34.Final:compile
[INFO] |  |  |  \- io.netty:netty-codec:jar:4.1.34.Final:compile
[INFO] |  |  +- com.github.jnr:jnr-ffi:jar:2.0.7:compile
[INFO] |  |  |  +- com.github.jnr:jffi:jar:1.2.10:compile
[INFO] |  |  |  +- com.github.jnr:jffi:jar:native:1.2.10:runtime
[INFO] |  |  |  +- org.ow2.asm:asm:jar:5.0.3:compile
[INFO] |  |  |  +- org.ow2.asm:asm-commons:jar:5.0.3:compile
[INFO] |  |  |  +- org.ow2.asm:asm-analysis:jar:5.0.3:compile
[INFO] |  |  |  +- org.ow2.asm:asm-tree:jar:5.0.3:compile
[INFO] |  |  |  +- org.ow2.asm:asm-util:jar:5.0.3:compile
[INFO] |  |  |  \- com.github.jnr:jnr-x86asm:jar:1.0.2:compile
[INFO] |  |  \- com.github.jnr:jnr-posix:jar:3.0.27:compile
[INFO] |  |     \- com.github.jnr:jnr-constants:jar:0.9.0:compile
[INFO] |  +- com.google.guava:guava:jar:19.0:compile
[INFO] |  \- org.slf4j:slf4j-api:jar:1.7.26:compile
[INFO] \- com.datastax.spark:spark-cassandra-connector-unshaded_2.11:jar:2.4.0:compile
[INFO]    +- org.scala-lang:scala-library:jar:2.11.12:compile
[INFO]    +- com.datastax.cassandra:cassandra-driver-mapping:jar:3.4.0:compile
[INFO]    +- commons-beanutils:commons-beanutils:jar:1.9.3:compile
[INFO]    |  \- commons-collections:commons-collections:jar:3.2.2:compile
[INFO]    +- org.joda:joda-convert:jar:1.2:compile
[INFO]    +- joda-time:joda-time:jar:2.9.9:compile
[INFO]    +- io.netty:netty-all:jar:4.1.34.Final:compile
[INFO]    +- com.twitter:jsr166e:jar:1.1.0:compile
[INFO]    \- org.scala-lang:scala-reflect:jar:2.11.12:compile
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS

score 0 · Accepted Answer

尝试用 log4j 替换 logback（删除 logback 依赖项），至少它在我们的类似情况下有所帮助。

score 0 · Accepted Answer

依赖项之间似乎存在冲突，但不知道确切是什么！这就是我所做的：Spring - Spark：jars/dependencies 之间的冲突

编辑：

正如上面链接中所解释的那样，我只需要更改它：

<plugin>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-maven-plugin</artifactId>
            <version>2.0.9.RELEASE</version>
            <executions>
            <execution>
                <goals>
                    <goal>repackage</goal>
                </goals>
            </execution>
        </executions>
            <configuration>
                <requiresUnpack>
                    <dependency>
                        <groupId>org.apache.spark</groupId>
                        <artifactId>spark-core_2.11</artifactId>
                    </dependency>
                    <dependency>
                        <groupId>org.apache.spark</groupId>
                        <artifactId>spark-streaming_2.11</artifactId>
                    </dependency>

                </requiresUnpack>
            </configuration>
        </plugin>

有了这个：

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-compiler-plugin</artifactId>
    <version>3.8.0</version>
    <configuration>
        <source>1.8</source>
        <target>1.8</target>
    </configuration>
</plugin>
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>2.4.1</version>
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-maven-plugin</artifactId>
            <version>2.0.9.RELEASE</version>
        </dependency>
    </dependencies>
    <configuration>
        <keepDependenciesWithProvidedScope>true</keepDependenciesWithProvidedScope>
        <createDependencyReducedPom>true</createDependencyReducedPom>
        <filters>
            <filter>
                <artifact>*:*</artifact>
                <excludes>
                    <exclude>META-INF/*.SF</exclude>
                    <exclude>META-INF/*.DSA</exclude>
                    <exclude>META-INF/*.RSA</exclude>
                </excludes>
            </filter>
        </filters>
    </configuration>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>shade</goal>
            </goals>
            <configuration>
                <transformers>
                    <transformer
                        implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                        <resource>META-INF/spring.handlers</resource>
                    </transformer>
                    <transformer
                        implementation="org.springframework.boot.maven.PropertiesMergingResourceTransformer">
                        <resource>META-INF/spring.factories</resource>
                    </transformer>
                    <transformer
                        implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                        <resource>META-INF/spring.schemas</resource>
                    </transformer>
                    <transformer
                        implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
                    <transformer
                        implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                        <mainClass>com.example.app.Mtbapp1</mainClass>
                    </transformer>
                </transformers>
            </configuration>
        </execution>
    </executions>
</plugin>

apache-spark - 如何修复“ClassCastException：无法分配实例”-在本地工作但不能在集群上独立运行

2 回答 2

Related

Reference