当前位置: 首页 > 知识库问答 >
问题:

Dataproc依赖项冲突-google api客户端

华烈
2023-03-14

我正在构建一个用于从云存储获取加密机密的库(在Scala中,使用Java客户端)。我正在使用以下google库:

"com.google.apis"  % "google-api-services-cloudkms" % "v1-rev26-1.23.0" exclude("com.google.guava", "guava-jdk5"),
"com.google.cloud" % "google-cloud-storage"         % "1.14.0",

一切都在本地正常工作,但是当我试图在Dataproc中运行我的代码时,我得到了以下错误:

Exception in thread "main" java.lang.NoSuchMethodError: com.google.api.client.googleapis.services.json.AbstractGoogleJsonClient$Builder.setBatchPath(Ljava/lang/String;)Lcom/google/api/client/googleapis/services/AbstractGoogleClient$Builder;
    at com.google.api.services.cloudkms.v1.CloudKMS$Builder.setBatchPath(CloudKMS.java:4250)
    at com.google.api.services.cloudkms.v1.CloudKMS$Builder.<init>(CloudKMS.java:4229)
    at gcp.encryption.EncryptedSecretsUser$class.clients(EncryptedSecretsUser.scala:111)
    at gcp.encryption.EncryptedSecretsUser$class.getEncryptedSecrets(EncryptedSecretsUser.scala:62)

我的代码中有问题的行是:

val kms: CloudKMS = new CloudKMS.Builder(credential.getTransport,
      credential.getJsonFactory,
      credential)
      .setApplicationName("Encrypted Secrets User")
      .build()

我在文档中看到,一些google库在Dataproc上可用(我使用的是图像版本为1.2.15的Spark集群)。但就我所见,GoogleAPI客户端的可传递依赖项与我在本地使用的相同(1.23.0)。那么为什么找不到这种方法呢?

在Dataproc上运行时,我应该以不同的方式设置依赖项吗?

最终在另一个项目中解决了这个问题。事实证明,除了着色所有的google依赖项(包括gcs-连接器!!),你还必须在JVM上注册你的着色类来处理gs://file-system。下面是适用于我的maven配置,类似的东西可以用sbt实现:

父POM:

<project xmlns="http://maven.apache.org/POM/4.0.0"...>
...
<properties>
    <!-- Spark version -->
    <spark.version>[2.2.1]</spark.version>
    <!-- Jackson-libs version pulled in by spark -->
    <jackson.version>[2.6.5]</jackson.version>
    <!-- Avro version pulled in by jackson -->
    <avro.version>[1.7.7]</avro.version>
    <!-- Kryo-shaded version pulled in by spark -->
    <kryo.version>[3.0.3]</kryo.version>
    <!-- Apache commons-lang version pulled in by spark -->
    <commons.lang.version>2.6</commons.lang.version>

    <!-- TODO: need to shade google libs because of version-conflicts on Dataproc. Remove this when Dataproc 1.3/2.0 is released -->
    <bigquery-conn.version>[0.10.6-hadoop2]</bigquery-conn.version>
    <gcs-conn.version>[1.6.5-hadoop2]</gcs-conn.version>
    <google-storage.version>[1.29.0]</google-storage.version>
    <!-- The guava version we want to use -->
    <guava.version>[23.2-jre]</guava.version>
    <!-- The google api version used by the google-cloud-storage lib -->
    <api-client.version>[1.23.0]</api-client.version>
    <!-- The google-api-services-storage version used by the google-cloud-storage lib -->
    <storage-api.version>[v1-rev114-1.23.0]</storage-api.version>

    <!-- Picked up by compiler and resource plugins -->
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>

...

<build>
    <pluginManagement>
        <plugins>
...

        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>3.1.1</version>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                    <configuration>
                        <minimizeJar>true</minimizeJar>
                        <filters>
                            <filter>
                                <artifact>com.google.**:*</artifact>
                                <includes>
                                    <include>**</include>
                                </includes>
                            </filter>
                            <filter>
                                <artifact>com.google.cloud.bigdataoss:gcs-connector</artifact>
                                <excludes>
                                    <!-- Register a provider with the shaded name instead-->
                                    <exclude>META-INF/services/org.apache.hadoop.fs.FileSystem</exclude>
                                </excludes>
                            </filter>
                        </filters>
                        <artifactSet>
                            <includes>
                                <include>com.google.*:*</include>
                            </includes>
                            <excludes>
                                <exclude>com.google.code.findbugs:jsr305</exclude>
                            </excludes>
                        </artifactSet>
                        <relocations>
                            <relocation>
                                <pattern>com.google</pattern>
                                <shadedPattern>com.shaded.google</shadedPattern>
                            </relocation>
                        </relocations>
                    </configuration>
                </execution>
            </executions>
        </plugin>
...
    </plugins>
</build>

<dependencyManagement>
    <dependencies>
        <dependency>
...
            <groupId>com.google.cloud.bigdataoss</groupId>
            <artifactId>gcs-connector</artifactId>
            <version>${gcs-conn.version}</version>
            <exclusions>
                <!-- conflicts with Spark dependencies -->
                <exclusion>
                    <groupId>org.apache.hadoop</groupId>
                    <artifactId>hadoop-common</artifactId>
                </exclusion>
                <!-- conflicts with Spark dependencies -->
                <exclusion>
                    <groupId>org.apache.hadoop</groupId>
                    <artifactId>hadoop-mapreduce-client-core</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>com.google.guava</groupId>
                    <artifactId>guava</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <!-- Avoid conflict with the version pulled in by the GCS-connector on Dataproc -->
            <groupId>com.google.apis</groupId>
            <artifactId>google-api-services-storage</artifactId>
            <version>${storage-api.version}</version>
        </dependency>
        <dependency>
            <groupId>commons-lang</groupId>
            <artifactId>commons-lang</artifactId>
            <version>${commons.lang.version}</version>
        </dependency>
        <dependency>
            <groupId>com.esotericsoftware</groupId>
            <artifactId>kryo-shaded</artifactId>
            <version>${kryo.version}</version>
        </dependency>
        <dependency>
            <groupId>com.fasterxml.jackson.core</groupId>
            <artifactId>jackson-databind</artifactId>
            <version>${jackson.version}</version>
        </dependency>
        <dependency>
            <groupId>com.google.api-client</groupId>
            <artifactId>google-api-client</artifactId>
            <version>${api-client.version}</version>
        </dependency>
        <dependency>
            <groupId>com.google.guava</groupId>
            <artifactId>guava</artifactId>
            <version>${guava.version}</version>
        </dependency>
    </dependencies>
</dependencyManagement>

<dependencies>
    <dependency>
        <groupId>com.google.cloud</groupId>
        <artifactId>google-cloud-storage</artifactId>
        <version>${google-storage.version}</version>
        <exclusions>
            <!-- conflicts with Spark dependencies -->
            <exclusion>
                <groupId>com.google.guava</groupId>
                <artifactId>guava</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>com.google.guava</groupId>
        <artifactId>guava</artifactId>
    </dependency>
...
</dependencies>

...
</project>

儿童POM:

    <dependencies>
    <!-- Libraries available on dataproc -->
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.11</artifactId>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.11</artifactId>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>com.google.cloud.bigdataoss</groupId>
        <artifactId>gcs-connector</artifactId>
    </dependency>
    <dependency>
        <groupId>com.esotericsoftware</groupId>
        <artifactId>kryo-shaded</artifactId>
        <scope>provided</scope><!-- Pulled in by spark -->
    </dependency>
    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-databind</artifactId>
        <scope>provided</scope><!-- Pulled in by spark -->
    </dependency>
</dependencies>

并添加一个名为org.apache.hadoop.fs.FileSystem的文件在path/to/you-project/src/main/Resources/META-INF/service下,其中包含您的阴影类的名称,例如:

# WORKAROUND FOR DEPENDENCY CONFLICTS ON DATAPROC
#
# Use the shaded class as a provider for the gs:// file system
#

com.shaded.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem

(请注意,此文件已从父POM中的gcs连接器库中筛选出来)

共有1个答案

唐阳飇
2023-03-14

这可能并不明显,但最新稳定的GCS连接器中的google api客户端版本实际上是1.20.0

原因是,这是将api客户端版本前推到1.23.0的提交,它是一系列提交的一部分,包括此依赖项着色提交,其总体目标是不再将可传递依赖项泄漏到作业类路径中,以避免将来出现版本冲突问题,代价是每个人都必须自带包含完整api客户机依赖项的胖罐子。

然而,事实证明,许多人已经开始依赖GCS连接器提供的api客户端来运行类路径,所以有一些生产工作负载在小版本升级中无法承受这样的变化;因此,升级后的GCS连接器它使用1.23.0,但也对其进行阴影处理,使其不再出现在作业类路径中,这是为未来的Dataproc 1.3或2.0版本保留的。

就你而言,您可以尝试使用1.20.0版本的依赖项(您可能还必须降级所包含的googlecloudstorage依赖项的版本,尽管1.22.0版本的依赖项在假设没有突破性更改的情况下仍然可以工作,因为setBatchPath实际上只在1.23.0中引入),或者,您可以尝试使用sbt assembly来隐藏您自己的所有依赖项。

我们可以验证setBatchPath仅在1.23.0中引入:

$ javap -cp google-api-client-1.22.0.jar com.google.api.client.googleapis.services.AbstractGoogleClient.Builder | grep set
  public com.google.api.client.googleapis.services.AbstractGoogleClient$Builder setRootUrl(java.lang.String);
  public com.google.api.client.googleapis.services.AbstractGoogleClient$Builder setServicePath(java.lang.String);
  public com.google.api.client.googleapis.services.AbstractGoogleClient$Builder setGoogleClientRequestInitializer(com.google.api.client.googleapis.services.GoogleClientRequestInitializer);
  public com.google.api.client.googleapis.services.AbstractGoogleClient$Builder setHttpRequestInitializer(com.google.api.client.http.HttpRequestInitializer);
  public com.google.api.client.googleapis.services.AbstractGoogleClient$Builder setApplicationName(java.lang.String);
  public com.google.api.client.googleapis.services.AbstractGoogleClient$Builder setSuppressPatternChecks(boolean);
  public com.google.api.client.googleapis.services.AbstractGoogleClient$Builder setSuppressRequiredParameterChecks(boolean);
  public com.google.api.client.googleapis.services.AbstractGoogleClient$Builder setSuppressAllChecks(boolean);

$ javap -cp google-api-client-1.23.0.jar com.google.api.client.googleapis.services.AbstractGoogleClient.Builder | grep set
  public com.google.api.client.googleapis.services.AbstractGoogleClient$Builder setRootUrl(java.lang.String);
  public com.google.api.client.googleapis.services.AbstractGoogleClient$Builder setServicePath(java.lang.String);
  public com.google.api.client.googleapis.services.AbstractGoogleClient$Builder setBatchPath(java.lang.String);
  public com.google.api.client.googleapis.services.AbstractGoogleClient$Builder setGoogleClientRequestInitializer(com.google.api.client.googleapis.services.GoogleClientRequestInitializer);
  public com.google.api.client.googleapis.services.AbstractGoogleClient$Builder setHttpRequestInitializer(com.google.api.client.http.HttpRequestInitializer);
  public com.google.api.client.googleapis.services.AbstractGoogleClient$Builder setApplicationName(java.lang.String);
  public com.google.api.client.googleapis.services.AbstractGoogleClient$Builder setSuppressPatternChecks(boolean);
  public com.google.api.client.googleapis.services.AbstractGoogleClient$Builder setSuppressRequiredParameterChecks(boolean);
  public com.google.api.client.googleapis.services.AbstractGoogleClient$Builder setSuppressAllChecks(boolean);
 类似资料:
  • 我有一个具有以下依赖项的GWT项目 > Gin 2.1.2依赖于guice 3.0,而owlapi 4.0.2依赖于guice 4.0-beta。 gin在客户端使用,而owlapi在服务器端使用。 我有什么办法才能让这件事成功?我是否可以在保留正常运行的GWT devmode的同时使用依赖关系范围?

  • 由一些模块组成的maven项目。我的一个模块正在使用google版本的guava依赖项。现在,我正在我的项目中集成另一个模块,该模块也使用guava但版本。 因此,我希望新模块使用guava版本,而其余项目使用guava版本。我尝试将guava的添加到新模块中,但没有成功。 更新:@Guillaume Darmont的答案为不同的模块解决了问题。但现在我的问题是,新模块有两个依赖项,其中一个使用g

  • 错误为:java.security.NoSuchProviderException:JCE无法验证提供程序BC。注意,我已经添加了这段代码:security.addProvider(new BouncyCastleProvider()); 这在使用spring boot embedded tomcat时可以很好地工作,但在导出到运行在wildfly服务器上的war文件时就不行了。 下面是我如何宣布

  • 因为Flatter\u math\u fork 获取:^4.6.1获取存储:2.0.3 dio:^4.0.4 dio\u http\u缓存:0.3.0 webview_flutter:^2.8.0flutter_html:^2.0.0url_launcher:^6.0.17 firebase\u消息:^11.2.4 cloud\u firestore:^3.1.5 firebase\u认证:^3.

  • 我试图向spark提交一个jar,但我的jar包含与spark内置jar(snakeyml和其他)冲突的依赖项。 我使用以下命令提交: 但我还是有同样的例外

  • 问题内容: 我有以下代码: 这正常工作。但是,我还需要在pom.xml中添加Xalan作为依赖项,当我这样做时,上面的代码现在会引发错误: 我认为这与Xalan的jar中有一个不同的Transformer实现有关。在不更改上述代码并将Xalan保留为依赖关系的情况下,如何解决此冲突? 问题答案: 从Xalan中排除Xerces可以解决此问题: