java - 无法验证 GCS 存储桶和 PKIX 路径构建失败 创建和暂存 GCP 数据流模板时出错

Question

我正在使用以下命令在云存储中创建和暂存 gcp 数据流模板：

mvn -X compile exec:java -Dexec.mainClass=main.java.TemplatePipeline -Dexec.args="--runner=DataflowRunner --project=hv-hcap-development --stagingLocation=gs://my-bucket/staging --templateLocation=gs://my-bucket/templates/template1"

构建失败并出现以下错误

[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  3.563 s
[INFO] Finished at: 2020-07-17T17:32:05-07:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:3.0.0:java (default-cli) on project dataflow-template: An exception occured while executing the Java class. Failed to construct instance from factory method DataflowRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions): InvocationTargetException: Unable to verify that GCS bucket gs://bucket-df-job exists. sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.codehaus.mojo:exec-maven-plugin:3.0.0:java (default-cli) on project dataflow-template: An exception occured while executing the Java class. Failed to construct instance from factory method DataflowRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Caused by: org.apache.maven.plugin.MojoExecutionException: An exception occured while executing the Java class. Failed to construct instance from factory method DataflowRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions)
    at org.codehaus.mojo.exec.ExecJavaMojo.execute (ExecJavaMojo.java:311)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:193)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)
Caused by: java.lang.RuntimeException: Failed to construct instance from factory method DataflowRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions)
    at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod (InstanceBuilder.java:224)
    at org.apache.beam.sdk.util.InstanceBuilder.build (InstanceBuilder.java:155)
    at org.apache.beam.sdk.PipelineRunner.fromOptions (PipelineRunner.java:55)
    at org.apache.beam.sdk.Pipeline.create (Pipeline.java:149)
    at main.java.TemplatePipeline.main (TemplatePipeline.java:95)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:254)
    at java.lang.Thread.run (Thread.java:748)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod (InstanceBuilder.java:214)
    at org.apache.beam.sdk.util.InstanceBuilder.build (InstanceBuilder.java:155)
    at org.apache.beam.sdk.PipelineRunner.fromOptions (PipelineRunner.java:55)
    at org.apache.beam.sdk.Pipeline.create (Pipeline.java:149)
    at main.java.TemplatePipeline.main (TemplatePipeline.java:95)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:254)
    at java.lang.Thread.run (Thread.java:748)
Caused by: java.lang.RuntimeException: Unable to verify that GCS bucket gs://bucket-df-job exists.
    at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.verifyPathIsAccessible (GcsPathValidator.java:86)
    at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.validateOutputFilePrefixSupported (GcsPathValidator.java:53)
    at org.apache.beam.runners.dataflow.DataflowRunner.fromOptions (DataflowRunner.java:274)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod (InstanceBuilder.java:214)
    at org.apache.beam.sdk.util.InstanceBuilder.build (InstanceBuilder.java:155)
    at org.apache.beam.sdk.PipelineRunner.fromOptions (PipelineRunner.java:55)
    at org.apache.beam.sdk.Pipeline.create (Pipeline.java:149)
    at main.java.TemplatePipeline.main (TemplatePipeline.java:95)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:254)
    at java.lang.Thread.run (Thread.java:748)
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.ssl.Alerts.getSSLException (Alerts.java:198)
    at sun.security.ssl.SSLSocketImpl.fatal (SSLSocketImpl.java:1967)
    at sun.security.ssl.Handshaker.fatalSE (Handshaker.java:331)
    at sun.security.ssl.Handshaker.fatalSE (Handshaker.java:325)
    at sun.security.ssl.ClientHandshaker.serverCertificate (ClientHandshaker.java:1689)
    at sun.security.ssl.ClientHandshaker.processMessage (ClientHandshaker.java:226)
    at sun.security.ssl.Handshaker.processLoop (Handshaker.java:1082)
    at sun.security.ssl.Handshaker.process_record (Handshaker.java:1010)
    at sun.security.ssl.SSLSocketImpl.readRecord (SSLSocketImpl.java:1079)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake (SSLSocketImpl.java:1388)
    at sun.security.ssl.SSLSocketImpl.startHandshake (SSLSocketImpl.java:1416)
    at sun.security.ssl.SSLSocketImpl.startHandshake (SSLSocketImpl.java:1400)
    at sun.net.www.protocol.https.HttpsClient.afterConnect (HttpsClient.java:559)
    at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect (AbstractDelegateHttpsURLConnection.java:185)
    at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0 (HttpURLConnection.java:1340)
    at sun.net.www.protocol.http.HttpURLConnection.getOutputStream (HttpURLConnection.java:1315)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream (HttpsURLConnectionImpl.java:264)
    at com.google.api.client.http.javanet.NetHttpRequest.execute (NetHttpRequest.java:113)
    at com.google.api.client.http.javanet.NetHttpRequest.execute (NetHttpRequest.java:84)
    at com.google.api.client.http.HttpRequest.execute (HttpRequest.java:1012)
    at com.google.auth.oauth2.UserCredentials.refreshAccessToken (UserCredentials.java:203)
    at com.google.auth.oauth2.OAuth2Credentials.refresh (OAuth2Credentials.java:157)
    at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata (OAuth2Credentials.java:145)
    at com.google.auth.oauth2.UserCredentials.getRequestMetadata (UserCredentials.java:281)
    at com.google.auth.http.HttpCredentialsAdapter.initialize (HttpCredentialsAdapter.java:91)
    at com.google.cloud.hadoop.util.ChainingHttpRequestInitializer.initialize (ChainingHttpRequestInitializer.java:52)
    at com.google.api.client.http.HttpRequestFactory.buildRequest (HttpRequestFactory.java:88)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.buildHttpRequest (AbstractGoogleClientRequest.java:422)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed (AbstractGoogleClientRequest.java:541)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed (AbstractGoogleClientRequest.java:474)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute (AbstractGoogleClientRequest.java:591)
    at com.google.cloud.hadoop.util.ResilientOperation$AbstractGoogleClientRequestExecutor.call (ResilientOperation.java:171)
    at com.google.cloud.hadoop.util.ResilientOperation.retry (ResilientOperation.java:67)
    at org.apache.beam.sdk.extensions.gcp.util.GcsUtil.getBucket (GcsUtil.java:521)
    at org.apache.beam.sdk.extensions.gcp.util.GcsUtil.bucketAccessible (GcsUtil.java:509)
    at org.apache.beam.sdk.extensions.gcp.util.GcsUtil.bucketAccessible (GcsUtil.java:482)
    at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.verifyPathIsAccessible (GcsPathValidator.java:83)
    at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.validateOutputFilePrefixSupported (GcsPathValidator.java:53)
    at org.apache.beam.runners.dataflow.DataflowRunner.fromOptions (DataflowRunner.java:274)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod (InstanceBuilder.java:214)
    at org.apache.beam.sdk.util.InstanceBuilder.build (InstanceBuilder.java:155)
    at org.apache.beam.sdk.PipelineRunner.fromOptions (PipelineRunner.java:55)
    at org.apache.beam.sdk.Pipeline.create (Pipeline.java:149)
    at main.java.TemplatePipeline.main (TemplatePipeline.java:95)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:254)
    at java.lang.Thread.run (Thread.java:748)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.validator.PKIXValidator.doBuild (PKIXValidator.java:450)
    at sun.security.validator.PKIXValidator.engineValidate (PKIXValidator.java:317)
    at sun.security.validator.Validator.validate (Validator.java:262)
    at sun.security.ssl.X509TrustManagerImpl.validate (X509TrustManagerImpl.java:330)
    at sun.security.ssl.X509TrustManagerImpl.checkTrusted (X509TrustManagerImpl.java:227)
    at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted (X509TrustManagerImpl.java:132)
    at sun.security.ssl.ClientHandshaker.serverCertificate (ClientHandshaker.java:1671)
    at sun.security.ssl.ClientHandshaker.processMessage (ClientHandshaker.java:226)
    at sun.security.ssl.Handshaker.processLoop (Handshaker.java:1082)
    at sun.security.ssl.Handshaker.process_record (Handshaker.java:1010)
    at sun.security.ssl.SSLSocketImpl.readRecord (SSLSocketImpl.java:1079)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake (SSLSocketImpl.java:1388)
    at sun.security.ssl.SSLSocketImpl.startHandshake (SSLSocketImpl.java:1416)
    at sun.security.ssl.SSLSocketImpl.startHandshake (SSLSocketImpl.java:1400)
    at sun.net.www.protocol.https.HttpsClient.afterConnect (HttpsClient.java:559)
    at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect (AbstractDelegateHttpsURLConnection.java:185)
    at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0 (HttpURLConnection.java:1340)
    at sun.net.www.protocol.http.HttpURLConnection.getOutputStream (HttpURLConnection.java:1315)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream (HttpsURLConnectionImpl.java:264)
    at com.google.api.client.http.javanet.NetHttpRequest.execute (NetHttpRequest.java:113)
    at com.google.api.client.http.javanet.NetHttpRequest.execute (NetHttpRequest.java:84)
    at com.google.api.client.http.HttpRequest.execute (HttpRequest.java:1012)
    at com.google.auth.oauth2.UserCredentials.refreshAccessToken (UserCredentials.java:203)
    at com.google.auth.oauth2.OAuth2Credentials.refresh (OAuth2Credentials.java:157)
    at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata (OAuth2Credentials.java:145)
    at com.google.auth.oauth2.UserCredentials.getRequestMetadata (UserCredentials.java:281)
    at com.google.auth.http.HttpCredentialsAdapter.initialize (HttpCredentialsAdapter.java:91)
    at com.google.cloud.hadoop.util.ChainingHttpRequestInitializer.initialize (ChainingHttpRequestInitializer.java:52)
    at com.google.api.client.http.HttpRequestFactory.buildRequest (HttpRequestFactory.java:88)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.buildHttpRequest (AbstractGoogleClientRequest.java:422)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed (AbstractGoogleClientRequest.java:541)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed (AbstractGoogleClientRequest.java:474)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute (AbstractGoogleClientRequest.java:591)
    at com.google.cloud.hadoop.util.ResilientOperation$AbstractGoogleClientRequestExecutor.call (ResilientOperation.java:171)
    at com.google.cloud.hadoop.util.ResilientOperation.retry (ResilientOperation.java:67)
    at org.apache.beam.sdk.extensions.gcp.util.GcsUtil.getBucket (GcsUtil.java:521)
    at org.apache.beam.sdk.extensions.gcp.util.GcsUtil.bucketAccessible (GcsUtil.java:509)
    at org.apache.beam.sdk.extensions.gcp.util.GcsUtil.bucketAccessible (GcsUtil.java:482)
    at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.verifyPathIsAccessible (GcsPathValidator.java:83)
    at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.validateOutputFilePrefixSupported (GcsPathValidator.java:53)
    at org.apache.beam.runners.dataflow.DataflowRunner.fromOptions (DataflowRunner.java:274)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod (InstanceBuilder.java:214)
    at org.apache.beam.sdk.util.InstanceBuilder.build (InstanceBuilder.java:155)
    at org.apache.beam.sdk.PipelineRunner.fromOptions (PipelineRunner.java:55)
    at org.apache.beam.sdk.Pipeline.create (Pipeline.java:149)
    at main.java.TemplatePipeline.main (TemplatePipeline.java:95)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:254)
    at java.lang.Thread.run (Thread.java:748)
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.provider.certpath.SunCertPathBuilder.build (SunCertPathBuilder.java:141)
    at sun.security.provider.certpath.SunCertPathBuilder.engineBuild (SunCertPathBuilder.java:126)
    at java.security.cert.CertPathBuilder.build (CertPathBuilder.java:280)
    at sun.security.validator.PKIXValidator.doBuild (PKIXValidator.java:445)
    at sun.security.validator.PKIXValidator.engineValidate (PKIXValidator.java:317)
    at sun.security.validator.Validator.validate (Validator.java:262)
    at sun.security.ssl.X509TrustManagerImpl.validate (X509TrustManagerImpl.java:330)
    at sun.security.ssl.X509TrustManagerImpl.checkTrusted (X509TrustManagerImpl.java:227)
    at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted (X509TrustManagerImpl.java:132)
    at sun.security.ssl.ClientHandshaker.serverCertificate (ClientHandshaker.java:1671)
    at sun.security.ssl.ClientHandshaker.processMessage (ClientHandshaker.java:226)
    at sun.security.ssl.Handshaker.processLoop (Handshaker.java:1082)
    at sun.security.ssl.Handshaker.process_record (Handshaker.java:1010)
    at sun.security.ssl.SSLSocketImpl.readRecord (SSLSocketImpl.java:1079)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake (SSLSocketImpl.java:1388)
    at sun.security.ssl.SSLSocketImpl.startHandshake (SSLSocketImpl.java:1416)
    at sun.security.ssl.SSLSocketImpl.startHandshake (SSLSocketImpl.java:1400)
    at sun.net.www.protocol.https.HttpsClient.afterConnect (HttpsClient.java:559)
    at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect (AbstractDelegateHttpsURLConnection.java:185)
    at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0 (HttpURLConnection.java:1340)
    at sun.net.www.protocol.http.HttpURLConnection.getOutputStream (HttpURLConnection.java:1315)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream (HttpsURLConnectionImpl.java:264)
    at com.google.api.client.http.javanet.NetHttpRequest.execute (NetHttpRequest.java:113)
    at com.google.api.client.http.javanet.NetHttpRequest.execute (NetHttpRequest.java:84)
    at com.google.api.client.http.HttpRequest.execute (HttpRequest.java:1012)
    at com.google.auth.oauth2.UserCredentials.refreshAccessToken (UserCredentials.java:203)
    at com.google.auth.oauth2.OAuth2Credentials.refresh (OAuth2Credentials.java:157)
    at com.google.auth.oauth2.OAuth2Credentials.getRequestMetadata (OAuth2Credentials.java:145)
    at com.google.auth.oauth2.UserCredentials.getRequestMetadata (UserCredentials.java:281)
    at com.google.auth.http.HttpCredentialsAdapter.initialize (HttpCredentialsAdapter.java:91)
    at com.google.cloud.hadoop.util.ChainingHttpRequestInitializer.initialize (ChainingHttpRequestInitializer.java:52)
    at com.google.api.client.http.HttpRequestFactory.buildRequest (HttpRequestFactory.java:88)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.buildHttpRequest (AbstractGoogleClientRequest.java:422)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed (AbstractGoogleClientRequest.java:541)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed (AbstractGoogleClientRequest.java:474)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute (AbstractGoogleClientRequest.java:591)
    at com.google.cloud.hadoop.util.ResilientOperation$AbstractGoogleClientRequestExecutor.call (ResilientOperation.java:171)
    at com.google.cloud.hadoop.util.ResilientOperation.retry (ResilientOperation.java:67)
    at org.apache.beam.sdk.extensions.gcp.util.GcsUtil.getBucket (GcsUtil.java:521)
    at org.apache.beam.sdk.extensions.gcp.util.GcsUtil.bucketAccessible (GcsUtil.java:509)
    at org.apache.beam.sdk.extensions.gcp.util.GcsUtil.bucketAccessible (GcsUtil.java:482)
    at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.verifyPathIsAccessible (GcsPathValidator.java:83)
    at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.validateOutputFilePrefixSupported (GcsPathValidator.java:53)
    at org.apache.beam.runners.dataflow.DataflowRunner.fromOptions (DataflowRunner.java:274)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod (InstanceBuilder.java:214)
    at org.apache.beam.sdk.util.InstanceBuilder.build (InstanceBuilder.java:155)
    at org.apache.beam.sdk.PipelineRunner.fromOptions (PipelineRunner.java:55)
    at org.apache.beam.sdk.Pipeline.create (Pipeline.java:149)
    at main.java.TemplatePipeline.main (TemplatePipeline.java:95)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:254)
    at java.lang.Thread.run (Thread.java:748)
[ERROR] 
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

gcloud auth application-default login我已经使用和从终端登录到 GCP export GOOGLE_CLOUD_CREDENTIALS=${PATH}

此外，我已将 google 云证书添加到 cacerts 中 sudo keytool -import -trustcacerts -file /path/to/google.cer -alias google -keystore $JAVA_HOME/jre/lib/security/cacerts，并验证它是否存在

非常感谢解决此问题的任何帮助。

Answer 1

除了确保您输入了正确的存储桶名称/路径之外，我还要确保：

1. 您已启用所有必需的 API。目前包括：Cloud Dataflow、Compute Engine、Stackdriver Logging、Cloud Storage、Cloud Storage JSON、BigQuery、Cloud Pub/Sub、Cloud Datastore 和 Cloud Resource Manager
2. 您通过创建具有项目 > 所有者角色的服务帐户来正确设置身份验证。

请参阅此链接以确保您满足所有要求，因为您正处于数据流的创建阶段。

Answer 2

编辑-2

我终于可以解决错误，现在我可以从 mvn 和 Eclipse 运行我的 Dataflow 运行器。我做了以下事情：

1. 我发现我在 pom 中的 SDK 版本到处都是，因此，将 directrunner 和 dataflowrunner 版本合并到 2.29.0。
2. 我手动将 ServiceAccountCredentials 范围设置为“https://www.googleapis.com/auth/cloud-platform”，我将其删除并仅添加了 GOOGLE_APPLICATION_CREDENTIALS 作为运行时 env 变量，并为我的 serviceaccount 密钥分配了缩小范围以用于数据流作业执行。
3. 为了运行 mvn compile exec:java ，在我的脚本中添加了 GOOGLE_APPLICATION_CREDENTIALS 以便为每个会话运行

编辑

我可以克服凭据错误，但是由于以下错误，我的工作再次失败：

java.io.IOException: Error matching file spec gs://gce_lab/staging/nashorn-BJZNQ7N8Lsfq-WSM0IMsRCwFMC3RIxBOEjrlB1YwKOw.jar

并查看一些较早WARN的 s，我发现我所有复制到暂存位置的操作都失败了……仍然在尝试调试。

我正在使用以下内容：

<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-java-core</artifactId>
  <version>2.33.0</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-runners-google-cloud-dataflow- java</artifactId>
  <version>2.33.0</version>
</dependency>
<dependency>
  <groupId>org.apache.beam</groupId>
  <artifactId>beam-sdks-java-io-google-cloud-platform</artifactId>
  <version>2.33.0</version>
</dependency>

@StackOverFlowUser - 我试图从 Eclipse 运行我的 DataFlow 管道并且遇到了完全相同的错误，尽管设置了正确的 SA 和其他细节。当我在DataflowPipelineOptions参考中添加以下 2 个参数时，它对我有用：

1. pipelineOptions.setPathValidatorClass(NoopPathValidator.class);
2. pipelineOptions.setTemplateLocation("gs://<template_lcoation>"); // all 3 -> temp, stage and template location.

Answer 3

这可能有两个原因。

1. 要么没有名称的存储桶 - bucket-df-job。如果是，请创建存储桶

2. 无论您正在使用并分配给什么服务帐户GOOGLE_APPLICATION_CREDENTIALS，该服务帐户都可以访问 GCS 存储桶吗？如果没有，请授予访问权限并重新尝试构建数据流模板。