401未经授权/匿名呼叫者无法storage.objects.list访问Google Cloud Storage存储桶

Question

我想通过embulk和digdag将数据从GCS传输到BigQuery。

但出现了错误。

com.google.api。客户古格里皮斯。json。GoogleJsonResponseException：401未经授权

。。。。。。。

错误：组织。embulk。配置。ConfigException：com.google.云存储StorageException：匿名调用方没有存储。对象。列出对Google云存储桶的访问权限。

命令：

embulk run XXXX.yaml

XXXX. yaml：

in:
  type: gcs
  bucket: <bucket name>
  path_prefix: <file path>
  auth_method: compute_engine
  parser:
    type: poi_excel
    sheets: <sheet name>
    skip_header_lines: 4
    columns:
    - {name: 'name', type: string}
        .
        .
        .
out:
  type: bigquery
  mode: replace
  project: <project name>
  dataset: <dataset name>
  table: <table name>
  auth_method: compute_engine
  schema_file: <file name of json type>
  gcs_bucket: <gcs tmp bucket name>

输出：

 $ embulk run target_item_bottoms_config.yaml
2020-07-22 14:27:36.559 +0900: Embulk v0.9.23
2020-07-22 14:27:37.609 +0900 [WARN] (main): DEPRECATION: JRuby org.jruby.embed.ScriptingContainer is directly injected.
2020-07-22 14:27:40.577 +0900 [INFO] (main): Gem's home and path are set by default: "/Users/oniki/.embulk/lib/gems"
2020-07-22 14:27:41.662 +0900 [INFO] (main): Started Embulk v0.9.23
2020-07-22 14:27:41.853 +0900 [INFO] (0001:transaction): Loaded plugin embulk-input-gcs (0.3.2)
2020-07-22 14:27:46.263 +0900 [INFO] (0001:transaction): Loaded plugin embulk-output-bigquery (0.6.4)
2020-07-22 14:27:46.369 +0900 [INFO] (0001:transaction): Loaded plugin embulk-parser-poi_excel (0.1.7)
org.embulk.exec.PartialExecutionException: org.embulk.config.ConfigException: com.google.cloud.storage.StorageException: Anonymous caller does not have storage.objects.list access to the Google Cloud Storage bucket.
    at org.embulk.exec.BulkLoader$LoaderState.buildPartialExecuteException(BulkLoader.java:340)
    at org.embulk.exec.BulkLoader.doRun(BulkLoader.java:566)
    at org.embulk.exec.BulkLoader.access$000(BulkLoader.java:35)
    at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:353)
    at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:350)
    at org.embulk.spi.Exec.doWith(Exec.java:22)
    at org.embulk.exec.BulkLoader.run(BulkLoader.java:350)
    at org.embulk.EmbulkEmbed.run(EmbulkEmbed.java:242)
    at org.embulk.EmbulkRunner.runInternal(EmbulkRunner.java:291)
    at org.embulk.EmbulkRunner.run(EmbulkRunner.java:155)
    at org.embulk.cli.EmbulkRun.runSubcommand(EmbulkRun.java:431)
    at org.embulk.cli.EmbulkRun.run(EmbulkRun.java:90)
    at org.embulk.cli.Main.main(Main.java:64)
    Suppressed: java.lang.NullPointerException
        at org.embulk.exec.BulkLoader.doCleanup(BulkLoader.java:463)
        at org.embulk.exec.BulkLoader$3.run(BulkLoader.java:397)
        at org.embulk.exec.BulkLoader$3.run(BulkLoader.java:394)
        at org.embulk.spi.Exec.doWith(Exec.java:22)
        at org.embulk.exec.BulkLoader.cleanup(BulkLoader.java:394)
        at org.embulk.EmbulkEmbed.run(EmbulkEmbed.java:245)
        ... 5 more
Caused by: org.embulk.config.ConfigException: com.google.cloud.storage.StorageException: Anonymous caller does not have storage.objects.list access to the Google Cloud Storage bucket.
    at org.embulk.input.gcs.AuthUtils.newClient(AuthUtils.java:81)
    at org.embulk.input.gcs.GcsFileInput.listFiles(GcsFileInput.java:49)
    at org.embulk.input.gcs.GcsFileInputPlugin.transaction(GcsFileInputPlugin.java:59)
    at org.embulk.spi.FileInputRunner.transaction(FileInputRunner.java:62)
    at org.embulk.exec.BulkLoader.doRun(BulkLoader.java:507)
    ... 11 more
Caused by: com.google.cloud.storage.StorageException: Anonymous caller does not have storage.objects.list access to the Google Cloud Storage bucket.
    at com.google.cloud.storage.spi.v1.HttpStorageRpc.translate(HttpStorageRpc.java:226)
    at com.google.cloud.storage.spi.v1.HttpStorageRpc.list(HttpStorageRpc.java:366)
    at com.google.cloud.storage.StorageImpl$8.call(StorageImpl.java:338)
    at com.google.cloud.storage.StorageImpl$8.call(StorageImpl.java:335)
    at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:105)
    at com.google.cloud.RetryHelper.run(RetryHelper.java:76)
    at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:50)
    at com.google.cloud.storage.StorageImpl.listBlobs(StorageImpl.java:334)
    at com.google.cloud.storage.StorageImpl.list(StorageImpl.java:290)
    at org.embulk.input.gcs.AuthUtils.newClient(AuthUtils.java:77)
    ... 15 more
Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 401 Unauthorized
{
  "code" : 401,
  "errors" : [ {
    "domain" : "global",
    "location" : "Authorization",
    "locationType" : "header",
    "message" : "Anonymous caller does not have storage.objects.list access to the Google Cloud Storage bucket.",
    "reason" : "required"
  } ],
  "message" : "Anonymous caller does not have storage.objects.list access to the Google Cloud Storage bucket."
}
    at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:150)
    at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
    at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:401)
    at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1097)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:499)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:549)
    at com.google.cloud.storage.spi.v1.HttpStorageRpc.list(HttpStorageRpc.java:356)
    ... 23 more

Error: org.embulk.config.ConfigException: com.google.cloud.storage.StorageException: Anonymous caller does not have storage.objects.list access to the Google Cloud Storage bucket.

我的环境：

$ gcloud config list
[compute]
region = us-east1
zone = us-east1-c
[core]
account = [e-mail]
disable_usage_reporting = False
project = <project ID>

Your active configuration is: [default]


$ gcloud auth list  
    Credentialed Accounts
ACTIVE  ACCOUNT
*       [e-mail]

To set the active account, run:
    $ gcloud config set account `ACCOUNT`



$ gsutil ls   
gs://<bucket name>

我的gcp IAM角色：

物主

我理解解决此错误的方法是授权。但我的偏好似乎不错。

怎么了？

Answer 1

我尝试在本地，并创建服务帐户密钥并保存在本地。

◾️XXXX年。亚马尔

之前

auth_method: compute_engine

之后

auth_method: json_key
json_keyfile: /path/to/json_keyfile.json

Answer 2

作为留档[1]，如果我们有401-未经授权的错误，那么可能有很多原因，请在下面列出相关的原因列表[点击链接1]，这可能有助于故障排除：

原因：AuthenticationRequiredRequesterPays

访问请求者付费桶需要身份验证。

原因：authError

此错误表示云存储请求中提供的授权存在问题。以下是将发生这种情况的一些情况：OAuth访问令牌已过期，需要刷新。这可以通过提前刷新访问令牌来避免，但代码也可以捕获此错误，刷新令牌并自动重试。提供了多个不匹配的授权；仅选择一种模式。OAuth访问令牌的绑定项目与提供的开发人员密钥关联的项目不匹配。授权标头的格式无法识别，或使用了不受支持的凭据类型。

原因：LockedDomainired

从cookie认证的站点下载内容时（例如，使用存储浏览器），响应将重定向到临时域。如果在域过期后访问所述域，则会发生此错误。再次发出原始请求，并接收新的重定向。

原因：push.webhookUrl未经授权

对存储的请求。对象。除非验证您拥有该域，否则watchAll将失败。

原因：必需

对需要授权的非公共方法进行了访问，但没有在授权标头中或通过其他方式提供任何访问。

[1]https://cloud.google.com/storage/docs/json_api/v1/status-codes#401_Unauthorized