尝试从Apache Beam访问Google Cloud Storage时出错

Question

我试图使用Apache Beam从Compute Engine VM简单访问Google Cloud存储。当然，我已经使用命令gcloud auth应用程序默认登录设置了默认应用程序登录，并为计算引擎服务号添加了对存储的访问权限。我使用DirectRunner运行了管道并得到错误：apache_beam.io.filesystem.BeamIOError：匹配操作失败，有异常{'gs：//{this MY BUCKETNAME}/source/sales_transactions.csv'： HttpForBiddenError（）}

#import print library
import logging

#import apache beam library
import apache_beam as beam

#import pipeline options.
from apache_beam.options.pipeline_options import  PipelineOptions

#Create a pipeline
plOps = beam.Pipeline(options=PipelineOptions())

#--------------------------------------------------
# 1.Read from a text file.
#--------------------------------------------------

#Read the file from Google Cloud Storage
transactions = ( plOps 
                | 'Read Transaction CSV'
                    >> beam.io.ReadFromText('gs://{THIS MY BUCKETNAME}/data/sales_transactions.csv')
                )

printSize(transactions,'Raw Transactions')

Answer 1

是 啊我解决了它。只需重新创建默认的应用程序凭据，我真的不知道它们出了什么问题。我已经毫不含糊地做到了。否则文件夹/根/。配置/gcloud不存在。但凭证出现了问题。

那么，如何解决它：1。sudo su 2。删除文件夹/根/。配置/gcloud 3。再次运行gcloud auth应用程序默认登录

祝你好运！

Answer 2

我已经部分解决了这个问题——阅读正常。我在DEVELOPER MACHINE上使用sudo su按root登录，Apache Beam可以访问从GS读取文件。但是当我尝试像这样写入GS存储桶时

#Write output to a text file
( custTypeCount | 'Write to GS Text'
        >> beam.io.WriteToText('gs://{MY BUCKET NAME}/output/customertype-summary.txt')
)  

脚本获取错误：

RuntimeError: HttpForbiddenError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/{MY BUCKET NAME}/o?uploadType=resumable&alt=json&name=output%2Fbeam-temp-customertype-summary.txt-7bea505ad0bf11e9b69c42010a800002%2F55a9057e-18e5-4171-9db4-9e55601b2a8d.customertype-summary.txt>: response: <{'status': '403', 'content-length': '208', 'vary': 'Origin, X-Origin', 'server': 'UploadServer', 'x-guploader-uploadid': 'AEnB2Upo4RBzVV1S51_uWhcCiK_uK_iOSRAdAb8HWMhxznaPr0JcHKWxKDLwHbtTIYvHuMjyESV4dZqAfN3TaWYMqr5gQeypcQ', 'date': 'Fri, 06 Sep 2019 16:00:44 GMT', 'content-type': 'application/json; charset=UTF-8', 'www-authenticate': 'Bearer realm="https://accounts.google.com/"'}>, content <{
 "error": {
  "errors": [
   {
    "domain": "global",
    "reason": "insufficientPermissions",
    "message": "Insufficient Permission"
   }
  ],
  "code": 403,
  "message": "Insufficient Permission"
 }
}
> [while running 'Write to GS Text/Write/WriteImpl/WriteBundles']

我不知道应该添加什么权限