在数据流中指定区域读取BigQuery Dataset python SDK

Question

我试图在DataFlow中读取bigquery数据集。它找不到我指定的bigquery数据集/表。

我的datalab vm、gcs bucket和bigquery dataset都位于欧洲西部2。

出于某种原因，它正在“美国”位置搜索数据集。

modules versions are apache-beam 2.5.0,google-cloud-dataflow 2.0.0, google-cloud-bigquery 0.25.0

搜索了文档，但无法找到为什么会发生这种情况的答案。

OUTPUT_DIR = "gs://some-bucket/some-folder/"

#dictionary of pipeline options
options = {
    "staging_location": "gs://some-bucket/some-folder/stage/"
    "temp_location": "gs://some-bucket/some-folder/tmp/"
    "job_name": job_name,
    "project": PROJECT,
    "runner": "DirectRunner",
    "location":'europe-west2',
    "region":'europe-west2',
}

#instantiate PipelineOptions object using options dictionary
opts = beam.pipeline.PipelineOptions(flags = [], **options)

#instantantiate Pipeline object using PipelineOptions
with beam.Pipeline(options=opts) as 
    outfile = "gs://some-bucket/some-folder/train.csv"
    (
      p | "read_train" >> beam.io.Read(beam.io.BigQuerySource(query = 
my_query, use_standard_sql = True))
        | "tocsv_train" >> beam.Map(to_csv)
        | "write_train" >> beam.io.Write(beam.io.WriteToText(outfile))
    )
print("Done")

HttpError：HttpError访问https://www.googleapis.com/bigquery/v2/projects/projects/queries/querystring:response:<{‘状态':'404',’内容-长度':'342','x-xss-保护':'0','x-content-type-options':'nosniff,'transfer-encoding':'chunked,'vary':'esf,'-content-encoding':'gzip,'-cache-control':'private,'date':'thu,2019年5月23日Origin“,”Content-Type“：”Application/JSON；Charset=UTF-8“}>,content<{”错误“：{”代码“：404,”消息“：”未找到：Dataset My_DataSet:视图在美国位置未找到“,”错误“：[{”消息“：”未找到：Dataset My_DataSet:视图在美国位置未找到“,”域“：”全局“,”原因“：”未找到“}],”状态“：”未找到“}}

Answer 1

在Apache Beam 2.5.0 Python SDK中，还不支持非美国查询源。

看起来像是在Apache Beam 2.8.0 Python SDK中添加了支持[Release Notes,PR,JIRA]。