我能让flex模板作业在开始处理数据之前不到10分钟吗？

Question

resource "google_dataflow_flex_template_job" "streaming_beam" {
  provider                = google-beta
  name                    = "streaming-beam"
  container_spec_gcs_path = module.streaming_beam_flex_template_file[0].fully_qualified_path
  parameters = {
    "input_subscription"    = google_pubsub_subscription.ratings[0].id
    "output_table"          = "${var.project}:beam_samples.streaming_beam_sql"
    "service_account_email" = data.terraform_remote_state.state.outputs.sa.email
    "network"               = google_compute_network.network.name
    "subnetwork"            = "regions/${google_compute_subnetwork.subnet.region}/subnetworks/${google_compute_subnetwork.subnet.name}"
  }
}

resource "google_dataflow_flex_template_job" "streaming_beam" {
  provider                = google-beta
  name                    = "streaming-beam"
  container_spec_gcs_path = module.streaming_beam_flex_template_file[0].fully_qualified_path
  parameters = {
    "input_subscription"    = google_pubsub_subscription.ratings[0].id
    "output_table"          = "${var.project}:beam_samples.streaming_beam_sql"
    "service_account_email" = data.terraform_remote_state.state.outputs.sa.email
    "network"               = google_compute_network.network.name
    "subnetwork"            = "regions/${google_compute_subnetwork.subnet.region}/subnetworks/${google_compute_subnetwork.subnet.name}"
    "flexrs_goal"           = "SPEED_OPTIMIZED" 
  }
}

2021-01-17 19:49:19.021 GMTStarting GCE实例，启动器-2021011711491611239867327455334861，以启动模板。
...
...
2021-01-17 19:59:05.381 GMTStarting 1欧洲工人-West1-d...
2021-01-17 19:59:12.256 GMTVM，启动器-2021011711491611239867327455334861，停止。

然后，我尝试明确地设置flexrs_goal=cost_optimized，看看它是否有什么不同，但这只导致了一个错误：

“无法创建工作流。原因：由于配置错误，无法创建工作流。流作业不支持实验功能flexible_resource_scheduling。有关进一步帮助，请与Google云支持部门联系。”

null

2021-01-17 19:51:03.381 GMT“info:apache_beam.runners.portability.stager：正在执行命令:['/usr/local/bin/python'、'-m'、'pip'、'download'、'--dest'、'/tmp/dataflow-requirements-cache'、'-r'、'/dataflow/template/requirements.txt'、'--exists-action'、'i'、'--no-binary'、':all:']”
2021-01-17来自PYPI“

无论这两个日志记录之间发生了什么，都是在state=queued中花费很长时间的主要原因。有人知道原因是什么吗？

Answer 1

RUN pip install -U apache-beam==<version>
RUN pip install -U -r ./requirements.txt