气流异常：数据流失败，返回代码为2

Question

我正在尝试执行一个DataflowPython文件，该文件使用DataFlowPythonOperator通过气流DAG从GCS存储桶读取文本文件。我已经能够独立地执行python文件，但是当我通过airflow执行它时，它失败了。我正在使用服务帐户对默认gcp连接进行身份验证。我在执行作业时遇到的错误是：

{gcp_dataflow_hook.py:108} INFO - Start waiting for DataFlow process to complete.
{models.py:1417} ERROR - DataFlow failed with return code 2
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 1374, in run
    result = task_copy.execute(context=context)
  File "/usr/local/lib/python2.7/dist-packages/airflow/contrib/operators/dataflow_operator.py", line 182, in execute
    self.py_file, self.py_options)
  File "/usr/local/lib/python2.7/dist-packages/airflow/contrib/hooks/gcp_dataflow_hook.py", line 152, in start_python_dataflow
    task_id, variables, dataflow, name, ["python"] + py_options)
  File "/usr/local/lib/python2.7/dist-packages/airflow/contrib/hooks/gcp_dataflow_hook.py", line 138, in _start_dataflow
    _Dataflow(cmd).wait_for_done()
  File "/usr/local/lib/python2.7/dist-packages/airflow/contrib/hooks/gcp_dataflow_hook.py", line 119, in wait_for_done
    self._proc.returncode))
Exception: DataFlow failed with return code 2

我的脚本：

from airflow import DAG
from airflow.contrib.operators.dataflow_operator import DataFlowPythonOperator
from datetime import datetime, timedelta 

# Default DAG parameters
default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'email': <email>,
    'email_on_failure': False,
    'email_on_retry': False,
    'start_date': datetime(2018, 4, 30),
    'retries': 1,
    'retry_delay': timedelta(minutes=1),
    'dataflow_default_options': {
        'project': '<Project ID>'        
    }
}

dag = DAG(
    dag_id='df_dag_readfromgcs', 
    default_args=default_args, 
    schedule_interval=timedelta(minutes=60)
    )

task1 = DataFlowPythonOperator(
    task_id='task1',    
    py_file='~<path>/1readfromgcs.py',
    gcp_conn_id='default_google_cloud_connection',
    dag=dag
)

我的数据流python文件（1readfromgcs.py）包含以下代码：

from __future__ import absolute_import

import argparse
import logging
import apache_beam as beam
import apache_beam.pipeline as pipeline
import apache_beam.io as beamio
from apache_beam.options.pipeline_options import PipelineOptions
from apache_beam.io import ReadFromText


def runCode(argv=None):
    parser = argparse.ArgumentParser()
    parser.add_argument('--input',
                        default='<Input file path>',
                        help='File name')   
    known_args, pipeline_args = parser.parse_known_args(argv) 

    pipeline_args.extend([
        '--project=<project name>',
        '--runner=DataflowRunner',
        '--job_name=<job name>',
        '--region=europe-west1',
        '--staging_location=<GCS staging location>',
        '--temp_location=<GCS temp location>'
    ])

    pipeline_options = PipelineOptions(pipeline_args)
    p = beam.pipeline.Pipeline(options=pipeline_options)

    rows = p | 'read' >> beam.io.ReadFromText(known_args.input) 

    p.run().wait_until_finish()

if __name__ == '__main__':
    logging.getLogger().setLevel(logging.INFO)
    runCode()

def wait_for_done(self):
        reads = [self._proc.stderr.fileno(), self._proc.stdout.fileno()]
        self.log.info("Start waiting for DataFlow process to complete.")
        while self._proc.poll() is None:
            ret = select.select(reads, [], [], 5)
            if ret is not None:
                for fd in ret[0]:
                    line = self._line(fd)
                    self.log.debug(line[:-1])
            else:
                self.log.info("Waiting for DataFlow process to complete.")
        if self._proc.returncode is not 0:
            raise Exception("DataFlow failed with return code {}".format(
                self._proc.returncode))

感谢您的想法并帮助解决我的问题。

Answer 1

此异常源于\u proc，它是子流程。它从shell返回一个退出代码。

我还没有使用这个组件。根据正在执行的内容，退出代码2将说明退出的原因。例如，bash中的退出代码意味着：

误用贝壳内置物

并且可能与

缺少关键字或命令，或权限问题

因此，它可能连接到底层数据流配置。尝试在模拟用户时手动执行该文件。