自调用lambda调用超时

Question

我们正在尝试开发一个自调用的lambda来成批处理S3文件。lambda角色具有附加调用所需的策略。

以下是自调用lambda的代码：

export const processFileHandler: Handler = async (
  event: S3CreateEvent,
  context: Context,
  callback: Callback,
) => {
  let bucket = loGet(event, 'Records[0].s3.bucket.name');
  let key = loGet(event, 'Records[0].s3.object.key');
  let totalFileSize = loGet(event, 'Records[0].s3.object.size');
  const lastPosition = loGet(event, 'position', 0);
  const nextRange = getNextSizeRange(lastPosition, totalFileSize);

  context.callbackWaitsForEmptyEventLoop = false;

  let data = await loadDataFromS3ByRange(bucket, key, nextRange);


  await database.connect();

  log.debug(`Successfully connected to the database`);

  const docs = await getParsedDocs(data, lastPosition);
  log.debug(`upserting ${docs.length} records to database`);
  if (docs.length) {
    try {
         // upserting logic
      log.debug(`total documents added: ${await docs.length}`);
    } catch (err) {
      await recurse(nextRange.end, event, context);
      log.debug(`error inserting docs: ${JSON.stringify(err)}`);
    }
  }

  if (nextRange.end < totalFileSize) {
    log.debug(`Last ${context.getRemainingTimeInMillis()} milliseconds left`);
    if (context.getRemainingTimeInMillis() < 10 * 10 * 10 * 6) {
      log.debug(`Less than 6000 milliseconds left`);
      log.debug(`Invoking next iteration`);
      await recurse(nextRange.end, event, context);
      callback(null, {
        message: `Lambda timed out processing file, please continue from LAST_POSITION: ${nextRange.start}`,
      });
    }

  } else {
    callback(null, { message: `Successfully completed the chunk processing task` });
  }
};

其中递归是对同一个lambda的调用调用。其余的事情都按预期工作，只要调用堆栈出现在这个调用请求上，它就会超时：

const recurse = async (position: number, event: S3CreateEvent, context: Context) => {
  let newEvent = Object.assign(event, { position });

  let request = {
    FunctionName: context.invokedFunctionArn,
    InvocationType: 'Event',
    Payload: JSON.stringify(newEvent),
  };

  let resp = await lambda.invoke(request).promise();
  console.log('Invocation complete', resp);

  return resp;
};

这是记录到CloudWatch的堆栈跟踪：

{
    "errorMessage": "connect ETIMEDOUT 63.32.72.196:443",
    "errorType": "NetworkingError",
    "stackTrace": [
        "Object._errnoException (util.js:1022:11)",
        "_exceptionWithHostPort (util.js:1044:20)",
        "TCPConnectWrap.afterConnect [as oncomplete] (net.js:1198:14)"
    ]
}

Answer 1

创建自调用lambda函数不是一个好主意。如果出现错误（也可能是AWS端的错误处理程序调用），lambda函数可能会重新运行几次。很难监控和调试。

我建议使用步骤函数。我相信本教程可以帮助使用Lambda迭代循环

如果您不想处理Step函数，那么可以为SQS队列创建一个Lambda触发器。如果希望下次运行lambda函数，则向队列传递一条消息。