aws emr add-steps一个spark应用程序

Question

我想用AWS CLI添加一个步骤作为spark应用程序，但我找不到一个工作命令，从AWS官方文档：https://docs.AWS.amazon.com/CLI/latest/reference/emr/add-steps.html中，他们列出了6个示例，没有一个是针对spark的。但我可以通过AWS控制台UI配置它，它运行良好，但为了效率，我希望能够通过AWS CLI这样做。

aws emr add-steps --cluster-id j-cluster-id --steps  Type=SPARK,Name='SPARK APP',ActionOnFailure=CONTINUE,Jar=s3://my-test/RandomJava-1.0-SNAPSHOT.jar,MainClass=JavaParquetExample1,Args=s3://my-test/my-file_0000_part_00.parquet,my-test --profile my-test --region us-west-2

JAR location : command-runner.jar
Main class : None
Arguments : spark-submit s3://my-test/my-file_0000_part_00.parquet my-test
Action on failure: Continue

正确的（已成功完成，通过AWS控制台UI配置）如下所示：

JAR location : command-runner.jar
Main class : None
Arguments : spark-submit --deploy-mode cluster --class sparkExamples.JavaParquetExample1 s3://my-test/RandomJava-1.0-SNAPSHOT.jar --s3://my-test/my-file_0000_part_00.parquet --my-test
Action on failure: Continue

如有任何帮助，不胜感激！

Answer 1

这似乎对我管用。我将使用步骤名我的步骤名向集群添加一个spark应用程序。假设您将文件命名为step-addition.sh。其内容如下：

#!/bin/bash
set -x

#cluster id
clusterId=$1
startDate=$2
endDate=$3

aws emr add-steps --cluster-id $clusterId --steps Type=Spark,Name='My step name',\
ActionOnFailure=TERMINATE_CLUSTER,Args=[\
"--deploy-mode","cluster","--executor-cores","1","--num-executors","20","--driver-memory","10g","--executor-memory","3g",\
"--class","your-package-structure-like-com.a.b.c.JavaParquetExample1",\
"--master","yarn",\
"--conf","spark.driver.my.custom.config1=my-value-1",\
"--conf","spark.driver.my.custom.config2=my-value-2",\
"--conf","spark.driver.my.custom.config.startDate=${startDate}",\
"--conf","spark.driver.my.custom.config.endDate=${endDate}",\
"s3://my-bucket/my-prefix/path-to-your-actual-application.jar"]

您可以像这样简单地执行上面的脚本：

bash $WORK_DIR/step-addition.sh $clusterId $startDate $endDate