使用Typesafe配置时在Spark YARN客户端和集群模式之间切换

Question

一直在努力解决使用Spark YARN处理多个配置文件以及在集群和客户端模式之间切换的问题。

在我的应用程序中，我需要加载两个配置文件：

1. 应用程序配置
2. 环境配置

示例-application.conf：

include required(file("env.conf"))

app {
  source
    {
      source-name: "some-source"
      source-type: "file"
      source-path: ${env.some-source-path}
    }
   ....
}

env.conf：

env {
  some-source-path: "/path/to/file"
}

代码：

// Spark submit that works:
$SPARK_HOME/bin/spark-submit --class ${APP_MAIN_CLASS} \
    --master yarn \
    --deploy-mode cluster \
    --name ${APP_INSTANCE} \
    --files ${APP_BASE_DIR}/conf/${ENV_NAME}/env.conf,${APP_BASE_DIR}/conf/example-application.conf \
    --principal ${PRINCIPAL_NAME} --keytab ${KEYTAB_PATH} \
    --jars ${JARS} \
    --num-executors 10 \
    --executor-memory 4g \
    --executor-cores 4 \
    ${APP_JAR} "example-application.conf" "$@"

// How above file is loaded in code:
val appConfFile = new File(configFileName)   // configFileName = "example-application.conf"
val conf = ConfigFactory.parseFile(appConfFile)

有什么建议如何设置，以便我可以轻松地交换集群和客户端模式？

谢谢！

Answer 1

作为spark-submit的一部分传递配置文件的完整路径&处理spark代码中的解压缩逻辑。

spark.submit.deploymode=client然后取完整路径，即${APP_BASE_DIR}/conf/example-application.conf

spark.submit.deploymode=cluster然后只取文件名，即example-application.conf

// Spark submit that works:
$SPARK_HOME/bin/spark-submit --class ${APP_MAIN_CLASS} \
    --master yarn \
    --deploy-mode cluster \
    --name ${APP_INSTANCE} \
    --files ${APP_BASE_DIR}/conf/${ENV_NAME}/env.conf,${APP_BASE_DIR}/conf/example-application.conf \
    --principal ${PRINCIPAL_NAME} --keytab ${KEYTAB_PATH} \
    --jars ${JARS} \
    --num-executors 10 \
    --executor-memory 4g \
    --executor-cores 4 \
    ${APP_JAR} ${APP_BASE_DIR}/conf/example-application.conf "$@"

// How above file is loaded in code:

val configFile = if(!spark.conf.get("spark.submit.deployMode").contains("client")) configFileName.split("/").last else configFileName

val appConfFile = new File(configFile)   // configFileName = "example-application.conf"
val conf = ConfigFactory.parseFile(appConfFile)