如何在火花日志中隐藏密钥密码？

Question

在运行spark作业时，可以在事件日志中以纯文本形式看到SSL密钥密码、keystorepassword。你能帮我如何从日志中隐藏这些密码吗？

当我看到下面的内容时，https://issues.apache.org/jira/browse/spark-16796似乎是他们修复了它，使其不受web UI的影响。但我不确定我能用原木修复它

你的帮助真的很感激！！

“{”事件“：”SparkListenerLogStart“，”Spark Version“：”2.1.1“}{”事件“：”SparkListenerBlockManagerAdded“，”块管理器ID“：{”Executor ID“：”Driver“，”Host“：”xx.xxx.xxx.xxx“，”端口“：43556}，”最大内存“：434031820,”时间戳“：1512750709305}{”事件“：”SparkListenerEnvironmentUpdate“，”JVM信息“：{”Java Home“：”版本“：”1.8.0_141（Oracle Corporation）“，”Scala版本“：”Version 2.11.8“}，”Spark属性“：{”spark.sql.warehouse.dir“：”hdfs:///user/spark/warehare“，”spark.yarn.dist.files“：”file://etc/spark/conf/hive-site.xml“，”spark.executor.extrajavaoptions“：”-verbose:gc-xx:+printgcdetails-xx:+printgcdetails unloadingenabled-xx：onoutofmemoryerror='kill-9%p'“，”spark.driver.host“：”xx.xxx.xx.xxx“，”spark.serializer.objectstreamreset“：”100“，”spark.history.fs.logdirectory“：”hdfs：///var/log/spark/apps“，”spark.eventlog.ena“bled“：”true“,”spark.driver.port“：”44832“,”spark.shuffle.service.enabled“：”true“,”spark.rdd.compress“：”true“,”spark.driver.extralibraryPath“：”/usr/lib/hadoop/lib/native://usr/lib/hadoop-lzo/lib/native“,”spark.ssl.keystore“：”/usr/share/aws/emr/security/conf/keystore.jks“,”4-src.zip“,”spark.ssl.enabled“：”true“,”spark.yarn.historyserver.address“：”ip-xx-xxx-xxx.xxx.com:18080“,”spark.ssl.truststore“：”/usr/share/aws/emr/security/conf/truststore.jks“,”spark.app.name“：”claim_line_fact_main“,”spark.scheduer.mode“：”fifo“,”密码“：”xxxxxx“，”spark.executor.id“：”driver“，”spark.driver.extrajavaoptions“：”-xx:+useconcmarksweepgc-xx:cmsinitiatingogoccupancyfraction=70-xx:maxheapfreeratio=70-xx:+cmsclassunloadingenabled-xx:onoutofmemoryerror='kill-9%p'“，”spark.submit.deploymode“：”client“，”spark.master“：”yarn“，ablesaslencryption“：”true“，”spark.authenticate“：”true“，”spark.ui.filters“：”org.apache.hadoop.yarn.server.webproxy.amfilter.amipfilter“，”spark.executor.extralibrarypath“：”/usr/lib/hadoop/lib/native://usr/lib/hadoop-lzo/lib/native“，”spark.executor.emory“：”5120m“，”spark.driver.extraclasspath“：”hadoop-lzo/lib/://usr/lib/hadoop/hadoop-aws.jar://usr/share/aws/aws/aws-java-sdk/://usr/share/aws/emr/emr/conf://usr/share/aws/emr/emr/aws/schar/aws/schar/conf/://usr/share/aws/security/conf://usr/share/aws/security/lib/“，”spark.eventlog.dir“：”hdfs：///var/log/spark/apps“”，“spark.dynamicallocation.enabled”：“true”，“spark.executor.extraclasspath”：“/usr/lib/hadoop-lzo/lib/://usr/lib/hadoop/hadoop-aws.jar：/usr/share/aws/aws-java-sdk/://usr/share/aws/aws/aws-java-sdk/://usr/share/aws/emr/emfs/conf：security/lib/“，”spark.executor.cores“：”4“，”spark.history.ui.port“：”18080“，”spark.driver.appuiaddress“：”http：//“，”spark.yarn.ispython“：”true“，”spark.ssl.truststorepassword“：”xxxxxx“，”spark.org.apache.hadoop.yarn.server.webproxy.amfilter.amipfilter.param.proxy_hosts“：”WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA“,”spark.org.apache.hadoop.yarn.server.webproxy.amfilter.amipfilter.param.proxy_URI_bases“

Answer 1

只能用INFO、WARN、ERROR记录的消息可以由log4j.properties文件控制。如果希望隐藏密码或任何与-d一起传递给spark的机密参数，可以通过删除spark-submit中的参数--verbose来隐藏。这对我有用