如何在火花中打开 TRACE 日志记录

Question

我注意到在 Spark 中的规则探索器每次催化剂更改计划时都会执行跟踪日志：

https://github . com/Apache/spark/blob/78801881 c 405 de 47 f 7 e 53 EEA 3 e 0420 DD 69593 DBD/SQL/catalyst/src/main/Scala/org/Apache/spark/SQL/catalyst/rules/rule executor . Scala # L93

我想知道的是，如何配置 spark 以便打开跟踪日志记录？我正在使用log4j，并遇到了以下文档：https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-logging.html

我已经在代码中钻研了一段时间，我发现你可以设置“log4j”。threshold=TRACE”使记录器的一部分处于跟踪模式，但我似乎无法让catalyst使用记录器来获取设置。

我做错了什么？

Answer 1

我刚刚尝试了一个简单的结构化流程序，在IntelliJ中从Kafka读取数据，以下语句对我有效，即给我跟踪级别日志：

< code>SparkSession.builder()。getOrCreate()。sparkContext()。setlog level(" TRACE ")；

下面是显示一些跟踪日志的输出的一部分：

...
...
18/10/12 23:56:02 TRACE package$ExpressionCanonicalizer: Fixed point reached for batch CleanExpressions after 1 iterations.
18/10/12 23:56:02 TRACE package$ExpressionCanonicalizer: Batch CleanExpressions has no effect.
18/10/12 23:56:02 TRACE package$ExpressionCanonicalizer: Fixed point reached for batch CleanExpressions after 1 iterations.
18/10/12 23:56:02 TRACE package$ExpressionCanonicalizer: Batch CleanExpressions has no effect.
18/10/12 23:56:02 TRACE package$ExpressionCanonicalizer: Fixed point reached for batch CleanExpressions after 1 iterations.
18/10/12 23:56:02 TRACE package$ExpressionCanonicalizer: Batch CleanExpressions has no effect.
18/10/12 23:56:02 TRACE package$ExpressionCanonicalizer: Fixed point reached for batch CleanExpressions after 1 iterations.
18/10/12 23:56:02 TRACE package$ExpressionCanonicalizer: Batch CleanExpressions has no effect.
+-----+----+-----+-----------------------+
|topic|key |value|timestamp              |
+-----+----+-----+-----------------------+
|test |null|hi345|2018-10-12 23:56:00.099|
+-----+----+-----+-----------------------+
18/10/12 23:56:02 DEBUG GenerateUnsafeProjection: code for input[0, string, true],input[1, string, true],input[2, string, true],input[3, string, true]:
/* 001 */ public java.lang.Object generate(Object[] references) {
/* 002 */   return new SpecificUnsafeProjection(references);
/* 003 */ }
/* 004 */
/* 005 */ class SpecificUnsafeProjection extends org.apache.spark.sql.catalyst.expressions.UnsafeProjection {
/* 006 */
/* 007 */   private Object[] references;
...
...

希望这有帮助！