sparkcontext.setcheckpointdir（hdfsPath）可以在不同的spark应用程序中设置相同的hdfsPath吗？

Question

作为以下文件：

setCheckpointDir
public void setCheckpointDir(String directory)
Set the directory under which RDDs are going to be checkpointed.
Parameters:
directory - path to the directory where checkpoint files will be stored (must be HDFS path if running in cluster)

Answer 1

问题：

1）如果不同的spark应用程序sparkcontext.setcheckpointdir（hdfsPath）设置相同的hdfsPath，是否存在冲突？

答：根据下面的例子没有冲突。多个applcaition可以使用相同的检查点目录。在这个唯一的哈希类型下，将创建一个文件夹以避免冲突。

package examples

import java.io.File

import org.apache.log4j.Level


object CheckPointTest extends App {
  import org.apache.spark.sql.{Dataset, SparkSession}
  val spark = SparkSession.builder().appName("CheckPointTest").master("local").getOrCreate()
  val logger = org.apache.log4j.Logger.getLogger("org")
  logger.setLevel(Level.WARN)
  import spark.implicits._

  spark.sparkContext.setCheckpointDir("/tmp/checkpoints")


  val csvData1: Dataset[String] = spark.sparkContext.parallelize(
    """
      |id
      | a
      | b
      | c

    """.stripMargin.lines.toList).toDS()
  val frame1 = spark.read.option("header", true).option("inferSchema",true).csv(csvData1).show

  val checkpointDir = spark.sparkContext.getCheckpointDir.get
  println(checkpointDir)

println("Number of Files in Check Point Directory " + getListOfFiles(checkpointDir).length)


  def getListOfFiles(dir: String):List[File] = {
    val d = new File(dir)
    if (d.exists && d.isDirectory) {
      d.listFiles.filter(_.isFile).toList
    } else {
      List[File]()
    }
  }
}

+---+
| id|
+---+
|  a|
|  b|
|  c|
+---+

file:/tmp/checkpoints/30e6f882-b49a-42cc-9e60-59adecf13166
Number of Files in Check Point Directory 0 // this indicates once application finished removed all the RDD/DS information.

user@f0189843ecbe [~/Downloads]$ ll /tmp/checkpoints/
total 0
drwxr-xr-x  2 user  wheel   64 Mar 27 14:08 a2396c08-14b6-418a-b183-a90a4ca7dba3
drwxr-xr-x  2 user  wheel   64 Mar 27 14:09 65c8ef5a-0e64-4e79-a050-7d1ee1d0e03d
drwxr-xr-x  2 user  wheel   64 Mar 27 14:09 5667758c-180f-4c0b-8b3c-912afca59f55
drwxr-xr-x  2 user  wheel   64 Mar 27 14:10 30e6f882-b49a-42cc-9e60-59adecf13166
drwxr-xr-x  6 user  wheel  192 Mar 27 14:10 .
drwxrwxrwt  5 root    wheel  160 Mar 27 14:10 ..
user@f0189843ecbe [~/Downloads]$ du -h /tmp/checkpoints/
  0B    /tmp/checkpoints//a2396c08-14b6-418a-b183-a90a4ca7dba3
  0B    /tmp/checkpoints//5667758c-180f-4c0b-8b3c-912afca59f55
  0B    /tmp/checkpoints//65c8ef5a-0e64-4e79-a050-7d1ee1d0e03d
  0B    /tmp/checkpoints//30e6f882-b49a-42cc-9e60-59adecf13166
  0B    /tmp/checkpoints/

1）即使多个应用程序运行parllel,check point目录下也会存在唯一的哈希，因为所有的RDD/DS信息都将被存储。

2）在成功地完全执行每个Spark应用程序之后，上下文清理器将删除其中的内容…这是我从上面的实际示例中观察到的。