在Apache Spark中捕获自定义异常

Question

ds.foreach(
        entry=> {
          validate(entry)
        })

case customException : CustomException =>
    //is never catched
case exception : SparkException =>
    //can be catched

有什么想法吗？

Answer 1

与其匹配基异常，不如尝试匹配原因：

import org.apache.spark.rdd.RDD

def ignoreArithmeticException(rdd: RDD[java.lang.Integer]) = try {
  rdd.foreach(1 / _)
} catch {
  case e: SparkException => e.getCause match  {
    case _: java.lang.ArithmeticException => 
      println("Ignoring ArithmeticException")
    case _ => throw e
  }
}

这将用于catch：

Try(ignoreArithmeticException(sc.parallelize(Seq(0))))

00/00/00 00:00:00 ERROR Executor: Exception in task 3.0 in stage 35.0 (TID 143)
java.lang.ArithmeticException: / by zero
    at
    ...
Ignoring ArithmeticException
res42: scala.util.Try[Unit] = Success(())

（虽然用了相当冗长的方式），但不会抓住：

Try(ignoreArithmeticException(sc.parallelize(Seq(null))))

00/00/00 00:00:00 ERROR Executor: Exception in task 3.0 in stage 38.0 (TID 155)
java.lang.NullPointerException
    at 
   ...
res52: scala.util.Try[Unit] =
Failure(org.apache.spark.SparkException: Job aborted due to stage failure: Task 3 in stage 38.0 failed 1 times, most recent failure: Lost task 3.0 in stage 38.0 (TID 155, localhost, executor driver): java.lang.NullPointerException ....