使用sbt测试运行时Spark测试失败

Question

我们已经为spark编写了单元测试，在本地模式下有4个线程。

当一个接一个地启动时，例如通过intellij或sbt testOnly，每个测试都运行良好。

当用sbt测试启动时，它们会出现如下错误

libraryDependencies ++= Seq(
  Dependencies.Test.concordion,
  Dependencies.`spark-sql` exclude("org.slf4j","slf4j-log4j12"),
  Dependencies.`better-files`
)

fork in test := true


dependencyOverrides += "com.google.guava" % "guava" % "11.0.2" 
dependencyOverrides += "com.fasterxml.jackson.core" % "jackson-databind" % "2.6.7.1"

我们使用的是一个带有多个子项目的sbt项目，其定义如下：

scalacOptions in ThisBuild ++= Seq(
  "-encoding", "UTF-8", // source files are in UTF-8
  "-deprecation", // warn about use of deprecated APIs
  "-Yrangepos", // use range positions for syntax trees
  "-language:postfixOps", //  enables postfix operators
  "-language:implicitConversions", // enables defining implicit methods and members
  "-language:existentials", // enables writing existential types
  "-language:reflectiveCalls", // enables reflection
  "-language:higherKinds", // allow higher kinded types without `import scala.language.higherKinds`
  "-unchecked", // warn about unchecked type parameters
  "-feature", // warn about misused language features
  /*"-Xlint",               // enable handy linter warnings
    "-Xfatal-warnings",     // turn compiler warnings into errors*/
  "-Ypartial-unification" // allow the compiler to unify type constructors of different arities
)

autoCompilerPlugins := true

addCompilerPlugin(Dependencies.`kind-projector`)
addCompilerPlugin(Dependencies.`better-monadic-for`)


// Define the root project, and make it compile all child projects
lazy val `datarepo` =
  project
    .in(file("."))
    .aggregate(
      `foo`,
      `foo-other`,
      `sparkusingproject`,
      `sparkusingproject-test`,
      `sparkusingproject-other`,
    )

// Define individual projects, the directories they reside in, and other projects they depend on
lazy val `foo` =
  project
    .in(file("foo"))
    .settings(Common.defaultSettings: _*)

lazy val `foo-other` =
  project
    .in(file("foo-other"))
    .dependsOn(`foo`)
    .settings(Common.defaultSettings: _*)

Answer 1

我只是在一个测试中遇到了这个异常，它是由于试图在一个线程中运行Spark操作而导致的，这个线程与我启动sparksession的线程不同。您可能希望在test中禁用ParallelExecution（无论如何，对于Spark集成测试，这是建议的）。

具体地说，我试图并行执行多个Spark操作，并且在Scala的executioncontext.global线程池中尝试执行这些操作。当我创建NewFixedPoolExecutor时，一切都开始正常工作。

AFAICT这是因为在datasource.scala:610中，Spark获得线程的ContextClassLoader：

    val loader = Utils.getContextOrSparkClassLoader