使用sbt在cmd中打包scala源代码：丢失“sparkcontext.class”时出错

Question

---SparkApp
  |---simple.sbt
  |---src
      |---main
          |---scala
              |--- SimpleApp.scala

name := "Simple Project"

version := "0.13.15"

scalaVersion := "2.11.8"

libraryDependencies += "org.apache.spark" %% "spark-core" % "2.1.1"

hadoop@master:~/Mycode/SparkApp$ sbt package
[warn] Executing in batch mode.
[warn]   For better performance, hit [ENTER] to switch to interactive mode, or
[warn]   consider launching sbt without any commands, or explicitly passing 'shell'
[info] Loading project definition from /home/hadoop/Mycode/SparkApp/project
[info] Set current project to Simple Project (in build file:/home/hadoop/Mycode/SparkApp/)
[info] Compiling 1 Scala source to /home/hadoop/Mycode/SparkApp/target/scala-2.11/classes...
[error] missing or invalid dependency detected while loading class file 'SparkContext.class'.
[error] Could not access term akka in package <root>,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'SparkContext.class' was compiled against an incompatible version of <root>.
[error] one error found
[error] (compile:compileIncremental) Compilation failed
[error] Total time: 2 s, completed May 16, 2017 1:08:53 PM

一些提示可能是问题所在：

1. 当我键入spark-shell时，我使用Scala版本2.11.8（Java HotSpot(TM)64位服务器VM，Java 1.8.0_131)获得了这一点，这与我键入java-version openjdk版本“1.8.0_121”时不同。这会是问题吗？
2. 安装SBT后我没有执行任何操作。我应该做一些设置吗？比如让sbt知道scala和spark的位置。怎么做？
3. 我没有maven，是吗？

------------------第二次编辑----------------

/* SimpleApp.scala */
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf

object SimpleApp {
  def main(args: Array[String]) {
    val logFile = "file:///usr/local/spark/README.md" // Should be some file on your system
    val conf = new SparkConf().setAppName("Simple Application")
    val sc = new SparkContext(conf)
    val logData = sc.textFile(logFile, 2).cache()
    val numAs = logData.filter(line => line.contains("a")).count()
    val numBs = logData.filter(line => line.contains("b")).count()
    println("Lines with a: %s, Lines with b: %s".format(numAs, numBs))
  }
}

Answer 1

如果您想开发Spark应用程序，您不必安装Spark。

在您作为Spark开发人员的早期（使用spark-shell和spark-submit)时，在本地安装Spark确实会有很大帮助，但强烈建议您不要这样做。

换句话说，您作为Spark包安装的内容与您在开发Spark应用程序时可以使用和想要使用的内容无关。

libraryDependencies += "org.apache.spark" %% "spark-core" % "2.1.1"

name := "Simple Project"

version := "0.13.15"

scalaVersion := "2.11.8"

libraryDependencies += "org.apache.spark" %% "spark-core" % "2.1.1"

sbt.version = 0.13.15

在执行SBT Package（在/home/hadoop/mycode/sparkapp中）时面临的问题是，您的应用程序定义了对Akka的依赖关系，您可以在错误消息中看到：

[info] Set current project to Simple Project (in build file:/home/hadoop/Mycode/SparkApp/)
[info] Compiling 1 Scala source to /home/hadoop/Mycode/SparkApp/target/scala-2.11/classes...
[error] missing or invalid dependency detected while loading class file 'SparkContext.class'.
[error] Could not access term akka in package <root>,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'SparkContext.class' was compiled against an incompatible version of <root>.
[error] one error found
[error] (compile:compileIncremental) Compilation failed

从Spark1.6开始，Akka就不再被Spark使用了，所以我想这个项目是以某种方式引用Akka库的，如果它们是Spark的，它就不应该引用。

很多猜测，我希望我们很快就能搞清楚。