build.sbt：如何添加火花依赖

Question

您好，我正在尝试在构建中下载spark core，spark streaming，twitter4j，以及spark streaming twitter。sbt文件如下：

name := "hello"

version := "1.0"

scalaVersion := "2.11.8"

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.6.1"
libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.4.1"

libraryDependencies ++= Seq(
  "org.twitter4j" % "twitter4j-core" % "3.0.3",
  "org.twitter4j" % "twitter4j-stream" % "3.0.3"
)

libraryDependencies += "org.apache.spark" % "spark-streaming-twitter_2.10" % "0.9.0-incubating"

我只是把这个库依赖联机，所以我不确定使用哪个版本等。

有人能向我解释一下我应该如何修复这个. sbt文件吗？我花了几个小时试图弄清楚，但没有一个建议奏效。我安装了scala通过家酿和我的版本2.11.8

我所有的错误都是关于：

Modules were resolved with conflicting cross-version suffixes.

Answer 1

这对我很有用：

name := "spark_local"

version := "0.1"

scalaVersion := "2.11.8"


libraryDependencies ++= Seq(
  "org.twitter4j" % "twitter4j-core" % "3.0.5",
  "org.twitter4j" % "twitter4j-stream" % "3.0.5",
  "org.apache.spark" %% "spark-core" % "2.0.0",
  "org.apache.spark" %% "spark-sql" % "2.0.0",
  "org.apache.spark" %% "spark-mllib" % "2.0.0",
  "org.apache.spark" %% "spark-streaming" % "2.0.0"
)

Answer 2

问题是您混合了Scala 2.11和2.10工件。你有：

scalaVersion := "2.11.8"

然后呢：

libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.4.1"

其中需要2.10工件。您还混合了Spark版本，而不是使用一致的版本：

// spark 1.6.1
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.6.1"

// spark 1.4.1
libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.4.1"

// spark 0.9.0-incubating
libraryDependencies += "org.apache.spark" % "spark-streaming-twitter_2.10" % "0.9.0-incubating"

下面是一个修复这两个问题的build.sbt：

name := "hello"

version := "1.0"

scalaVersion := "2.11.8"

val sparkVersion = "1.6.1"

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % sparkVersion,
  "org.apache.spark" %% "spark-streaming" % sparkVersion,
  "org.apache.spark" %% "spark-streaming-twitter" % sparkVersion
)

您也不需要手动添加twitter4j依赖项，因为它们是通过spark streaming twitter以传递方式添加的。