Spark 2.0 Cassandra Scala Shell错误：NoClassDefFoundError

Question

我已经将Spark2.0 shell配置为与datastax cassandra连接器一起运行。

spark-shell --packages datastax:spark-cassandra-connector:1.5.1-s_2.11

sc.stop
import org.apache.spark
import org.apache.spark._
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
import com.datastax.spark
import com.datastax.spark._
import com.datastax.spark.connector
import com.datastax.spark.connector._
import com.datastax.spark.connector.cql
import com.datastax.spark.connector.cql._
import com.datastax.spark.connector.cql.CassandraConnector
import com.datastax.spark.connector.cql.CassandraConnector._

val conf = new SparkConf(true).set("spark.cassandra.connection.host", "dbserver")
val sc = new SparkContext("spark://localhost:7077", "test", conf)
val table = sc.cassandraTable("keyspace", "users")
println(table.count)
println(table.first)

scala> val table = sc.cassandraTable("keyspace", "users")

java.lang.NoClassDefFoundError: com/datastax/spark/connector/cql/CassandraConnector$
at com.datastax.spark.connector.SparkContextFunctions.cassandraTable$default$3(SparkContextFunctions.scala:48)
... 62 elided

Answer 1

正如我已经说过的，一个选择是构建一个包含所有cassandra连接器依赖项的胖罐子。你可以这样做：

$ git clone https://github.com/datastax/spark-cassandra-connector.git
$ cd spark-cassandra-connector
$ sbt assembly

然后简单地通过命令行参数将本地jar注入spark shell。