无法在spark-shell中连接到Cassandra

Question

我正在尝试使用Spark-cassandra连接器连接到spark shell中的远程cassandra集群。但它抛出了一些不寻常的错误。

我按照spark-cassandra connector的github页面中提到的常规操作

 $SPARK_HOME/bin/spark-shell --packages datastax:spark-cassandra-connector:2.0.0-s_2.11

import com.datastax.spark.connector._
import org.apache.spark.sql.cassandra._

 val rdd=sc.cassandraTable("test","user")

    java.lang.NoClassDefFoundError: org/apache/commons/configuration/ConfigurationException
  at java.lang.ClassLoader.defineClass1(Native Method)
  at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
  at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
  at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
  at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
  at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
  at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
  at java.security.AccessController.doPrivileged(Native Method)
  at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
  at com.datastax.spark.connector.cql.CassandraConnectorConf$.apply(CassandraConnectorConf.scala:257)
  at com.datastax.spark.connector.cql.CassandraConnector$.apply(CassandraConnector.scala:189)
  at com.datastax.spark.connector.SparkContextFunctions.cassandraTable$default$3(SparkContextFunctions.scala:52)
  ... 53 elided
Caused by: java.lang.ClassNotFoundException: org.apache.commons.configuration.ConfigurationException
  at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

Scala Version-2.3.1连接器版本=2.1.0

Answer 1

对于Scala 2.3.1版本，您必须使用Spark Connector version>=2.3.0-当前版本是2.3.2:

spark-shell --packages com.datastax.spark:spark-cassandra-connector_2.11:2.3.2 \
    --conf spark.cassandra.connection.host=<IP>