当尝试使用spark-cassandra-connector将Cassandra行映射到case对象时，找不到此类型错误的RowReaderFactory

Question

我试图得到一个简单的示例，使用Apache Spark 1.1.1、Cassandra 2.0.11和spark-cassandra-connector(V1.1.0)将行从Cassandra映射到scala case类。我在spark-cassandra-connector github页面、planetcassandra.org、datastax上查看了文档，并进行了搜索；但没有发现其他人遇到这个问题。所以来了...

case class SubHuman(id:String, firstname:String, lastname:String, isGoodPerson:Boolean)
val foo = sc.cassandraTable[SubHuman]("nicecase", "human").select("id","firstname","lastname","isGoodPerson").toArray

No RowReaderFactory can be found for this type

编译错误只是稍微冗长一些：

> compile
[info] Compiling 1 Scala source to /home/bkarels/dev/simple-case/target/scala-2.10/classes...
[error] /home/bkarels/dev/simple-case/src/main/scala/com/bradkarels/simple/SimpleApp.scala:82: No RowReaderFactory can be found for this type
[error]     val foo = sc.cassandraTable[SubHuman]("nicecase", "human").select("id","firstname","lastname","isGoodPerson").toArray
[error]                                          ^
[error] one error found
[error] (compile:compile) Compilation failed
[error] Total time: 1 s, completed Dec 10, 2014 9:01:30 AM
>

Scala源代码：

package com.bradkarels.simple

import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
import com.datastax.spark.connector._
import com.datastax.spark.connector.rdd._
// Likely don't need this import - but throwing darts hits the bullseye once in a while...
import com.datastax.spark.connector.rdd.reader.RowReaderFactory

object CaseStudy {

  def main(args: Array[String]) {
    val conf = new SparkConf(true)
      .set("spark.cassandra.connection.host", "127.0.0.1")

    val sc = new SparkContext("spark://127.0.0.1:7077", "simple", conf)

    case class SubHuman(id:String, firstname:String, lastname:String, isGoodPerson:Boolean)
    val foo = sc.cassandraTable[SubHuman]("nicecase", "human").select("id","firstname","lastname","isGoodPerson").toArray
  }
}

去掉了那些烦人的代码行，所有的代码编译都很好，组装工作正常，我可以正常执行其他Spark操作。例如，如果我删除问题行并插入：

val rdd:CassandraRDD[CassandraRow] = sc.cassandraTable("nicecase", "human")

name := "Simple Case"

version := "0.0.1"

scalaVersion := "2.10.4"

libraryDependencies ++= Seq(
    "org.apache.spark" %% "spark-core" % "1.1.1",
    "org.apache.spark" %% "spark-sql" % "1.1.1",
    "com.datastax.spark" %% "spark-cassandra-connector" % "1.1.0" withSources() withJavadoc()
  )

所以问题是我错过了什么？希望这是一些愚蠢的事情，但如果你遇到了这一点，并能帮助我度过这个令人困惑的小问题，我将非常感谢。请让我知道，如果有任何其他细节，将有助于故障排除。

谢谢你。

Answer 1

总的来说，这可能是我对Scala的新手，但我通过将case类声明移出main方法来解决这个问题。简化后的源代码如下所示：

package com.bradkarels.simple

import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
import com.datastax.spark.connector._
import com.datastax.spark.connector.rdd._

object CaseStudy {

  case class SubHuman(id:String, firstname:String, lastname:String, isGoodPerson:Boolean)

  def main(args: Array[String]) {
    val conf = new SparkConf(true)
      .set("spark.cassandra.connection.host", "127.0.0.1")

    val sc = new SparkContext("spark://127.0.0.1:7077", "simple", conf)

    val foo = sc.cassandraTable[SubHuman]("nicecase", "human").select("id","firstname","lastname","isGoodPerson").toArray
  }
}

完整的源代码（更新和修正）可以在github https://github.com/bradkarels/spark-cassandra-to-scala-case-class上找到