与toDF有关的问题是，值toDF不是组织的成员。阿帕奇。火花rdd。RDD

Question

我附加了错误的代码片段“值toDF不是org.apache.spark.rdd.RDD的成员”。我正在使用scala 2.11.8和火花2.0.0。你能帮我解决API toDF（）的这个问题吗？

import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.SQLContext
import org.apache.spark.SparkConf
import org.apache.spark.sql.{Row, SparkSession}
import org.apache.spark.sql.functions._

object HHService {
    case class Services(
    uhid:String,
    locationid:String,
    doctorid:String,
    billdate:String,
    servicename:String,
    servicequantity:String,
    starttime:String,
    endtime:String,
    servicetype:String,
    servicecategory:String,
    deptname:String
    )

    def toService = (p: Seq[String]) => Services(p(0), p(1),p(2),p(3),p(4),p(5),p(6),p(7),p(8),p(9),p(10))

    def main(args: Array[String]){
        val warehouseLocation = "file:${system:user.dir}/spark-warehouse"
        val spark = SparkSession
            .builder
            .appName(getClass.getSimpleName)
            .config("spark.sql.warehouse.dir", warehouseLocation)
        .enableHiveSupport()
            .getOrCreate()
        val sc = spark.sparkContext 

        val sqlContext = spark.sqlContext;

        import spark.implicits._
        import sqlContext.implicits._

        val hospitalDataText = sc.textFile("D:/Books/bboks/spark/Intellipaat/Download/SparkHH/SparkHH/services.csv")
        val header = hospitalDataText.first()
        val hospitalData= hospitalDataText.filter(a => a!= header)
        //val HData = hospitalData.map(_.split(",")).map(p=>Services(p(0), p(1),p(2),p(3),p(4),p(5),p(6),p(7),p(8),p(9),p(10)))
        val HData = hospitalData.map(_.split(",")).map(toService(_))

        val hosService=HData.toDF()
    }

}

Answer 1

在DataBricks中使用笔记本电脑将代码转换为简单函数时也遇到了同样的问题。不得不让学生退出函数，一切都很顺利：

%scala

case class className(param1 : String,
                      param2 : String,
                      ...
                      lastoaram : Double)

def myFunction(params) = {
a lot of code 
...
var myVarBasedOnClasseDefinition = Seq(myVarBasedOnClasseDefinition ("init","init","init",0.0,0.0,"init",0.0))
for(iteration <- iterator) myVarBasedOnClasseDefinition = myVarBasedOnClasseDefinition ++ additionnalSequence
display(myVarBasedOnClasseDefinition.toDF())
}

希望这会有所帮助，因为句子"case class to be out of method"在我的搜索开始时似乎并没有真正适用于我的案例，使用像代码这样的程序运行良好。

Answer 2

1] 需要获得如下所示的sqlContext。

val sqlContext = new org.apache.spark.sql.SQLContext(sc)
import sqlContext.implicits._

这解决了我的问题。下面的代码片段用于获取sqlContext. val sqlContext=spark.sqlContext（这样它就可以使用spack-shell）

2]case类需要脱离方法。这在大多数博客中也提到过。