如何将Spark Dataframe写入HBase？

Question

我试图将spark dataframe写入hbase并关注了其他几个博客，其中一个就是这个，但它不起作用。

但是，我可以成功地从HBASE读取数据为dataframe。此外，一些帖子使用了org.apache.hadoop.hbase.spark格式，其他帖子使用了org.apache.spark.sql.execution.datasources.hbase格式。我不确定该用哪一个。Spark-2.2.2；HBASE-1.4.7；Scala-2.11.12和HortonWorks SHC 1.1.0-2.1-S2.11从这里开始。

守则如下：

case class UserMessageRecord(
                          rowkey: String,
                          Name: String,
                          Number: String,
                          message: String,
                          lastTS: String
                        )//this has been defined outside of the object scope

val exmple = List(UserMessageRecord("86325980047644033486","enrique","123455678",msgTemplate,timeStamp))

import spark.sqlContext.implicits._

val userDF = exmple.toDF()

//write to HBase
userDF.write
      .options(Map(HBaseTableCatalog.tableCatalog -> catalog))
      .format("org.apache.spark.sql.execution.datasources.hbase").save() //exception here

//read from HBase and it's working fine
def withCatalog(cat: String): DataFrame = {
      spark.sqlContext
    .read
    .options(Map(HBaseTableCatalog.tableCatalog->cat))
    .format("org.apache.spark.sql.execution.datasources.hbase")
    .load()
    }
    val df = withCatalog(catalog)
df.show()

这里有个例外：

在org.apache.hadoop.hbase.security.userprovider.instantiate（userprovider.java:122)在org.apache.hadoop.hbase.client.connectionfactory.createConnection（connectionfactory.java:214)在org.apache.hadoop.hbase.client.crenectionfactory.createConnection（connectionfactory.java:119)在.io.sparkhadoopmapreduceWriter$.write(sparkhadoopmapreduceWriter.scala:76)在org.apache.spark.rdd.pairrddfunctions$$anonfunt$saveasnewapihadoopdataSet$1.在org.apache.spark.rdd.pairrddfunctions.scale.apirrddfunctions.scale.apirrddfunctions.scala:1085)在S.Scala:1085)在org.apache.spark.rdd.rddOperationScope$.WithScope(rddOperationScope.scala:151)在org.apache.spark.rdd.rddOperatIonScope$.WithScope(rddoperationScope.scala:112)在org.apache.spark.rdd.rdd.rdd.wairrddfunctions.saveasnewapihadoopDataSet(Pairrddfunctions.scala:1084)在org.apache.spark.sql.execution.datasources.hbase.hbaserelation.inser(hbaserelation.scala:218)在sql.execution.datasources.datasource.writ(datasource.scala:469)在org.apache.spark.sql.execution.sql.execution.command.run（saveintodatasourceCommand.scala:48)在(commands.scala:74)在org.apache.spark.sql.execution.sparkplan$$anonfun$execute$1.apply(sparkplan.scala:117)在org.apache。spark.sql.execution.sparkplan$$anonfun$execute$1.Applic(sparkplan.scala:117)在org.apache.spark.sql.execution.sparkplan$$anonfun$executeQuery$1.Applic(sparkplan.scala:138)在org.apache.spark.rdd.rdperationScope$.withscope(rddoperationscala:151)在org.apache.spark.sql.execution.sparkplan.execution.query，在execution.tordd$lzycompute(queryexecution.scala:92)在org.apache.spark.sql.execution.queryexecution.tordd(queryexecution.scala:92)在org.apache.spark.sql.dataframewriter.runcommand(dataframewriter.scala:609)在org.apache.spark.sql.dataframewriter.scala:233)在scala）

Answer 1

正如在这里所讨论的，我对SparkSession builder进行了额外的配置更改，异常消失了。然而，我不清楚原因和解决办法。

val spark: SparkSession = SparkSession.builder()
  .master("local[1]")
  .appName("HbaseSparkWrite")
  .config("spark.hadoop.validateOutputSpecs", false)
  .getOrCreate()