使用phoenix连接器将Spark dataframe写入Hbase

Question

sql.write().format("org.apache.phoenix.spark")
    .mode(SaveMode.Overwrite).option("table", targetTable)
    .option("zkUrl", "localhost:2181:/hbase-unsecure)
    .insertInto(targetTable);

java.lang.NullPointerException
at org.apache.phoenix.hive.PhoenixStorageHandler.configureJobProperties(PhoenixStorageHandler.java:185)
at org.apache.phoenix.hive.PhoenixStorageHandler.configureOutputJobProperties(PhoenixStorageHandler.java:130)
at org.apache.spark.sql.hive.HiveTableUtil$.configureJobPropertiesForStorageHandler(TableReader.scala:324)
at org.apache.spark.sql.hive.SparkHiveWriterContainer.<init>(hiveWriterContainers.scala:67)
at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.sideEffectResult$lzycompute(InsertIntoHiveTable.scala:226)
at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.sideEffectResult(InsertIntoHiveTable.scala:142)
at org.apache.spark.sql.hive.execution.InsertIntoHiveTable.doExecute(InsertIntoHiveTable.scala:310)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:115)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:136)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:133)
at org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:114)
at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:86)
at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:86)
at org.apache.spark.sql.DataFrameWriter.insertInto(DataFrameWriter.scala:259)
at org.apache.spark.sql.DataFrameWriter.insertInto(DataFrameWriter.scala:239)
at com.lti.unitrax.data.load.IncrementalHiveTableLoadUnitraxMain.fullDataLoad(IncrementalHiveTableLoadUnitraxMain.java:166)
at com.lti.unitrax.data.load.TestDataLoad.main(TestDataLoad.java:38)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:627)

Answer 1

我知道我在比赛中迟到了，但看到了这个帖子。以为我的答案能帮到别人。

以下是我在实现中所做的工作

1. df=dataframe
2. zookeeperurl=集群的zookeeper URL
3. _tgttable=要在其中写入数据的表

df.write.format("org.apache.phoenix.spark")
  .mode(org.apache.spark.sql.SaveMode.Overwrite)
  .options(collection.immutable.Map("zkUrl" -> zookeeperURL, "table" -> _tgtTable)) 
  .save()