使用SavetPhoenix方法在Hbase上加载/保存RDD时出现异常

Question

import org.apache.spark.SparkContext
import org.apache.spark.sql._
import org.apache.phoenix.spark._

// Load INPUT_TABLE
object MainTest2 extends App {
  val sc = new SparkContext("local", "phoenix-test")
  val sqlContext = new SQLContext(sc)
  val df = sqlContext.load("org.apache.phoenix.spark", Map("table" -> "INPUT_TABLE",
    "zkUrl" -> "localhost:3888"))
}

name := "spark-to-hbase"

version := "1.0"

scalaVersion := "2.11.8"

libraryDependencies ++= Seq(
  "org.apache.hadoop" % "hadoop-mapreduce-client-core" % "2.3.0",
  "org.apache.phoenix" % "phoenix-core" % "4.11.0-HBase-1.3",
  "org.apache.spark" % "spark-core_2.11" % "2.1.1",
  "org.apache.spark" % "spark-sql_2.11" % "2.1.1",
  "org.apache.phoenix" % "phoenix-spark" % "4.11.0-HBase-1.3"
)

线程“main”java.lang.noClassDefounderror：org.apache.phoenix.query.configurationFactory$configurationFactory.java:49)在org.apache.phoenix.query.configurationFactory$configurationFactory$configurationFactory.java:46)在org.apache.phoenix.util.phoenixContextExecutor.call（phoenixContextExecutor.java:76)在在org.apache.phoenix.jdbc.phoenix.jdbc.phoenixDriver.InitializeConnectionCache(PhoenixDriver.java:151)在org.apache.phoenix.jdbc.phoenixDriver.phoenixDriver.phoenixDriver.(phoenxDriver.java:142)在org.apache.phoenix.jdbc.phoenixDriver.(phoenxDriver.java:69)在在org.apache.spark.sql.execution.datasources.logicalRelation.(logicalRelation.scala:40)在org.apache.spark.sql.sparksession.baserelationtoDataFrame(Sparksession.scala:389)在org.apache.spark.sql.dataframe.load(DataframeReader.scala:146)在org.apache.spark.sql.dataframeReader.load(DataframeReader.scala:125)在maintest2$1(maintest2.scala:9)在maintest2$delayedinit$body.apply(maintest2.scala:6)在scala.function0$class.apply$mcv$sp(function0.scala:34)在scala.runtime.abstractfunction0.apply$mcv$sp(abstractfunction0.scala:12)在scala.app$$anonfunin$main$1.apply(app.scala:76)在scala.app$$anonfunt$main$1在ableforwarder.scala:35)位于scala.app$class.main(app.scala:76)，位于maintest2$.main(maintest2.scala:6)，位于maintest2.main(maintest2.scala)，原因如下：java.lang.ClassNotFoundException:org.apache.hadoop.hbase.hbaseConfiguration在java.net.urlClassLoader.findClass(urlClassLoader.java:381)在java.lang.ClassLoader.loadClass(classLoader.java:424)在sun.misc.launcher$appClassLoader.loadClass(launcher.java:331)在java.lang.ClassLoader.loadClass(classLoader.java:357)...26多个

我已经尝试更改hadoop-env.sh中的HADOOP_CLASSPATH，就像上一篇文章中所说的那样。

我能做些什么来克服这个问题？

Answer 1

我找到了解决问题的办法。正如异常所说，我的编译器无法找到类HBaseConfiguration。HBaseConfiguration在org.apache.hadoop.hbase库中使用，因此需要进行编译。我注意到，HBaseConfiguration类并不像我想象的那样存在于org.apache.Hadoop库中。对于安装在我的PC计算机上的hbase 1.3.1版本，我设法在位于我的hbase_home/lib文件夹中的hbase-common-1.3.1 jar中找到了这个类。

然后在构建中包含此依赖项。sbt：

“org.apache.hbase”%“hbase-公共”%“1.3.1”