Spark Hbase连接器(SHC)没有从Hbase表返回任何数据

Question

我正在按照spark hbase connector basic示例阅读spark2 shell version 2.2.0中的hbase表。代码看起来很正常，但是当我运行df.show（）命令时，我看不到任何结果，它似乎永远都在运行。

import org.apache.spark.sql.{ DataFrame, Row, SQLContext }
import org.apache.spark.sql.execution.datasources.hbase._

val sqlContext = new org.apache.spark.sql.SQLContext(sc); 

def catalog = s"""{
         |"table":{"namespace":"default", "name":"testmeta"},
         |"rowkey":"vgil",
         |"columns":{
            |"id":{"cf":"rowkey", "col":"vgil", "type":"string"},
           |"col1":{"cf":"pp", "col":"dtyp", "type":"string"}
         |}
       |}""".stripMargin


def withCatalog(cat: String): DataFrame = { sqlContext.read.options(Map(HBaseTableCatalog.tableCatalog->cat)).format("org.apache.spark.sql.execution.datasources.hbase").load()}

val df = withCatalog(catalog)

df.show()

hbase(main):001:0> scan 'testmeta'
ROW                                 COLUMN+CELL                                                                                            
 fmix                            column=pp:dtyp, timestamp=1541714925380, value=ss1                                                     
 fmix                            column=pp:lati, timestamp=1541714925371, value=41.50                                                   
 fmix                            column=pp:long, timestamp=1541714925374, value=-81.61                                                  
 fmix                            column=pp:modm, timestamp=1541714925377, value=ABC                                                                                                   
 vgil                            column=pp:dtyp, timestamp=1541714925405, value=ss2                                                     
 vgil                            column=pp:lati, timestamp=1541714925397, value=41.50                                                   

提前感谢您的帮助！

Answer 1

经过大量的尝试，我发布了我的答案，所以我发现添加--conf选项来启动spark shell有助于我连接到HBase。

spark2-shell --master yarn --deploy-mode client --packages com.hortonworks:shc-core:1.1.1-2.1-s_2.11,it.nerdammer.bigdata:spark-hbase-connector_2.10:1.0.3 --repositories http://repo.hortonworks.com/content/groups/public/ --conf spark.hbase.host=192.168.xxx.xxx --files /mnt/fs1/opt/cloudera/parcels/CDH-5.13.0-1.cdh5.13.0.p0.29/share/doc/hbase-solr-doc-1.5+cdh5.13.0+71/demo/hbase-site.xml

然后，下面的代码片段可以获取一个列限定符的值。

val hBaseRDD_iacp = sc.hbaseTable[(String)]("testmeta").select("lati").inColumnFamily("pp").withStartRow("vg").withStopRow("vgz")      
object myschema {
      val column1 = StructField("column1",  StringType)
      val struct = StructType(Array(column1))
    }


val rowRDD = hBaseRDD.map(x => Row(x))
val myDf = sqlContext.createDataFrame(rowRDD,myschema.struct)  
myDf.show()