Spark 2.2查询配置单元表时dataframe NumberFormatException上的Thrift服务器错误

Question

我有运行Spark2(V2.2)的Hortonworks HDP 2.6.3。我的测试用例非常简单：

>

使用一些随机值创建配置单元表。端口10000上的配置单元

在10016打开Spark Thrift服务器

null

beeline> !connect jdbc:hive2://localhost:10000/default hive hive

create table test1 (id int, desc varchar(40));

insert into table test1 values (1,"aa"),(2,"bb");

我有一个错误：

我怀疑它与id列有关，所以我更改为：df.select（“desc”）.show（）

然后我得到了这个奇怪的结果：

+----+
|desc|
+----+
|desc|
|desc|
+----+

beeline> !connect jdbc:hive2://localhost:10016/default hive hive

select * from test1;

+-----+-------+--+
| id  | desc  |
+-----+-------+--+
| 1   | aa    |
| 2   | bb    |
+-----+-------+--+

create table test4 (id String, desc String);
insert into table test4 values ("1","aa"),("2","bb");
select * from test4;

>>> df = sqlContext.read.format("jdbc").options(driver="org.apache.hive.jdbc.HiveDriver", url="jdbc:hive2://localhost:10016/default", dbtable="test4",user="hive", password="hive").option("fetchsize", "10").load()
>>> df.select("*").show()
+---+----+
| id|desc|
+---+----+
| id|desc|
| id|desc|
+---+----+

由于某种原因，它返回了列名？！

案例2

直线：

create table test5 (id int, desc varchar(40)) STORED AS ORC;
insert into table test5 values (1,"aa"),(2,"bb");
select * from test5;

beeline> !connect jdbc:hive2://localhost:10000/default hive hive
Connecting to jdbc:hive2://localhost:10000/default
Connected to: Apache Hive (version 1.2.1000.2.5.3.0-37)
Driver: Hive JDBC (version 1.2.1000.2.5.3.0-37)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://localhost:10000/default> create table test2 (id String, desc String) STORED AS ORC;
No rows affected (0.3 seconds)
0: jdbc:hive2://localhost:10000/default> insert into table test2 values ("1","aa"),("2","bb");
INFO  : Session is already open
INFO  : Dag name: insert into table tes..."1","aa"),("2","bb")(Stage-1)
INFO  : Tez session was closed. Reopening...
INFO  : Session re-established.
INFO  :

INFO  : Status: Running (Executing on YARN cluster with App id application_1514019042819_0006)

INFO  : Map 1: -/-
INFO  : Map 1: 0/1
INFO  : Map 1: 0(+1)/1
INFO  : Map 1: 1/1
INFO  : Loading data to table default.test2 from webhdfs://demo.myapp.local:40070/apps/hive/warehouse/test2/.hive-staging_hive_2017-12-23_04-29-54_569_601147868480753216-3/-ext-10000
INFO  : Table default.test2 stats: [numFiles=1, numRows=2, totalSize=317, rawDataSize=342]
No rows affected (15.414 seconds)
0: jdbc:hive2://localhost:10000/default> select * from table2;
Error: Error while compiling statement: FAILED: SemanticException [Error 10001]: Line 1:14 Table not found 'table2' (state=42S02,code=10001)
0: jdbc:hive2://localhost:10000/default> select * from test2;
+-----------+-------------+--+
| test2.id  | test2.desc  |
+-----------+-------------+--+
| 1         | aa          |
| 2         | bb          |
+-----------+-------------+--+
2 rows selected (0.364 seconds)

beeline> !connect jdbc:hive2://localhost:10016/default hive hive
Connecting to jdbc:hive2://localhost:10016/default
1: jdbc:hive2://localhost:10016/default> create table test3 (id String, desc String) STORED AS ORC;
+---------+--+
| Result  |
+---------+--+
+---------+--+
No rows selected (1.234 seconds)
1: jdbc:hive2://localhost:10016/default> insert into table test3 values ("1","aa"),("2","bb");
+---------+--+
| Result  |
+---------+--+
+---------+--+
No rows selected (9.111 seconds)
1: jdbc:hive2://localhost:10016/default> select * from test3;
+-----+-------+--+
| id  | desc  |
+-----+-------+--+
| 1   | aa    |
| 2   | bb    |
+-----+-------+--+
2 rows selected (3.387 seconds)

>>> df = sqlContext.read.format("jdbc").options(driver="org.apache.hive.jdbc.HiveDriver", url="jdbc:hive2://localhost:10016/default", dbtable="test3",user="hive", password="hive").load()
>>> df.select("*").show()
+---+----+
| id|desc|
+---+----+
+---+----+

# Detailed Table Information  | CatalogTable(
    Table: `default`.`test3`
    Owner: hive
    Created: Sat Dec 23 04:37:14 PST 2017
    Last Access: Wed Dec 31 16:00:00 PST 1969
    Type: MANAGED
    Schema: [`id` string, `desc` string]
    Properties: [totalSize=620, numFiles=2, transient_lastDdlTime=1514032656, STATS_GENERATED_VIA_STATS_TASK=true]
    Storage(Location: webhdfs://demo.myapp.local:40070/apps/hive/warehouse/test3, InputFormat: org.apache.hadoop.hive.ql.io.orc.OrcInputFormat, OutputFormat: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat, Serde: org.apache.hadoop.hive.ql.io.orc.OrcSerde, Properties: [serialization.format=1]))

显示CREATE TABLE TEST3；

CREATE TABLE `test3`(`id` string, `desc` string)
ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
WITH SERDEPROPERTIES (
  'serialization.format' = '1'
)
STORED AS
  INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
  OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
TBLPROPERTIES (
  'totalSize' = '620',
  'numFiles' = '2',
  'transient_lastDdlTime' = '1514032656',
  'STATS_GENERATED_VIA_STATS_TASK' = 'true'
)

su-spark-c“hdfs dfs-cat webhdfs：//demo.myapp.local:40070/apps/hive/warehouse/test3/part-00000”

Answer 1

即使您使用特定的数据类型创建配置单元表，插入时表中的基础数据还是以字符串格式存储。

所以当spark试图读取数据时，它会使用转移子来查找数据类型。它在hive metaxore中以int形式出现，在文件中以string形式出现，并抛出cast异常。

解决方案

create table test1 (id String, desc String);

 create table test1 (id int, desc varchar(40) STORED AS ORC);