问题： hive我修改了 默认的f <property> <name>fs.defaultFS</name> <value>hdfs://。。。:8020</value> </property> 修改为<property> <name>fs.defaultFS</name> <value>s3://。。。</value> </property> 修改后执行窗口函数的时候会执行失败,如果不执行窗口函数的没问题。
失败hive命令简化版：select * ,row_number() over (partition by id order by utime desc) as sort_num from mobi_warehouse.table ； 成功命令简化版：select * from mobi_warehouse.table；
错误：FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask
查看hive运行日志，发现报错如下：
INFO [main([])]: session.SessionState (SessionState.java:close(1573)) - Error closing tez session java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.io.IOException: Incomplete HDFS URI, no host: hdfs:///apps/tez/tez.tar.gz at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.isOpen(TezSessionState.java:163) ~[hive-exec-2.3.2-amzn-2.jar:2.3.2-amzn-2] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.toString(TezSessionState.java:125) ~[hive-exec-2.3.2-amzn-2.jar:2.3.2-amzn-2] .....
这个错误指明无法找到 tez 执行引擎的 library 档案，这个路径 hdfs:///apps/tez/tez.tar.gz 表示它存储于 HDFS 上。 由于您已修改 fs.defaultFS 至 S3 桶，因此可行的解决方法是将 tez.tar.gz 也复制到您的 S3 桶中。 以下是我的操作步骤。
1. 将 tez.tar.gz 从 HDFS 上复制到 master node 的本地磁盘 $ hdfs dfs -copyToLocal /apps/tez/tez.tar.gz ./
2. 将 tez.tar.gz 复制到 S3 桶中 $ aws s3 cp ./tez.tar.gz s3://S3桶名称/
3. 修改 /etc/tez/conf/tez-site.xml, 找到 tez.lib.uris property, 并修改其值如下
<property>
<name>tez.lib.uris</name>
<value>s3://S3桶名称/tez.tar.gz</value>
</property>
修改后执行，发现仍然报如下错误：
Caused by: java.util.concurrent.ExecutionException: java.io.IOException: The ownership on the staging directory s3://mobimagic-microl-risk/tmp/hive/hadoop/_tez_session _dir/5db8dd84-5154-460d-9a6d-f875adb6fd6b is not as expected. It is owned by . The directory must be owned by the submitter hadoop or by hadoop at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_121] at java.util.concurrent.FutureTask.get(FutureTask.java:206) ~[?:1.8.0_121] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.isOpen(TezSessionState.java:168) ~[hive-exec-2.1.1-amzn-0.jar:2.1.1-amzn-0] ... 14 more Caused by: java.io.IOException: The ownership on the staging directory s3://mobimagic-microl-risk/tmp/hive/hadoop/_tez_session_dir/5db8dd84-5154-460d-9a6d-f875adb6fd6b is not as expected. It is owned by . The directory must be owned by the submitter hadoop or by hadoop
最终发现，当前hadoop版本是亚马逊的2.7.2，升级到2.8.3就根据如上配置就没问题了，把emr版本提高到5.12以上就可以了
如果帮到你了，请点下赞
转载注明出处