Oozie蜂巢行动悬而未决，心脏永远跳动

Question

我试图通过我在顺化中创建的Oozie工作流运行Hive操作，但是操作“心跳”是永远的，并且不会执行HiveSQL。

我读过其他关于心脏永远跳动的文章，但这篇文章似乎发生在另一个地方，在SQL语句被解析之后。我检查了集群中每个节点的内存，并验证了任务计数参数是否合理。

以下是hive-config.xml文件：

<configuration>

<property>   
  <name>javax.jdo.option.ConnectionURL</name>   
  <value>jdbc:hive://10.1.10.250:10000/testdb</value>   
  <description>JDBC connect string</description> 
</property>

<property>   
  <name>javax.jdo.option.ConnectionDriverName</name>   
  <value>org.apache.hadoop.hive.jdbc.HiveDriver</value>   
  <description>JDBC driver</description> 
</property>

</configuration>

我知道Hive连接正在工作，因为如果提供了错误的SQL语句、错误的URL或错误的驱动程序名称，操作将失败。

下面是操作stdout日志：

  [...truncated]

  =================================================================

  >>> Invoking Hive command line now >>>

  4283 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - <PERFLOG method=Driver.run from=org.apache.hadoop.hive.ql.Driver>
  4284 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - <PERFLOG method=TimeToSubmit from=org.apache.hadoop.hive.ql.Driver>
  4284 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - <PERFLOG method=compile from=org.apache.hadoop.hive.ql.Driver>
  4339 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - <PERFLOG method=parse from=org.apache.hadoop.hive.ql.Driver>
  4354 [main] INFO  hive.ql.parse.ParseDriver  - Parsing command: create table testdb.temp99 (col1 int)
  4665 [main] INFO  hive.ql.parse.ParseDriver  - Parse Completed
  4667 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - </PERFLOG method=parse start=1418968298270 end=1418968298598 duration=328 from=org.apache.hadoop.hive.ql.Driver>
  4667 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - <PERFLOG method=semanticAnalyze from=org.apache.hadoop.hive.ql.Driver>
  4733 [main] INFO  org.apache.hadoop.hive.ql.parse.SemanticAnalyzer  - Starting Semantic Analysis
  4735 [main] INFO  org.apache.hadoop.hive.ql.parse.SemanticAnalyzer  - Creating table testdb.temp99 position=13
  4760 [main] INFO  org.apache.hadoop.hive.ql.Driver  - Semantic Analysis Completed
  4775 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - </PERFLOG method=semanticAnalyze start=1418968298598 end=1418968298706 duration=108 from=org.apache.hadoop.hive.ql.Driver>
  4784 [main] INFO  org.apache.hadoop.hive.ql.Driver  - Returning Hive schema: Schema(fieldSchemas:null, properties:null)
  4784 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - </PERFLOG method=compile start=1418968298215 end=1418968298715 duration=500 from=org.apache.hadoop.hive.ql.Driver>
  4785 [main] INFO  org.apache.hadoop.hive.ql.Driver  - Concurrency mode is disabled, not creating a lock manager
  4785 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - <PERFLOG method=Driver.execute from=org.apache.hadoop.hive.ql.Driver>
  4785 [main] INFO  org.apache.hadoop.hive.ql.Driver  - Starting command: create table testdb.temp99 (col1 int)
  4792 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - </PERFLOG method=TimeToSubmit start=1418968298215 end=1418968298723 duration=508 from=org.apache.hadoop.hive.ql.Driver>
  4792 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - <PERFLOG method=runTasks from=org.apache.hadoop.hive.ql.Driver>
  4792 [main] INFO  org.apache.hadoop.hive.ql.log.PerfLogger  - <PERFLOG method=task.DDL.Stage-0 from=org.apache.hadoop.hive.ql.Driver>
  4815 [main] INFO  hive.ql.exec.DDLTask  - Default to LazySimpleSerDe for table testdb.temp99
  4935 [main] INFO  org.apache.hadoop.hive.metastore.HiveMetaStore  - 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
  4959 [main] INFO  org.apache.hadoop.hive.metastore.ObjectStore  - ObjectStore, initialize called
  5261 [main] INFO  DataNucleus.Persistence  - Property datanucleus.cache.level2 unknown - will be ignored
  Heart beat
  Heart beat
  [...forever...]

为什么工作流心脏在日志中的这个点跳动而不是继续？

附件：

与此配置单元操作关联的Oozie工作流是：

<workflow-app name="Hive-copy" xmlns="uri:oozie:workflow:0.4">
    <start to="Hive"/>
    <action name="Hive">
        <hive xmlns="uri:oozie:hive-action:0.2">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
              <job-xml>/user/test/hive-config.xml</job-xml>
            <script>/user/test/test.sql</script>
            <file>hive-config.xml#hive-config.xml</file>
        </hive>
        <ok to="end"/>
        <error to="kill"/>
    </action>
    <kill name="kill">
        <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name="end"/>
</workflow-app>

Answer 1

此问题与nodeManager有关。尝试在mapred-site.xml文件中设置以下属性可修复此问题：

<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<value>50 </value> 
</property>
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<value>50 </value> 
</property>

和这期差不多：https://groups.google.com/a/cloudera.org/forum/?fromgroups=#！主题/cdh-user/v0BHtQ0hlBg