纱线作业获得的资源似乎比Ambari纱线经理报告的要少

Question

➜  h2o-3.26.0.2-hdp3.1 hadoop jar h2odriver.jar -nodes 4 -mapperXmx 5g -output /home/ml1/hdfsOutputDir
Determining driver host interface for mapper->driver callback...
    [Possible callback IP address: 192.168.122.1]
    [Possible callback IP address: 172.18.4.49]
    [Possible callback IP address: 127.0.0.1]
Using mapper->driver callback IP address and port: 172.18.4.49:46721
(You can override these with -driverif and -driverport/-driverportrange and/or specify external IP using -extdriverif.)
Memory Settings:
    mapreduce.map.java.opts:     -Xms5g -Xmx5g -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Dlog4j.defaultInitOverride=true
    Extra memory percent:        10
    mapreduce.map.memory.mb:     5632
Hive driver not present, not generating token.
19/08/07 12:37:19 INFO client.RMProxy: Connecting to ResourceManager at hw01.ucera.local/172.18.4.46:8050
19/08/07 12:37:19 INFO client.AHSProxy: Connecting to Application History server at hw02.ucera.local/172.18.4.47:10200
19/08/07 12:37:19 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /user/ml1/.staging/job_1565057088651_0007
19/08/07 12:37:21 INFO mapreduce.JobSubmitter: number of splits:4
19/08/07 12:37:21 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1565057088651_0007
19/08/07 12:37:21 INFO mapreduce.JobSubmitter: Executing with tokens: []
19/08/07 12:37:21 INFO conf.Configuration: found resource resource-types.xml at file:/etc/hadoop/3.1.0.0-78/0/resource-types.xml
19/08/07 12:37:21 INFO impl.YarnClientImpl: Submitted application application_1565057088651_0007
19/08/07 12:37:21 INFO mapreduce.Job: The url to track the job: http://HW01.ucera.local:8088/proxy/application_1565057088651_0007/
Job name 'H2O_80092' submitted
JobTracker job ID is 'job_1565057088651_0007'
For YARN users, logs command is 'yarn logs -applicationId application_1565057088651_0007'
Waiting for H2O cluster to come up...
19/08/07 12:37:38 INFO client.RMProxy: Connecting to ResourceManager at hw01.ucera.local/172.18.4.46:8050
19/08/07 12:37:38 INFO client.AHSProxy: Connecting to Application History server at hw02.ucera.local/172.18.4.47:10200

----- YARN cluster metrics -----
Number of YARN worker nodes: 4

----- Nodes -----
Node: http://HW03.ucera.local:8042 Rack: /default-rack, RUNNING, 1 containers used, 5.0 / 15.0 GB used, 1 / 3 vcores used
Node: http://HW04.ucera.local:8042 Rack: /default-rack, RUNNING, 0 containers used, 0.0 / 15.0 GB used, 0 / 3 vcores used
Node: http://hw05.ucera.local:8042 Rack: /default-rack, RUNNING, 0 containers used, 0.0 / 15.0 GB used, 0 / 3 vcores used
Node: http://HW02.ucera.local:8042 Rack: /default-rack, RUNNING, 0 containers used, 0.0 / 15.0 GB used, 0 / 3 vcores used

----- Queues -----
Queue name:            default
    Queue state:       RUNNING
    Current capacity:  0.08
    Capacity:          1.00
    Maximum capacity:  1.00
    Application count: 1
    ----- Applications in this queue -----
    Application ID:                  application_1565057088651_0007 (H2O_80092)
        Started:                     ml1 (Wed Aug 07 12:37:21 HST 2019)
        Application state:           FINISHED
        Tracking URL:                http://HW01.ucera.local:8088/proxy/application_1565057088651_0007/
        Queue name:                  default
        Used/Reserved containers:    1 / 0
        Needed/Used/Reserved memory: 5.0 GB / 5.0 GB / 0.0 GB
        Needed/Used/Reserved vcores: 1 / 1 / 0

Queue 'default' approximate utilization: 5.0 / 60.0 GB used, 1 / 12 vcores used

----------------------------------------------------------------------

ERROR: Unable to start any H2O nodes; please contact your YARN administrator.

       A common cause for this is the requested container size (5.5 GB)
       exceeds the following YARN settings:

           yarn.nodemanager.resource.memory-mb
           yarn.scheduler.maximum-allocation-mb

----------------------------------------------------------------------

For YARN users, logs command is 'yarn logs -applicationId application_1565057088651_0007'

注意

错误：无法启动任何H2O节点；请与您的纱线管理员联系。

造成这种情况的一个常见原因是所要求的容器大小(5.5GB)超过了以下纱线设置：

  yarn.nodemanager.resource.memory-mb
  yarn.scheduler.maximum-allocation-mb

yarn.scheduler.maximum-allocation-vcores=3
yarn.nodemanager.resource.cpu-vcores=3
yarn.nodemanager.resource.memory-mb=15GB
yarn.scheduler.maximum-allocation-mb=15GB

[myuser@HW03 ~]$ yarn jar /usr/hdp/3.1.0.0-78/hadoop-mapreduce/hadoop-mapreduce-examples.jar pi 1000 1000
Number of Maps  = 1000
Samples per Map = 1000
....

----节点----

节点:http://hw03.ucera.local:8042 rack://default-rack，正在运行，使用了1个容器，使用了5.0/15.0GB，使用了1/3个vcore

节点:http://hw04.ucera.local:8042 rack://default-rack，正在运行，使用了0个容器，0.0/15.0GB，0/3个VCore

从这两件事来看，似乎既没有超过15GB节点限制，也没有超过60GB集群限制，那么为什么会抛出这些错误呢？那么这个情况呢，我在这里误解了吗？可以做什么来修复（再次，希望能够使用所有明显的60GB纱线资源的工作没有错误）？有任何修复的调试建议吗？

更新：

问题似乎与如何正确更改HDP/AMBARI创建的用户的uid有关？而且，如果节点上存在用户，并且具有具有正确权限的hdfs://user/ 目录（正如我从Hortonworks论坛的一篇文章中所认为的那样），这一事实不足以被确认为集群上的“存在”。

[root@HW01 ~]# clush -ab id hdfs
---------------
HW[01-04] (4)
---------------
uid=1017(hdfs) gid=1005(hadoop) groups=1005(hadoop),1003(hdfs)
---------------
HW05
---------------
uid=1021(hdfs) gid=1006(hadoop) groups=1006(hadoop),1004(hdfs)
[root@HW01 ~]# 
[root@HW01 ~]#
# wondering what else is using a uid 1021 across the nodes 
[root@HW01 ~]# clush -ab id 1021
---------------
HW[01-04] (4)
---------------
uid=1021(hbase) gid=1005(hadoop) groups=1005(hadoop)
---------------
HW05
---------------
uid=1021(hdfs) gid=1006(hadoop) groups=1006(hadoop),1004(hdfs)

在发布答案之前，将进一步研究这一点。我认为基本上需要更多地澄清HDP何时认为用户“存在”在集群中。

Answer 1

问题似乎与如何正确更改HDP/AMBARI创建的用户的uid有关？而且，在节点上存在一个用户，并且拥有一个具有正确权限的hdfs：//user/目录（正如我从Hortonworks论坛帖子中得到的结论），这一事实不足以被确认为集群上的“存在”。这与我与Hortonworks专家的讨论相一致，他们说使用纱线的用户必须存在于集群的所有数据阳极上。

为存在于所有集群节点上的不同用户（在本例中是Ambari创建的hdfs用户）运行hadoop jar命令（即使Ambari创建的该用户跨节点具有不同的UID（如果这是一个问题，IDK))，并且具有hdfs：//user/hdfs dir，发现h2o jar按预期运行。

我最初的印象是，用户只需要存在于正在使用的任何客户端机器上，并需要hdfs://community.cloudera.com/t5/support-questions/adding-a-new-user-to-the-cluster/m-p/130319/highlight/true#m93005)。

附注：

由此产生的一个令人担忧/令人困惑的事情是，事实上Ambari显然在不同的集群节点上创建了具有不同uid和gid值的hdfs用户，例如...

[root@HW01 ~]# clush -ab id hdfs
---------------
HW[01-04] (4)
---------------
uid=1017(hdfs) gid=1005(hadoop) groups=1005(hadoop),1003(hdfs)
---------------
HW05
---------------
uid=1021(hdfs) gid=1006(hadoop) groups=1006(hadoop),1004(hdfs)
[root@HW01 ~]# 
[root@HW01 ~]#
# wondering what else is using a uid 1021 across the nodes 
[root@HW01 ~]# clush -ab id 1021
---------------
HW[01-04] (4)
---------------
uid=1021(hbase) gid=1005(hadoop) groups=1005(hadoop)
---------------
HW05
---------------
uid=1021(hdfs) gid=1006(hadoop) groups=1006(hadoop),1004(hdfs)

在这里对此进行更多的研究：HDFS NFS位置使用奇怪的数字用户名值来表示目录权限