Error while deploying YARN cluster: Couldn't deploy Yarn cluster
java.lang.RuntimeException: Couldn't deploy Yarn cluster
at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploy(AbstractYarnClusterDescriptor.java:443)
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:630)
at org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(FlinkYarnSessionCli.java:486)
at org.apache.flink.yarn.cli.FlinkYarnSessionCli$1.call(FlinkYarnSessionCli.java:483)
at org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40)
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:483)
Caused by: org.apache.flink.yarn.AbstractYarnClusterDescriptor$YarnDeploymentException: The YARN application unexpectedly switched to state FAILED during deployment.
Diagnostics from YARN: Application application_1553768779226_0005 failed 1 times due to AM Container for appattempt_1553768779226_0005_000001 exited with exitCode: -103
For more detailed output, check application tracking page:http://uplooking01:8088/proxy/application_1553768779226_0005/Then, click on links to logs of each attempt.
Diagnostics: Container [pid=2322,containerID=container_1553768779226_0005_01_000001] is running beyond virtual memory limits. Current usage: 75.1 MB of 1 GB physical memory used; 2.1 GB of 2.1 GB virtual memor
y used. Killing container.Dump of the process-tree for container_1553768779226_0005_01_000001 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 2322 2321 2322 2322 (bash) 0 0 108650496 300 /bin/bash -c /opt/jdk/bin/java -Xmx424m -Dlog.file=/opt/hadoop/logs/userlogs/application_1553768779226_0005/container_1553768779226_0005_01_000001/jobma
nager.log -Dlogback.configurationFile=file:logback.xml -Dlog4j.configuration=file:log4j.properties org.apache.flink.yarn.YarnApplicationMasterRunner 1> /opt/hadoop/logs/userlogs/application_1553768779226_0005/container_1553768779226_0005_01_000001/jobmanager.out 2> /opt/hadoop/logs/userlogs/application_1553768779226_0005/container_1553768779226_0005_01_000001/jobmanager.err |- 2330 2322 2322 2322 (java) 267 71 2165399552 18925 /opt/jdk/bin/java -Xmx424m -Dlog.file=/opt/hadoop/logs/userlogs/application_1553768779226_0005/container_1553768779226_0005_01_000001/jobmanager.lo
g -Dlogback.configurationFile=file:logback.xml -Dlog4j.configuration=file:log4j.properties org.apache.flink.yarn.YarnApplicationMasterRunner
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
Failing this attempt. Failing the application.
If log aggregation is enabled on your cluster, use this command to further investigate the issue:
yarn logs -applicationId application_1553768779226_0005
at org.apache.flink.yarn.AbstractYarnClusterDescriptor.startAppMaster(AbstractYarnClusterDescriptor.java:902)
at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:582)
at org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploy(AbstractYarnClusterDescriptor.java:441)
... 9 more
2019-03-28 06:46:32,070 INFO org.apache.flink.yarn.YarnClusterDescriptor - Cancelling deployment from Deployment Failure Hook
2019-03-28 06:46:32,070 INFO org.apache.flink.yarn.YarnClusterDescriptor - Killing YARN application
2019-03-28 06:46:32,082 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Killed application application_1553768779226_0005
出现此类错误,主要的原因是Current usage: 75.1 MB of 1 GB physical memory used; 2.1 GB of 2.1 GB virtual memor
y used. Killing container.
字面原因是容器内存不够,实际上是flink on yarn启动时检查虚拟内存造成的
所以修改配置文件,让它不检查就没事了
修改etc/hadoop/yarn-site.xml
<property>
<name>yarn.nodemanager.pmem-check-enabled</name>
<value>false</value>
</property>
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property>