将本地计算机/容器设置为边缘节点，以便在远程群集上提交spark应用程序

Question

我试图将容器设置为边缘节点，以便在安装了spark/Hadoop(Spark2.2.0)的dev集群上运行spark应用程序。

null

我还尝试将SPARK_LOCAL_IP设置为dev集群边缘机器，我们通常在这里提交spark-submit。

这就是我在容器上提交spark-submit的方式：

spark-submit-主纱-queue stark-total-executor-cores 4-executor-cores 2-executor-memory 10g-driver-memory 10g-name Test_Spark-class ca.ix.hubble.insights.runner.jobrunneroptimized hdfs://novus-nameservice/tmp/airflowtest/insights.jar

如果我这样做，我将得到以下错误：

java.net.BindException: Cannot assign requested address: Service 'sparkDriver' failed after 16 retries (on a random free port)! Consider explicitly setting the appropriate binding address for the service 'sparkDriver' (for example spark.driver.bindAddress for SparkDriver) to the correct binding address.
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:461)
        at sun.nio.ch.Net.bind(Net.java:453)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:222)
        at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:127)
        at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:501)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1218)
        at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:496)
        at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:481)
        at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:965)
        at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:210)
        at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:353)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:446)
        at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
        at java.lang.Thread.run(Thread.java:748)```

Answer 1

顺便说一下，这是我自己想出来的，下面是我如何将本地机器设置为边缘节点：

1. 将安装hadoop的远程服务器上的spark2,hadoop目录复制到本地的/tmp/。您还可以在本地安装hadoop和spark，然后将core-site.xml、hdfs-site.xml和yarn-site.xml从集群替换为本地默认文件。您可能还希望为Spark.
带来spark-env.sh
2. 创建HADOOP_HOME环境变量并指向本地，例如导出HADOOP_HOME=/tmp/hadoop
3. 创建了SPARK_HOME环境变量并指向本地，例如export SPARK_HOME=/tmp/spark2
4. 创建HADOOP_CONF_DIR，并指向本地，例如export HADOOP_CONF_DIR=/tmp/hadoop/conf
5. 安装java并设置JAVA_HOME环境变量
6. 将所有环境变量bin目录添加到PATH中，例如export PATH=$PATH:${JAVA_HOME}/bin:${SPARK_HOME}/bin:${HADOOP_HOME}/bin
7. 请确保所有远程服务器主机名和ip地址项都在/etc/hosts的本地主机文件中
8. 为了将spark-submit应用程序从本地使用部署模式作为集群提交到远程hadoop服务器，例如

spark-submit --master yarn  --queue default --deploy-mode cluster --total-executor-cores 4 --executor-cores 2 --executor-memory 10g --driver-memory 10g --name Test_Spark --class ca.test.TestClass hdfs://hostname/tmp/test.jar```