编译spark-bench
彭鸿哲
- 执行
<SPARK_BENCH_HOME>/bin/build-all.sh - 编译的结果：
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ SQLApp ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /usr/lib/hadoop/spark-bench/SQL/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ SQLApp ---
[INFO] No sources to compile
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile) @ SQLApp ---
[WARNING] Expected all dependencies to require Scala version: 2.11.8
[WARNING] com.twitter:chill_2.11:0.8.0 requires scala version: 2.11.7
[WARNING] Multiple versions of scala libraries detected!
[INFO] /usr/lib/hadoop/spark-bench/SQL/src/main/scala:-1: info: compiling
[INFO] Compiling 2 source files to /usr/lib/hadoop/spark-bench/SQL/target/classes at 1555683622896
[WARNING] warning: there were 6 deprecation warnings; re-run with -deprecation for details
[WARNING] one warning found
[INFO] prepare-compile in 0 s
[INFO] compile in 30 s
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ SQLApp ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /usr/lib/hadoop/spark-bench/SQL/src/test/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.3:testCompile (default-testCompile) @ SQLApp ---
[INFO] No sources to compile
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:testCompile (scala-test-compile) @ SQLApp ---
[WARNING] Expected all dependencies to require Scala version: 2.11.8
[WARNING] com.twitter:chill_2.11:0.8.0 requires scala version: 2.11.7
[WARNING] Multiple versions of scala libraries detected!
[INFO] No sources to compile
[INFO]
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ SQLApp ---
[INFO] No tests to run.
[INFO]
[INFO] --- maven-jar-plugin:2.6:jar (default-jar) @ SQLApp ---
[INFO] Building jar: /usr/lib/hadoop/spark-bench/SQL/target/SQLApp-1.0.jar
[INFO]
[INFO] ------------------------< com.ibm:StreamingApp >------------------------
[INFO] Building StreamingApp 1.0 [20/20]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ StreamingApp ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /usr/lib/hadoop/spark-bench/Streaming/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ StreamingApp ---
[INFO] No sources to compile
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile) @ StreamingApp ---
[WARNING] Expected all dependencies to require Scala version: 2.11.8
[WARNING] com.twitter:chill_2.11:0.8.0 requires scala version: 2.11.7
[WARNING] Multiple versions of scala libraries detected!
[INFO] /usr/lib/hadoop/spark-bench/Streaming/src/main/scala:-1: info: compiling
[INFO] Compiling 3 source files to /usr/lib/hadoop/spark-bench/Streaming/target/classes at 1555683654197
[INFO] prepare-compile in 0 s
[INFO] compile in 28 s
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ StreamingApp ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /usr/lib/hadoop/spark-bench/Streaming/src/test/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.3:testCompile (default-testCompile) @ StreamingApp ---
[INFO] No sources to compile
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:testCompile (scala-test-compile) @ StreamingApp ---
[WARNING] Expected all dependencies to require Scala version: 2.11.8
[WARNING] com.twitter:chill_2.11:0.8.0 requires scala version: 2.11.7
[WARNING] Multiple versions of scala libraries detected!
[INFO] No sources to compile
[INFO]
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ StreamingApp ---
[INFO] No tests to run.
[INFO]
[INFO] --- maven-jar-plugin:2.6:jar (default-jar) @ StreamingApp ---
[INFO] Building jar: /usr/lib/hadoop/spark-bench/Streaming/target/StreamingApp-1.0.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] SparkBench 2.0 ..................................... SUCCESS [04:13 min]
[INFO] Common 1.0 ......................................... SUCCESS [08:53 min]
[INFO] TerasortApp 1.0 .................................... SUCCESS [09:05 min]
[INFO] PCAApp 1.0 ......................................... SUCCESS [ 52.619 s]
[INFO] LogisticRegressionApp 1.0 .......................... SUCCESS [ 12.751 s]
[INFO] KMeansApp 1.0 ...................................... SUCCESS [ 25.529 s]
[INFO] LinearRegressionApp 1.0 ............................ SUCCESS [ 11.970 s]
[INFO] SVMApp 1.0 ......................................... SUCCESS [ 53.626 s]
[INFO] DecisionTreeApp 1.0 ................................ SUCCESS [ 25.526 s]
[INFO] MFApp 1.0 .......................................... SUCCESS [01:16 min]
[INFO] PageRankApp 1.0 .................................... SUCCESS [ 54.788 s]
[INFO] TriangleCountApp 1.0 ............................... SUCCESS [ 18.763 s]
[INFO] StronglyConnectedComponentApp 1.0 .................. SUCCESS [ 18.674 s]
[INFO] ShortestPathsApp 1.0 ............................... SUCCESS [ 20.431 s]
[INFO] SVDPlusPlusApp 1.0 ................................. SUCCESS [ 24.232 s]
[INFO] LabelPropagationApp 1.0 ............................ SUCCESS [ 24.352 s]
[INFO] ConnectedComponentApp 1.0 .......................... SUCCESS [ 41.188 s]
[INFO] PregelOperationApp 1.0 ............................. SUCCESS [ 22.888 s]
[INFO] SQLApp 1.0 ......................................... SUCCESS [02:15 min]
[INFO] StreamingApp 1.0 ................................... SUCCESS [ 29.864 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 33:03 min
[INFO] Finished at: 2019-04-19T14:21:23Z
[INFO] ------------------------------------------------------------------------
Build all done!
- 执行 spark程序时候，报错：
19/04/20 00:56:16 INFO util.log: Logging initialized @10732ms
19/04/20 00:56:16 INFO server.Server: jetty-9.2.z-SNAPSHOT
19/04/20 00:56:16 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1905bff{/jobs,null,AVAILABLE}
19/04/20 00:56:16 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@2c13a8{/jobs/json,null,AVAILABLE}
19/04/20 00:56:16 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@16b1987{/jobs/job,null,AVAILABLE}
19/04/20 00:56:16 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@144435b{/jobs/job/json,null,AVAILABLE}
19/04/20 00:56:16 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@cca26f{/stages,null,AVAILABLE}
19/04/20 00:56:16 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@b7550d{/stages/json,null,AVAILABLE}
19/04/20 00:56:16 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@143df7d{/stages/stage,null,AVAILABLE}
19/04/20 00:56:16 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@541c15{/stages/stage/json,null,AVAILABLE}
19/04/20 00:56:16 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@142162a{/stages/pool,null,AVAILABLE}
19/04/20 00:56:16 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@18122b2{/stages/pool/json,null,AVAILABLE}
19/04/20 00:56:16 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@12a0c8{/storage,null,AVAILABLE}
19/04/20 00:56:16 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@7aa82b{/storage/json,null,AVAILABLE}
19/04/20 00:56:16 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@12c0e88{/storage/rdd,null,AVAILABLE}
19/04/20 00:56:16 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1d82fed{/storage/rdd/json,null,AVAILABLE}
19/04/20 00:56:16 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1bd0be{/environment,null,AVAILABLE}
19/04/20 00:56:16 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@16b0ae6{/environment/json,null,AVAILABLE}
19/04/20 00:56:16 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6804cd{/executors,null,AVAILABLE}
19/04/20 00:56:16 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@55f7ea{/executors/json,null,AVAILABLE}
19/04/20 00:56:16 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@9949be{/executors/threadDump,null,AVAILABLE}
19/04/20 00:56:16 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@13a45f1{/executors/threadDump/json,null,AVAILABLE}
19/04/20 00:56:16 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1a87e37{/static,null,AVAILABLE}
19/04/20 00:56:16 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@eb318f{/,null,AVAILABLE}
19/04/20 00:56:16 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@a14b55{/api,null,AVAILABLE}
19/04/20 00:56:17 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1ad977d{/stages/stage/kill,null,AVAILABLE}
19/04/20 00:56:17 INFO server.ServerConnector: Started ServerConnector@98fc68{HTTP/1.1}{0.0.0.0:4040}
19/04/20 00:56:17 INFO server.Server: Started @11586ms
19/04/20 00:56:21 INFO client.RMProxy: Connecting to ResourceManager at slaver40/172.20.110.130:8032
19/04/20 00:56:23 INFO ipc.Client: Retrying connect to server: slaver40/172.20.110.130:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedS
leep(maxRetries=10, sleepTime=1000 MILLISECONDS)19/04/20 00:56:24 INFO ipc.Client: Retrying connect to server: slaver40/172.20.110.130:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedS
leep(maxRetries=10, sleepTime=1000 MILLISECONDS)19/04/20 00:56:25 INFO ipc.Client: Retrying connect to server: slaver40/172.20.110.130:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedS
leep(maxRetries=10, sleepTime=1000 MILLISECONDS)19/04/20 00:56:26 INFO ipc.Client: Retrying connect to server: slaver40/172.20.110.130:8032. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedS
leep(maxRetries=10, sleepTime=1000 MILLISECONDS)19/04/20 00:56:27 INFO ipc.Client: Retrying connect to server: slaver40/172.20.110.130:8032. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedS
leep(maxRetries=10, sleepTime=1000 MILLISECONDS)19/04/20 00:56:28 INFO ipc.Client: Retrying connect to server: slaver40/172.20.110.130:8032. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedS
leep(maxRetries=10, sleepTime=1000 MILLISECONDS)19/04/20 00:56:29 INFO ipc.Client: Retrying connect to server: slaver40/172.20.110.130:8032. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedS
leep(maxRetries=10, sleepTime=1000 MILLISECONDS)19/04/20 00:56:30 INFO ipc.Client: Retrying connect to server: slaver40/172.20.110.130:8032. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedS
leep(maxRetries=10, sleepTime=1000 MILLISECONDS)19/04/20 00:56:31 INFO ipc.Client: Retrying connect to server: slaver40/172.20.110.130:8032. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedS
leep(maxRetries=10, sleepTime=1000 MILLISECONDS)19/04/20 00:56:32 INFO ipc.Client: Retrying connect to server: slaver40/172.20.110.130:8032. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedS
leep(maxRetries=10, sleepTime=1000 MILLISECONDS)
找到解决方法。
没有在Hadoop NameNode上开启 yarn。
第一步，在Hadoop NameNode 开启 Hadoop HDFS，$HADOOP_HOME/sbin/start-dfs.sh
第二部，在Hadoop NameNode 开启yarn，$HADOOP_HOME/sbin/start-yarn.sh
第三步，在spark Master 开启spark，$SPARK_HOME/sbin/start-all.sh
参考 hadoop-connecting-to-resourcemanager-failed
参考 https://github.com/CODAIT/spark-bench/tree/legacy
类似资料：