启动spark-shell或执行spark-submit失败的问题

启动spark-shell或执行spark-submit失败的问题

楼主搭建了一套CDH大数据平台，其中包括Spark2服务，近来在终端启动spark-shell以及利用spark-submit提交任务时，分别报以下错误：

[root@cdh0 ~]# spark-shell
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
        at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:123)
        at org.apache.spark.deploy.SparkSubmitArguments$$anonfun$mergeDefaultSparkProperties$1.apply(SparkSubmitArguments.scala:123)
        at scala.Option.getOrElse(Option.scala:120)
        at org.apache.spark.deploy.SparkSubmitArguments.mergeDefaultSparkProperties(SparkSubmitArguments.scala:123)
        at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:109)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:114)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        ... 7 more

---

[root@cdh0 wordcount]# spark-submit --class "SimpleApp" /home/spark/mycode/wordcount/target/scala-2.11/simple-project_2.11-1.0.jar  
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
        at org.apache.spark.deploy.SparkSubmitArguments.handleUnknown(SparkSubmitArguments.scala:457)
        at org.apache.spark.launcher.SparkSubmitOptionParser.parse(SparkSubmitOptionParser.java:178)
        at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:103)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:114)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
        ... 5 more

如上面所示，两个指令一开始的错误提示是一致的，说明这两个错误的原因也是类似的。因为我是采用Cloudera Manager进行服务管理的，所以不存在其他朋友说的要对配置文件进行更改的情况，我的理解是配置系统早已构建好了。
那到底是因为什么呢？其实原因很简单，文章一开始也说了，我的环境是Spark2而不是Spark，所以在执行上述两条指令时要在“spark”后面加上个“2”，即：

[root@cdh0 ~]# spark2-shell

[root@cdh0 wordcount]# spark2-submit --class "SimpleApp" /home/spark/mycode/wordcount/target/scala-2.11/simple-project_2.11-1.0.jar 

这两个指令的存储位置如下：

[root@cdh0 bin]# pwd
/opt/cloudera/parcels/SPARK2-2.4.0.cloudera2-1.cdh5.13.3.p0.1041012/bin
[root@cdh0 bin]# ll
total 12
-rwxr-xr-x 1 root root 692 Apr 24  2019 pyspark2
-rwxr-xr-x 1 root root 653 Apr 24  2019 spark2-shell
-rwxr-xr-x 1 root root 654 Apr 24  2019 spark2-submit

由上面可以看到，确实是spark2-shell和spark2-submit。