PySparkjava.lang.异常引起的错误：java.lang.StringIndexOutOfBoundsExctive

Question

我按照这些说明，并在我的机器上安装了Apache Spark（PySpark）2.3.1，其规格如下：

• Ubuntu 18.04

当我通过从shell调用pyspark或通过直接在我的应用程序中创建会话来间接创建Spark会话时：

spark = pyspark.sql.SparkSession.builder.appName('test').getOrCreate()

我得到以下例外：

Exception in thread "main" java.lang.ExceptionInInitializerError
    at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
    at org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
    at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273)
....
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:137)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.StringIndexOutOfBoundsException: begin 0, end 3, length 2
    at java.base/java.lang.String.checkBoundsBeginEnd(String.java:3107)
    at java.base/java.lang.String.substring(String.java:1873)
    at org.apache.hadoop.util.Shell.<clinit>(Shell.java:52)
    ... 22 more
Traceback (most recent call last):
  File "/home/welshamy/tools/anaconda3/lib/python3.6/site-packages/pyspark/python/pyspark/shell.py", line 38, in <module>
    SparkContext._ensure_initialized()
  File "/home/welshamy/tools/anaconda3/lib/python3.6/site-packages/pyspark/context.py", line 292, in _ensure_initialized
    SparkContext._gateway = gateway or launch_gateway(conf)
  File "/home/welshamy/tools/anaconda3/lib/python3.6/site-packages/pyspark/java_gateway.py", line 93, in launch_gateway
    raise Exception("Java gateway process exited before sending its port number")
Exception: Java gateway process exited before sending its port number

如果我使用Jupyter笔记本，我也会在笔记本中得到这个异常：

Exception: Java gateway process exited before sending the driver its port number

我找到并遵循的所有解决方案[1,2,3]都指向环境变量定义，但没有一个对我有用。

Answer 1

PySpark 2.3.1不支持JDK 10。您需要安装JDK 8并将JAVA_HOME环境变量设置为指向它。

如果你正在使用Ubuntu（或*nix）：

>

安装JDK 8

sudo apt-get install openjdk-8-jdk

将以下行添加到~/。bashrc文件：

export JAVA_HOME="/usr/lib/jvm/java-8-openjdk-amd64"

在Windows下，安装JDK 8并设置JAVA_HOME。