Pyspark：Error-Java网关进程在向驱动程序发送端口号之前退出

Question

from pyspark import SparkConf
from pyspark.sql import SparkSession

if __name__ == '__main__':
    SPARK_CONFIGURATION = SparkConf().setAppName("OPL").setMaster("local[*]")
    SPARK_SESSION = SparkSession.builder\
        .config(conf=SPARK_CONFIGURATION)\
        .getOrCreate()

    print("Hello world")

这是回溯

Neon was unexpected at this time.
Traceback (most recent call last):
  File "C:\Users\IBM_ADMIN\Documents\Eclipse Neon for Liberty on Bluemix\OPL_Interface\src\Test\SparkTest.py", line 12, in <module>
    .config(conf=SPARK_CONFIGURATION)\
  File "C:\Users\IBM_ADMIN\Documents\spark-2.1.0-bin-hadoop2.7\python\pyspark\sql\session.py", line 169, in getOrCreate
    sc = SparkContext.getOrCreate(sparkConf)
  File "C:\Users\IBM_ADMIN\Documents\spark-2.1.0-bin-hadoop2.7\python\pyspark\context.py", line 307, in getOrCreate
    SparkContext(conf=conf or SparkConf())
  File "C:\Users\IBM_ADMIN\Documents\spark-2.1.0-bin-hadoop2.7\python\pyspark\context.py", line 115, in __init__
    SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
  File "C:\Users\IBM_ADMIN\Documents\spark-2.1.0-bin-hadoop2.7\python\pyspark\context.py", line 256, in _ensure_initialized
    SparkContext._gateway = gateway or launch_gateway(conf)
  File "C:\Users\IBM_ADMIN\Documents\spark-2.1.0-bin-hadoop2.7\python\pyspark\java_gateway.py", line 95, in launch_gateway
    raise Exception("Java gateway process exited before sending the driver its port number")
Exception: Java gateway process exited before sending the driver its port number

我将PyDev与Eclipse Eclipse Neon.2发行版(4.6.2)一起使用。配置如下：Libraries环境

注意：我使用的是最新的Spark版本:spark-2.1.0-bin-hadoop2.7

Answer 1

我的同事和我也都经历了同样的问题，它阻碍了我们，使我们的头发拔了一段时间。我们尝试了一系列建议（Java路径中没有空格、设置/取消设置PYSPARK_SUBMIT_ARGS env var、...)都没有用。

解决这个问题的方法是切换到Spark2.3.1。我们尝试使用2.2.1和2.3.0。

希望这能帮一些人省去一些烦恼。