安装spark 2.4.4后尝试运行pyspark时，如何修复“TypeError：一个整数是必需的（got type bytes）”错误

Question

我已经安装了OpenJDK13.0.1、Python3.8和Spark2.4.4。测试安装的说明是从spark安装的根目录运行.\bin\pyspark。我不确定我是否错过了安装spark的一个步骤，比如设置一些环境变量，但我找不到任何进一步的详细说明。

我可以在我的机器上运行python解释器，所以我确信它安装正确，并且运行“Java版本”会给我预期的响应，所以我不认为问题出在这两个方面。

我从cloudpickly.py获得错误的堆栈跟踪：

Traceback (most recent call last):
  File "C:\software\spark-2.4.4-bin-hadoop2.7\bin\..\python\pyspark\shell.py", line 31, in <module>
    from pyspark import SparkConf
  File "C:\software\spark-2.4.4-bin-hadoop2.7\python\pyspark\__init__.py", line 51, in <module>
    from pyspark.context import SparkContext
  File "C:\software\spark-2.4.4-bin-hadoop2.7\python\pyspark\context.py", line 31, in <module>
    from pyspark import accumulators
  File "C:\software\spark-2.4.4-bin-hadoop2.7\python\pyspark\accumulators.py", line 97, in <module>
    from pyspark.serializers import read_int, PickleSerializer
  File "C:\software\spark-2.4.4-bin-hadoop2.7\python\pyspark\serializers.py", line 71, in <module>
    from pyspark import cloudpickle
  File "C:\software\spark-2.4.4-bin-hadoop2.7\python\pyspark\cloudpickle.py", line 145, in <module>
    _cell_set_template_code = _make_cell_set_template_code()
  File "C:\software\spark-2.4.4-bin-hadoop2.7\python\pyspark\cloudpickle.py", line 126, in _make_cell_set_template_code
    return types.CodeType(
TypeError: an integer is required (got type bytes)

Answer 1

这是因为您使用的是Python3.8。pyspark的最新pip版本（撰写本文时为pyspark 2.4.4)不支持Python3.8。暂时降级到Python3.7，您应该会没事的。