登录
当前位置： 首页 > 面试题库 >

使用count（）和first（）时，iPython Notebook中的PySpark引发Py4JJavaError

郭均
2023-03-14
问题内容：

我在Mac（Sierra 10.12.3 Beta）中的virtualenv上的iPython笔记本（python
v.3.6）中使用PySpark（v.2.1.0）。

1.我通过在Terminal上进行拍摄来启动了iPython笔记本-

 PYSPARK_PYTHON=python3 PYSPARK_DRIVER_PYTHON=ipython PYSPARK_DRIVER_PYTHON_OPTS="notebook" /Applications/spark-2.1.0-bin-hadoop2.7/bin/pyspark

2.将我的文件加载到Spark Context并确保其加载-

>>>lines = sc.textFile("/Users/PanchusMac/Dropbox/Learn_py/Virtual_Env/pyspark/README.md")

>>>for i in lines.collect(): 
    print(i)

效果很好，并在控制台上打印了结果，如下所示：

# Apache Spark

Spark is a fast and general cluster computing system for Big Data. It provides
high-level APIs in Scala, Java, Python, and R, and an optimized engine that
supports general computation graphs for data analysis. It also supports a
rich set of higher-level tools including Spark SQL for SQL and DataFrames,
MLlib for machine learning, GraphX for graph processing,
and Spark Streaming for stream processing.

<http://spark.apache.org/>


## Online Documentation

You can find the latest Spark documentation, including a programming
guide, on the [project web page](http://spark.apache.org/documentation.html).
This README file only contains basic setup instructions.

还检查了sc-

>>>print(sc)

<pyspark.context.SparkContext object at 0x101ce4cc0>
  1. 现在，当我尝试通过RDD执行 lines.count()或lines.first()运行时，出现以下错误-
    Py4JJavaError                             Traceback (most recent call last)
<ipython-input-33-44aeefde846d> in <module>()
----> 1 lines.count()

/Applications/spark-2.1.0-bin-hadoop2.7/python/pyspark/rdd.py in count(self)
   1039         3
   1040         """
-> 1041         return self.mapPartitions(lambda i: [sum(1 for _ in i)]).sum()
   1042 
   1043     def stats(self):

/Applications/spark-2.1.0-bin-hadoop2.7/python/pyspark/rdd.py in sum(self)
   1030         6.0
   1031         """
-> 1032         return self.mapPartitions(lambda x: [sum(x)]).fold(0, operator.add)
   1033 
   1034     def count(self):

/Applications/spark-2.1.0-bin-hadoop2.7/python/pyspark/rdd.py in fold(self, zeroValue, op)
    904         # zeroValue provided to each partition is unique from the one provided
    905         # to the final reduce call
--> 906         vals = self.mapPartitions(func).collect()
    907         return reduce(op, vals, zeroValue)
    908

/Applications/spark-2.1.0-bin-hadoop2.7/python/pyspark/rdd.py in collect(self)
    807         """
    808         with SCCallSiteSync(self.context) as css:
--> 809             port = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd())
    810         return list(_load_from_socket(port, self._jrdd_deserializer))
    811

/Applications/spark-2.1.0-bin-hadoop2.7/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py in __call__(self, *args)
   1131         answer = self.gateway_client.send_command(command)
   1132         return_value = get_return_value(
-> 1133             answer, self.gateway_client, self.target_id, self.name)
   1134 
   1135         for temp_arg in temp_args:

/Applications/spark-2.1.0-bin-hadoop2.7/python/pyspark/sql/utils.py in deco(*a, **kw)
     61     def deco(*a, **kw):
     62         try:
---> 63             return f(*a, **kw)
     64         except py4j.protocol.Py4JJavaError as e:
     65             s = e.java_exception.toString()

/Applications/spark-2.1.0-bin-hadoop2.7/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
    317                 raise Py4JJavaError(
    318                     "An error occurred while calling {0}{1}{2}.\n".
--> 319                     format(target_id, ".", name), value)
    320             else:
    321                 raise Py4JError(

Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 14.0 failed 1 times, most recent failure: Lost task 1.0 in stage 14.0 (TID 22, localhost, executor driver): org.apache.spark.SparkException: 
Error from python worker:
  Traceback (most recent call last):
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 183, in _run_module_as_main
      mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 109, in _get_module_details
      __import__(pkg_name)
    File "<frozen importlib._bootstrap>", line 961, in _find_and_load
    File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 646, in _load_unlocked
    File "<frozen importlib._bootstrap>", line 616, in _load_backward_compatible
    File "/Applications/spark-2.1.0-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/__init__.py", line 44, in <module>
    File "<frozen importlib._bootstrap>", line 961, in _find_and_load
    File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 646, in _load_unlocked
    File "<frozen importlib._bootstrap>", line 616, in _load_backward_compatible
    File "/Applications/spark-2.1.0-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/context.py", line 36, in <module>
    File "<frozen importlib._bootstrap>", line 961, in _find_and_load
    File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 646, in _load_unlocked
    File "<frozen importlib._bootstrap>", line 616, in _load_backward_compatible
    File "/Applications/spark-2.1.0-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/java_gateway.py", line 25, in <module>
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/platform.py", line 886, in <module>
      "system node release version machine processor")
    File "/Applications/spark-2.1.0-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/serializers.py", line 393, in namedtuple
  TypeError: namedtuple() missing 3 required keyword-only arguments: 'verbose', 'rename', and 'module'
PYTHONPATH was:
  /Applications/spark-2.1.0-bin-hadoop2.7/python/lib/pyspark.zip:/Applications/spark-2.1.0-bin-hadoop2.7/python/lib/py4j-0.10.4-src.zip:/Applications/spark-2.1.0-bin-hadoop2.7/jars/spark-core_2.11-2.1.0.jar:/Applications/spark-2.1.0-bin-hadoop2.7/python/lib/py4j-0.10.4-src.zip:/Applications/spark-2.1.0-bin-hadoop2.7/python/:
java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:392)
    at org.apache.spark.api.python.PythonWorkerFactory.startDaemon(PythonWorkerFactory.scala:166)
    at org.apache.spark.api.python.PythonWorkerFactory.createThroughDaemon(PythonWorkerFactory.scala:89)
    at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:65)
    at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:116)
    at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:128)
    at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:63)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
    at org.apache.spark.scheduler.Task.run(Task.scala:99)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1435)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1423)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1422)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1422)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)
    at scala.Option.foreach(Option.scala:257)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:802)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1650)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1605)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1594)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:628)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1918)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1931)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1944)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1958)
    at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:935)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
    at org.apache.spark.rdd.RDD.collect(RDD.scala:934)
    at org.apache.spark.api.python.PythonRDD$.collectAndServe(PythonRDD.scala:453)
    at org.apache.spark.api.python.PythonRDD.collectAndServe(PythonRDD.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:280)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:214)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.spark.SparkException: 
Error from python worker:
  Traceback (most recent call last):
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 183, in _run_module_as_main
      mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/runpy.py", line 109, in _get_module_details
      __import__(pkg_name)
    File "<frozen importlib._bootstrap>", line 961, in _find_and_load
    File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 646, in _load_unlocked
    File "<frozen importlib._bootstrap>", line 616, in _load_backward_compatible
    File "/Applications/spark-2.1.0-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/__init__.py", line 44, in <module>
    File "<frozen importlib._bootstrap>", line 961, in _find_and_load
    File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 646, in _load_unlocked
    File "<frozen importlib._bootstrap>", line 616, in _load_backward_compatible
    File "/Applications/spark-2.1.0-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/context.py", line 36, in <module>
    File "<frozen importlib._bootstrap>", line 961, in _find_and_load
    File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
    File "<frozen importlib._bootstrap>", line 646, in _load_unlocked
    File "<frozen importlib._bootstrap>", line 616, in _load_backward_compatible
    File "/Applications/spark-2.1.0-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/java_gateway.py", line 25, in <module>
    File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/platform.py", line 886, in <module>
      "system node release version machine processor")
    File "/Applications/spark-2.1.0-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/serializers.py", line 393, in namedtuple
  TypeError: namedtuple() missing 3 required keyword-only arguments: 'verbose', 'rename', and 'module'
PYTHONPATH was:
  /Applications/spark-2.1.0-bin-hadoop2.7/python/lib/pyspark.zip:/Applications/spark-2.1.0-bin-hadoop2.7/python/lib/py4j-0.10.4-src.zip:/Applications/spark-2.1.0-bin-hadoop2.7/jars/spark-core_2.11-2.1.0.jar:/Applications/spark-2.1.0-bin-hadoop2.7/python/lib/py4j-0.10.4-src.zip:/Applications/spark-2.1.0-bin-hadoop2.7/python/:
java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:392)
    at org.apache.spark.api.python.PythonWorkerFactory.startDaemon(PythonWorkerFactory.scala:166)
    at org.apache.spark.api.python.PythonWorkerFactory.createThroughDaemon(PythonWorkerFactory.scala:89)
    at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:65)
    at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:116)
    at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:128)
    at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:63)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
    at org.apache.spark.scheduler.Task.run(Task.scala:99)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    ... 1 more

有人可以解释一下哪里出了问题吗？注意：当我在Mac Terminal中执行相同的操作时，它们按预期工作。


问题答案：

Pyspark 2.1.0与python
3.6不兼容，请参阅https://issues.apache.org/jira/browse/SPARK-19019。

您需要使用早期的python版本，或者可以尝试从github构建master或2.1分支，它应该可以工作。



 类似资料：
  • 使用concat（）和first（）实现RxJava并发调用的缓存

    我正在使用RxJava和和操作符： 返回从缓存列表构建的可观察，而方法使用Retrofit获取实体。 除非两个用户快速订阅返回的可观察对象，否则这非常有效。我猜在进行第二次订阅时，第一次订阅的网络请求没有完成。在这种情况下，执行了两个网络请求。我想避免。 我尝试创建一个单线程调度程序，以便仅在第一次调用结束时执行第二次调用，但没有运气： 以及： 我曾尝试将subscribeOn调用放在可观察链的较

  • 什么时候使用df.value_counts（）和df.groupby（'...'）。count（）？

    问题内容： 我在Pandas听说过，通常有多种方法可以做同一件事，但是我想知道- 如果我要按特定列中的值对数据进行分组并计算具有该值的项目数，那么什么时候使用有意义，什么时候使用有意义？ 问题答案： 有差额收益： 结果对象将按降序排列，以便第一个元素是最频繁出现的元素。 但不是，它对输出排序（由中的列创建）。 是用于按功能汇总所有列的，因此它计算不包括s的值。 因此，如果仅需要一列，则： 样品：

  • 我可以同时使用COUNT（）和DISTINCT吗？

    问题内容： 我想计算mysql表中的行数，而不要包含重复的条目， 我可以用吗？ 问题答案： 当然。

  • count(*),count(1)和count(列名)的区别？

    本文向大家介绍count(*),count(1)和count(列名)的区别？相关面试题，主要包含被问及count(*),count(1)和count(列名)的区别？时的应答技巧和注意事项，需要的朋友参考一下 count(*),count(1)在统计的时候不会忽略Null,count(列名)在统计的时候会忽略Null。若列名为主键,count(列名)会比count(1),count(*)快,反之则c

  • 使用jar时引发异常

  • 使用kafka和HBase引发的冲突

    当我使用“Spark Streaming”读取“Kafka”（需要sasl验证），然后将数据存储到“HBase”时，“HBase”给出以下错误 java.io.IOException：java.lang.Reflect.InvocationTargetException在org.apache.hadoop.hbase.client.ConnectionFactor.CreateConnection

相关阅读

在MySQL查询中一起使用DISTINCT和COUNTMySQL中的COUNT（id）与COUNT（*）是否存在UPDATE count = count + 1的并发？在SQL中使用COUNT函数何时在MySQL中使用单引号，双引号和反引号

相关文章

Pattern和Matcher的使用.NET Core包的引用Java的方法引用MongoDB索引使用注意事项Python count()统计字符串出现的次数

相关问答

在pyspark中使用UDF和simpe数据帧中断iPythonNotebook中的烧瓶应用程序导致ZMQerror[重复]使用pyspark.ml.linalg.vectors的pyspark saveAsSequenceFile在Google Colab中使用pyspark使用count和groupby查询Hibernate条件

相关工具

good-first-issuesawesome-first-timersMalagu Serverless FirstAsian Word Count网络发言防和谐器

相关文档

PySpark 中文教程应用引擎 V2 开发文档厘米游戏引擎接入指南和开发文档HeyThings IoT 服务平台开发指引物联网应用开发 IoT Studio 使用指南
快捷导航： 新手教程 算法原理 架构设计 Java进阶 数据库进阶 大厂专栏 面试经验 编程笔记 编程问答 所有专题 文档资料 工具软件 电子书籍 小牛导航
在线工具： 房贷计算器 个税计算器 Linux命令查询 Json格式化 正则表达式 颜色转换 AES加解密 SHA1加密 MD5加密 毒鸡汤 字数统计 随机密码生成 进制转换 Base64编解码 励志句子
Copyright © 2019-2024 小牛知识库@xnip.cn. All Rights Reserved.