在同一JVM中同时运行多个spark实例的最佳实践？

Question

   16/12/06 08:18:40 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and
     have sufficient resources
    16/12/06 08:18:55 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and
     have sufficient resources
    16/12/06 08:18:59 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (172.26.7.195:44690) with ID 52
    16/12/06 08:18:59 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, 172.26.7.195, partition 0, ANY, 7307 bytes)
    16/12/06 08:18:59 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 0 on executor id: 52 hostname: 172.26.7.195.
    16/12/06 08:19:00 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (172.26.7.192:38343) with ID 53
    16/12/06 08:19:02 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, 172.26.7.195): java.io.IOException: org.apache.spark.SparkException: Failed
     to get broadcast_1_piece0 of broadcast_1
            at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1260)
            at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:174)
            at org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:65)
            at org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:65)
            at org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:89)
            at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
            at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:67)
            at org.apache.spark.scheduler.Task.run(Task.scala:85)
            at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:745)

我的查询如下：

>

在我的应用程序中，我需要在计划中同时运行多个pyspark应用程序。是否有任何方法可以同时从spark驱动程序运行多个pyspark应用程序，从而创建单独的sparkcontext对象？

如果第一个查询的答案是否定的，那么我可以运行一个应用程序从驱动程序，另一个从执行程序，但我可以一次运行它。

null

配置:VM-1:Hadoop主节点、Spark驱动程序和执行程序、Mongodb VM-2:Hadoop数据节点、Spark执行程序Pyspark应用程序在VM-1中正常的crontab条目中运行

Answer 1

我也试图做类似的事情，但得到了块管理器注册错误。我试图从同一个节点启动两个不同的pyspark shell，经过多次搜索后，我意识到可能两个pyspark shell都使用相同的驱动程序JVM，当一个shell占用另一个BlockManager时，另一个shell开始提供异常。

因此，我决定使用另一种方法，即使用不同的节点来启动驱动程序，并使用

pyspark --master <spark-master url> --total-executor-cores <num of cores to use>  

现在我不再得到错误。