Spark独立集群调优

Question

--driver-memory = 7GB (default - 1core is used)
--worker-memory = 43GB (all remaining cores - 7 cores)

17/12/14 03:29:39 WARN HeartbeatReceiver: Removing executor 2 with no recent heartbeats: 3658237 ms exceeds timeout 3600000 ms  
17/12/14 03:29:39 ERROR TaskSchedulerImpl: Lost executor 2 on 10.150.143.81: Executor heartbeat timed out after 3658237 ms  
17/12/14 03:29:39 WARN TaskSetManager: Lost task 23.0 in stage 316.0 (TID 9449, 10.150.143.81, executor 2): ExecutorLostFailure (executor 2 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 3658237 ms  
17/12/14 03:29:39 WARN TaskSetManager: Lost task 9.0 in stage 318.0 (TID 9459, 10.150.143.81, executor 2): ExecutorLostFailure (executor 2 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 3658237 ms  
17/12/14 03:29:39 WARN TaskSetManager: Lost task 8.0 in stage 318.0 (TID 9458, 10.150.143.81, executor 2): ExecutorLostFailure (executor 2 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 3658237 ms  
17/12/14 03:29:39 WARN TaskSetManager: Lost task 5.0 in stage 318.0 (TID 9455, 10.150.143.81, executor 2): ExecutorLostFailure (executor 2 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 3658237 ms  
17/12/14 03:29:39 WARN TaskSetManager: Lost task 7.0 in stage 318.0 (TID 9457, 10.150.143.81, executor 2): ExecutorLostFailure (executor 2 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 3658237 ms

应用程序不是那么占用内存，有两个连接和写数据集到目录。同样的代码在spark-shell上运行没有任何失败。

寻找群集调优或任何配置设置，这将减少执行器被杀死。

Answer 1

首先，我建议不要给任何应用程序分配总共50GB的RAM如果您的实例正好有50GB的RAM。系统应用程序的其余部分也需要一些RAM来工作，应用程序不使用的RAM被系统用来缓存文件和减少磁盘读取量。JVM本身在外部也有很小的内存开销。

如果您的spark作业使用了所有内存，那么您的实例将不可避免地交换，如果它交换了，它将开始出现错误行为。您可以通过运行命令HTOP轻松地检查内存使用情况，并查看服务器是否正在进行交换。您还应该确保swapiness减少到0，这样它就不会进行交换，除非它真的必须进行交换。

鉴于您提供的信息，我只能说这些，如果这没有帮助，您应该考虑提供更多的信息，如您的spark工作的完整准确参数。