Spark executor、driver、executor cores、executor memory的值

Question

我对Spark执行器、驱动程序、执行器内核、执行器内存的价值有些怀疑。

1. 如果集群上没有运行应用程序，如果您提交作业，Spark执行器、执行器核心、执行器内存的默认值是多少？
2. 如果我们想计算您要提交的作业所需的Spark执行器、执行器核心、执行器内存的值，您将如何做到这一点？

Answer 1

如果集群上没有运行的应用程序，如果您正在提交作业，那么Spark executor、executor core、executor memory的默认值是多少？

默认值存储在安装火花的集群中的spark-defaults.conf中。因此您可以验证这些值。通常默认值是。

检查默认值。请参阅本文件

如果我们要计算您要提交的作业所需的Spark executor、executor core、executor memory的值，您将如何进行计算？

取决于以下事项

>

你有什么类型的工作，即洗牌密集或只有地图操作。如果是洗牌，你可能需要更多内存。

数据大小越大，数据大小越大，内存使用率越高

群集约束。你能负担得起多少内存。

基于这些因素，您需要从一些数字开始，然后查看spark UI，您需要了解瓶颈并增加或减少内存占用。

一个警告是，保持执行器内存超过40G可能会更有效率，因为JVM GC会变得更慢。拥有太多内核也可能会减慢进程。

Answer 2

Avishek的回答涉及默认值。我将说明如何计算最佳值。举个例子，

示例：6个节点，每个节点有16个核和64Gb RAM

每个执行器都是JVM实例。因此可以在节点上执行多个执行器。

让我们从选择每个执行器的核心数开始：

Number of cores = Concurrent tasks as executor can run 

One may think if there is higher concurrency, performance will be better. However, experiments have shown that spark jobs perform well when the number of cores = 5.

If number of cores > 5, it leads to poor performance.

Note that 1 core and 1 Gb is needed for OS and Hadoop Daemons.

现在，计算执行者的数量：

As discussed earlier, there are 15 cores available for each node and we are planning for 5 cores per executors.

Thus number of executors per node = 15/5 = 3
Total number of executors = 3*6 = 18

Out of all executors, 1 executor is needed for AM management by YARN.
Thus, final executors count = 18-1 = 17 executors.

每个执行器的内存：

Executor per node = 3
RAM available per node = 63 Gb (as 1Gb is needed for OS and Hadoop Daemon)
Memory per executor = 63/3 = 21 Gb.

Some memory overhead is required by spark. Which is max(384, 7% of memory per executor).
Thus, 7% of 21 = 1.47
As 1.47Gb > 384Mb, subtract 1.47 from 21.
Hence, 21 - 1.47 ~ 19 Gb

最终编号：

Executors - 17, Cores 5, Executor Memory - 19 GB

注：

1. Sometimes one may feel to allocate lesser memory than 19 Gb. As memory decreases, the number of executors will increase and the number of cores will decrease. As discussed earlier, number of cores = 5 is best value. However, if you reduce it will still give good results. Just dont exceed value beyond 5.

2. Memory per executor should be less than 40 else there will be a considerable GC overhead.