spark 是否优化了广播变量的网络流量？

Question

知道Spark每个工作节点使用多个执行器，并且每个执行器都在自己的JVM中运行，我想知道Spark /if如何优化广播变量的流量。希望它为每个工作节点进行一次下载，然后将已经序列化的数据发送到该特定节点上的执行器。另一种选择是每次执行器需要时下载广播数据（因此必须在特定节点上多次下载相同的数据）。

Answer 1

是的，Spark确实使用种子广播优化了广播。引用消息来源

* A BitTorrent-like implementation of [[org.apache.spark.broadcast.Broadcast]].
*
* The mechanism is as follows:
*
* The driver divides the serialized object into small chunks and
* stores those chunks in the BlockManager of the driver.
*
* On each executor, the executor first attempts to fetch the object from its BlockManager. If
* it does not exist, it then uses remote fetches to fetch the small chunks from the driver and/or
* other executors if available. Once it gets the chunks, it puts the chunks in its own
* BlockManager, ready for other executors to fetch from.
*
* This prevents the driver from being the bottleneck in sending out multiple copies of the
* broadcast data (one per executor).

过去有另一种广播实现（HTTP广播），但在2.0中完全删除了。