当不满足所有选择条件时，Spark将选择哪一个联接？

Question

我们知道在Spark中有三种类型的连接——广播连接、随机连接和排序合并连接：

• 当小表连接大表时，使用广播加入;
• 当小表大于广播加入阈值时，使用随机连接;
• 当大表连接，并且连接键可以排序时，使用排序-合并连接;

如果存在两个大表的连接并且无法对连接键进行排序，会发生什么情况？Spark 将选择哪种联接类型？

Answer 1

Spark 3.0及更高版本支持以下类型的连接：

• 广播哈希连接（BHJ）
• 随机散列连接
• 随机排序合并联接（SMJ）
• 广播嵌套循环联接（BNLJ）
• 笛卡尔积联接

他们的选择在SparkStrategies.scala的源代码中得到了最好的概述：

  /**
   * Select the proper physical plan for join based on join strategy hints, the availability of
   * equi-join keys and the sizes of joining relations. Below are the existing join strategies,
   * their characteristics and their limitations.
   *
   * - Broadcast hash join (BHJ):
   *     Only supported for equi-joins, while the join keys do not need to be sortable.
   *     Supported for all join types except full outer joins.
   *     BHJ usually performs faster than the other join algorithms when the broadcast side is
   *     small. However, broadcasting tables is a network-intensive operation and it could cause
   *     OOM or perform badly in some cases, especially when the build/broadcast side is big.
   *
   * - Shuffle hash join:
   *     Only supported for equi-joins, while the join keys do not need to be sortable.
   *     Supported for all join types except full outer joins.
   *
   * - Shuffle sort merge join (SMJ):
   *     Only supported for equi-joins and the join keys have to be sortable.
   *     Supported for all join types.
   *
   * - Broadcast nested loop join (BNLJ):
   *     Supports both equi-joins and non-equi-joins.
   *     Supports all the join types, but the implementation is optimized for:
   *       1) broadcasting the left side in a right outer join;
   *       2) broadcasting the right side in a left outer, left semi, left anti or existence join;
   *       3) broadcasting either side in an inner-like join.
   *     For other cases, we need to scan the data multiple times, which can be rather slow.
   *
   * - Shuffle-and-replicate nested loop join (a.k.a. cartesian product join):
   *     Supports both equi-joins and non-equi-joins.
   *     Supports only inner like joins.
   */
object JoinSelection extends Strategy with PredicateHelper { ...

如上所述，应用选择的结果不仅取决于表的大小和键的可排序性，还取决于连接类型（INNER、LEFT/RIGHT、FULL）和连接键条件（相等与非相等/θ）。总的来说，在您的情况下，您将看到Shuffle Hash或Broadcast Nested Loop。