什么时候键不能排序排序合并加入火花？

Question

当我读到关于排序合并连接的文章时，它说这是继广播连接之后火花中最首选的一个，但前提是连接键是可排序的。我的问题是什么时候连接键可以不可排序？任何数据类型都可以排序。你能帮我理解一个键可能不可排序的场景吗？

Answer 1

看见https://www.waitingforcode.com/apache-spark-sql/sort-merge-join-spark-sql/read.优秀的网站。

并非所有类型都可以排序。例如CalendarIntervalType。

引用:

"for not sortable keys the sort merge join" should "not be used" in {
import sparkSession.implicits._
// Here we explicitly define the schema. Thanks to that we can show
// the case when sort-merge join won't be used, i.e. when the key is not sortable
// (there are other cases - when broadcast or shuffle joins can be chosen over sort-merge
//  but it's not shown here).
// Globally, a "sortable" data type is:
// - NullType, one of AtomicType
// - StructType having all fields sortable
// - ArrayType typed to sortable field
// - User Defined DataType backed by a sortable field
// The method checking sortability is   org.apache.spark.sql.catalyst.expressions.RowOrdering.isOrderable
// As  you see, CalendarIntervalType is not included in any of above points,
// so even if the data structure is the same (id + login for customers, id + customer id + amount for orders)
// with exactly the same number of rows, the sort-merge join won't be applied here.

这是一个老帖子，从v3开始可以进行比较。https://spark . Apache . org/docs/3 . 0 . 0/API/Scala/org/Apache/spark/SQL/types/calendar intervaltype . html

但它表明了这一点。

另外，非等值连接呢？