用dataframe scala中的另一个日期对当前日期进行子基化

Question

+----------------+
|date            |
+----------------+
|8/11/2017 15:00 |
|9/11/2017 10:00 |
+----------------+

+----------------+------------+
|date            | difference |
+----------------+------------+
|8/11/2017 15:00 | xxxxxxxxxx |
|9/11/2017 10:00 | xxxxxxxxxx |
+----------------+------------+

val current = current_timestamp()
df.withColumn("difference", current - unix_timestamp(col(date)))

但是得到这个错误

org.apache.spark.sql.analysisException：无法解析“(current_timestamp()-unix_timestamp(date，'yyyy-mm-dd hh:mm:ss'))”，原因是数据类型不匹配：“(current_timestamp()-unix_timestamp(date，'yyyy-mm-ss'))中的类型不同（timestamp和bigint）。；；

我也试过

val current = BigInt(System.currenttimeMillis / 1000)
df.withColumn("difference", current - unix_timestamp(col(date))) 

val current = unix_timestamp(current_timestamp())
but the col "difference" is null

谢谢

Answer 1

必须为UNIX_TIMESTAMP使用正确的格式：

df.withColumn("difference", current_timestamp().cast("long") - unix_timestamp(col("date"), "dd/mm/yyyy HH:mm"))

或使用最新版本：

to_timestamp(col("date"), "dd/mm/yyyy HH:mm") - current_timestamp())

获取interval列。