Spark数据帧中的不同记录计数

Question

我试图在火花数据帧中显示几个不同列的不同计数，以及对第一列进行分组后的记录计数。
所以如果我有col1、col2和col3，我想按col1分组，然后显示col2的不同计数和col3的不同计数。然后，我想在col1的同一组之后显示记录计数。最后，在一个agg语句中完成这一切…有什么想法吗？

Answer 1

下面是您正在寻找的代码

df.groupBy("COL1").agg(countDistinct("COL2"),countDistinct("COL3"),count($"*")).show

=======以下测试===========

scala>  val lst = List(("a","x","d"),("b","D","s"),("ss","kk","ll"),("a","y","e"),("b","c","y"),("a","x","y"));
lst: List[(String, String, String)] = List((a,x,d), (b,D,s), (ss,kk,ll), (a,y,e), (b,c,y), (a,x,y))

scala> val rdd=sc.makeRDD(lst);
rdd: org.apache.spark.rdd.RDD[(String, String, String)] = ParallelCollectionRDD[7] at makeRDD at <console>:26

scala> val df = rdd.toDF("COL1","COL2","COL3");
df: org.apache.spark.sql.DataFrame = [COL1: string, COL2: string ... 1 more field]

scala> df.printSchema
root
 |-- COL1: string (nullable = true)
 |-- COL2: string (nullable = true)
 |-- COL3: string (nullable = true)


scala> df.groupBy("COL1").agg(countDistinct("COL2"),countDistinct("COL3"),count($"*")).show
+----+--------------------+--------------------+--------+
|COL1|count(DISTINCT COL2)|count(DISTINCT COL3)|count(1)|
+----+--------------------+--------------------+--------+
|  ss|                   1|                   1|       1|
|   b|                   2|                   2|       2|
|   a|                   2|                   3|       3|
+----+--------------------+--------------------+--------+


scala>