仅当dataframe中存在列时应用withColumn

Question

我在Java 8中使用SPARK-SQL-2.4.1v。我有一个场景，我将被传递列名为list/seq，对于这些列，我只需要执行某些操作，如sum、avg、percentages等。

数据库中的示例数据。

val data = List(
  ("20", "score", "school", "2018-03-31", 14 , 12 , 20),
  ("21", "score", "school", "2018-03-31", 13 , 13 , 21),
  ("22", "rate", "school", "2018-03-31", 11 , 14, 22),
  ("21", "rate", "school", "2018-03-31", 13 , 12, 23)
 )
    val df = data.toDF("id", "code", "entity", "date", "column1", "column2" ,"column3")
.select("id", "code", "entity", "date", "column2") /// these are passed for each run....this set will keep changing.



  Dataset<Row> enrichedDs = df
             .withColumn("column1_org",col("column1"))
             .withColumn("column1",
                     when(col("column1").isNotNull() , functions.callUDF("lookUpData",col("column1").cast(DataTypes.StringType)))
                  );

上述逻辑仅适用于在选择列中“column1”可用时。这在第二个集合中是失败的，因为“column1”不是select，所以我需要一些理解为什么这只适用于作为“column1”的选定列是可用的。我需要一些逻辑来实现这一点。

Answer 1

检查这是否有帮助-

可以筛选出列，只处理有效的列

scala prettyprint-override">df.show(false)
    /**
      * +---+-----+------+----------+-------+-------+-------+
      * |id |code |entity|date      |column1|column2|column3|
      * +---+-----+------+----------+-------+-------+-------+
      * |20 |score|school|2018-03-31|14     |12     |20     |
      * |21 |score|school|2018-03-31|13     |13     |21     |
      * |22 |rate |school|2018-03-31|11     |14     |22     |
      * |21 |rate |school|2018-03-31|13     |12     |23     |
      * +---+-----+------+----------+-------+-------+-------+
      */
    // list of columns
    val cols = Seq("column1", "column2" ,"column3", "column4")
    val processColumns = cols.filter(df.columns.contains).map(sqrt)
    df.select(processColumns: _*).show(false)

    /**
      * +------------------+------------------+-----------------+
      * |SQRT(column1)     |SQRT(column2)     |SQRT(column3)    |
      * +------------------+------------------+-----------------+
      * |3.7416573867739413|3.4641016151377544|4.47213595499958 |
      * |3.605551275463989 |3.605551275463989 |4.58257569495584 |
      * |3.3166247903554   |3.7416573867739413|4.69041575982343 |
      * |3.605551275463989 |3.4641016151377544|4.795831523312719|
      * +------------------+------------------+-----------------+
      */