rlike中的pyspark dataframe如何从dataframe列中逐行传递字符串值[重复]

Question

运行时得到错误meesagge

    df.withColumn("match_str", df.text1.rlike(df.match)).show(truncate=False)

        Py4JError: An error occurred while calling o2165.rlike. Trace:
        py4j.Py4JException: Method rlike([class org.apache.spark.sql.Column]) does not exist

你知道什么变通办法或解决办法吗？

    df = spark.createDataFrame([
        (1, 'test1 test1_0|test1 test0', 'This is a test1 test1_0'),
        (2, 'test2 test2_0|test1 test0', None),
        (3, 'Nan', 5.2, 23, 'Nan'),
        (4, 'test4 test4_0|test1 test0', 'This is a test4 test4_0'),
       ], ['id', 'match', 'text1'])



    +---+-------------------------+-----------------------+
    |id |match                    |text1                  |
    +---+-------------------------+-----------------------+
    |1  |test1 test1_0|test1 test0|This is a test1 test1_0|
    |2  |test2 test2_0|test1 test0|null                   |
    |3  |Nan                      |Nan                    |
    |4  |test4 test4_0|test1 test0|This is a test4 test4_0|
    +---+-------------------------+-----------------------+

    root
     |-- id: long (nullable = true)
     |-- match: string (nullable = true)
     |-- text1: string (nullable = true)


    df.withColumn("match_str", df.text1.rlike(df.select(df.match).head()["match"])).show(truncate=False)

注意:df.select(df.match).head()[“match”]传递值第一行匹配，在本例中匹配“test1 test1_0test1 test0”到所有行。我想逐行传递rlike值。像

null

    +---+-------------------------+-----------------------+---------+
    |id |match                    |text1                  |match_str|
    +---+-------------------------+-----------------------+---------+
    |1  |test1 test1_0|test1 test0|This is a test1 test1_0|true     |
    |2  |test2 test2_0|test1 test0|null                   |null     |
    |3  |Nan                      |Nan                    |false    |
    |4  |test4 test4_0|test1 test0|This is a test4 test4_0|false    |
    +---+-------------------------+-----------------------+---------+

    df.withColumn("match_str", df.text1.rlike(df.match)).show(truncate=False)

        Py4JError: An error occurred while calling o2165.rlike. Trace:
        py4j.Py4JException: Method rlike([class org.apache.spark.sql.Column]) does not exist

预期成果：

    +---+-------------------------+-----------------------+---------+
    |id |match                    |text1                  |match_str|
    +---+-------------------------+-----------------------+---------+
    |1  |test1 test1_0|test1 test0|This is a test1 test1_0|true     |
    |2  |test2 test2_0|test1 test0|null                   |false    |
    |3  |Nan                      |Nan                    |true     |
    |4  |test4 test4_0|test1 test0|This is a test4 test4_0|true     |
    +---+-------------------------+-----------------------+---------+

Answer 1

不幸的是，pyspark.sql.column.rlike()方法只接受text模式，而不接受其他列作为模式（但是您可以使用UDF-S)根据需要调整它）。

您的问题的快速解决方案是使用Pyspark sqlrlike（就像普通sqlrlike):

>>> from pyspark.sql import *
>>> from pyspark.sql.functions import *
>>> df = sqlContext.createDataFrame([
...     (1, 'test1 test1_0|test1 test0', 'This is a test1 test1_0'),
...     (2, 'test2 test2_0|test1 test0', None),
...     (3, 'Nan', 'Nan'),
...     (4, 'test4 test4_0|test1 test0', 'This is a test4 test4_0')
...    ], ['id', 'match', 'text1'])
>>> df.select("id", "match", "text1", expr("text1 rlike concat('(', match, ')$') as match_str")).show()
+---+--------------------+--------------------+---------+
| id|               match|               text1|match_str|
+---+--------------------+--------------------+---------+
|  1|test1 test1_0|tes...|This is a test1 t...|     true|
|  2|test2 test2_0|tes...|                null|     null|
|  3|                 Nan|                 Nan|     true|
|  4|test4 test4_0|tes...|This is a test4 t...|     true|
+---+--------------------+--------------------+---------+

只是稍微修改一下您的示例，因为您在那里操作字符串，并且“nan”字符串等于其他“nan”字符串：

>>>
... df2 = sqlContext.createDataFrame([
...     (1, 'test1 test1_0|test1 test0', 'This is a test1 test1_0x'),
...     (2, 'test2 test2_0|test1 test0', None),
...     (3, 'NanA', 'Nan'),
...     (4, 'test4 test4_0|test1 test0', 'This is a test4 test4_0')
...    ], ['id', 'match', 'text1'])
>>>
... df2.select("id", "match", "text1", expr("text1 rlike concat('(', match, ')$') as match_str")).show()
+---+--------------------+--------------------+---------+
| id|               match|               text1|match_str|
+---+--------------------+--------------------+---------+
|  1|test1 test1_0|tes...|This is a test1 t...|    false|
|  2|test2 test2_0|tes...|                null|     null|
|  3|                NanA|                 Nan|    false|
|  4|test4 test4_0|tes...|This is a test4 t...|     true|
+---+--------------------+--------------------+---------+