python - 用 replace_regex 替换 pyspark 中的括号

Question

+---+------------+
|  A|           B|
+---+------------+
| x1|        [s1]|
| x2|   [s2 (A2)]|
| x3|   [s3 (A3)]|
| x4|   [s4 (A4)]|
| x5|   [s5 (A5)]|
| x6|   [s6 (A6)]|
+---+------------+

期望的结果：

+---+------------+-------+
|A  |B           |value  |
+---+------------+-------+
|x1 |[s1]        |[s1]   |
|x2 |[s2 (A2)]   |[s2]   |
|x3 |[s3 (A3)]   |[s3]   |
|x4 |[s4 (A4)]   |[s4]   |
|x5 |[s5 (A5)]   |[s5]   |
|x6 |[s6 (A6)]   |[s6]   |
+---+------------+-------+

当我应用下面的每个代码时，它们之前的括号和空格没有被替换：

from pyspark.sql.functions import expr
df.withColumn("C",
               expr('''transform(B, x-> regexp_replace(x, ' \\(A.\\)', ''))''')).show(truncate=False)

或者

df.withColumn("C",
               expr('''transform(B, x-> regexp_replace(x, ' \(A.\)', ''))''')).show(truncate=False)

得到的结果：

+---+------------+------------+
|A  |B           |value       |
+---+------------+------------+
|x1 |[s1]        |[s1]        |
|x2 |[s2 (A2)]   |[s2 ()]     |
|x3 |[s3 (A3)]   |[s3 ()]     |
|x4 |[s4 (A4)]   |[s4 ()]     |
|x5 |[s5 (A5)]   |[s5 ()]     |
|x6 |[s6 (A6)]   |[s6 ()]     |
+---+------------+------------+

Answer 1

您可以创建一个 UDF，从数组中删除与正则表达式匹配的所有元素r"\(.*\)"。如有必要，您可以根据需要更改正则表达式以匹配r"\(A.\)"。

import re
replaced = F.udf(lambda arr: [s for s in arr if not re.compile(r"\(.*\)").match(s)], \
                 T.ArrayType(T.StringType()))
df.withColumn("value", replaced("B")).show()

Answer 2

您可以拆分数组值并仅从first index数组中获取。

• （或）使用regexp_replace函数。

Example:

df.show()
#+---+---------+
#|  A|        B|
#+---+---------+
#| x1|     [s1]|
#| x2|[s2 (A2)]|
#+---+---------+

df.printSchema()
#root
# |-- A: string (nullable = true)
# |-- B: array (nullable = true)
# |    |-- element: string (containsNull = true)

df.withColumn("C",expr('''transform(B,x -> split(x,"\\\s+")[0])''')).show()

#using regexp_replace function
df.withColumn("C",expr('''transform(B,x -> regexp_replace(x,"(\\\s+.*)",""))''')).show()
df.withColumn("C",expr('''transform(B,x -> regexp_replace(x,"(\\\s+\\\((?i)A.+\\\))",""))''')).show()
#+---+---------+----+
#|  A|        B|   C|
#+---+---------+----+
#| x1|     [s1]|[s1]|
#| x2|[s2 (A2)]|[s2]|
#+---+---------+----+