0

尝试从 pyspark 中的数据框中删除嵌套列不起作用。

这是我的代码中的 sinppet：

`from pyspark.sql import functions as F from pyspark.sql.types import IntegerType , BooleanType from pyspark.sql.functions import udf

#simple filter function
@F.udf(returnType=BooleanType())
def my_filter(x):
    if (x != df_new.a.b) :
       return True
    else :
       return False

#df.filter(my_filter('id', 'category')).show()

def drop_col(df, struct_nm, delete_struct_child_col_nm):
#fields_to_keep = filter(lambda x:  x != delete_struct_child_col_nm, df.select("{}.*".format(struct_nm)).columns)
fields_to_keep = filter(lambda x:  my_filter(x) , df.select("{}.*".format(struct_nm)).columns)
fields_to_keep = list(map(lambda x:  "{}.{}".format(struct_nm, x), fields_to_keep))
return df.withColumn(struct_nm, struct(fields_to_keep))

drop_col（df_new，“a”，df_new.ab）`

我使用 UDF 是因为尝试以下行不起作用。因为这不是！= 符号不起作用，也不能使用波浪号

#fields_to_keep = filter(lambda x:  x != delete_struct_child_col_nm,     df.select("{}.*".format

编辑：有人在评论中询问架构，所以我提供它

 root
 |-- a: struct (nullable = false)
 |    |-- rawEntity: string (nullable = true)
 |    |-- entityType: string (nullable = true)
 |    |    |-- element: struct (containsNull = true)
 |    |    |    |-- StoreId: string (nullable = true)
 |    |    |    |-- AppId: string (nullable = true)
 |    |-- Timestamps: array (nullable = true)
 |    |    |-- element: long (containsNull = true)
 |    |-- User: string (nullable = true)
 |    |-- b: array (nullable = true)                //trying to drop this
 |    |    |-- element: string (containsNull = true)
 |    |-- KeywordsFull: array (nullable = true)
 |    |    |-- element: struct (containsNull = true)
 |    |    |    |-- Keywords: string (nullable = true)
 |    |    |-- element: string (containsNull = true)
4

0 回答 0