apache-spark - 保存 pyspark 数据框后列的可空性不正确

Question

当保存一个 pyspark 数据框并添加了一个带有“withColumn”函数的新列时，可空性从 false 变为 true。

版本信息：Python 3.7.3/Spark2.4.0-cdh6.1.1

>>> l = [('Alice', 1)]
>>> df = spark.createDataFrame(l)
>>> df.printSchema()
root
 |-- _1: string (nullable = true)
 |-- _2: long (nullable = true)

>>> from pyspark.sql.functions import lit
>>> df = df.withColumn('newCol', lit('newVal'))
>>> df.printSchema()
root
 |-- _1: string (nullable = true)
 |-- _2: long (nullable = true)
 |-- newCol: string (nullable = false)

>>> df.write.saveAsTable('default.withcolTest', mode='overwrite')

>>> spark.sql("select * from default.withcolTest").printSchema()
root
 |-- _1: string (nullable = true)
 |-- _2: long (nullable = true)
 |-- newCol: string (nullable = true)

为什么在持久化数据帧时，newCol添加函数的列的可为空标志会发生变化？withColumn