4

假设数据源（csv）包含 2 列：code:string和my_date:date. 我尝试添加额外的列 -my_date_string:string并将其写入兽人格式，并在该列上使用分区：

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job

## @params: [JOB_NAME]
args = getResolvedOptions(sys.argv, ['JOB_NAME'])

sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)
datasource0 = glueContext.create_dynamic_frame.from_catalog(database = "my_database", table_name = "my_table", transformation_ctx = "datasource0")
applymapping1 = ApplyMapping.apply(frame = datasource0, mappings =[
                   ("code", "string", "code", "string"),
                   ("my_date", "date", "my_date", "date"),
                   ("my_date_string", "string", "my_date_string", "string")
                   ],
    transformation_ctx="applymapping1")
resolvechoice2 = ResolveChoice.apply(frame = applymapping1, choice = "make_struct", transformation_ctx = "resolvechoice2")
dropnullfields3 = DropNullFields.apply(frame = resolvechoice2, transformation_ctx = "dropnullfields3")

partitioned_dataframe = dropnullfields3.toDF().repartition("my_date_string")

partitioned_dataframe.toDF().write.orc(
    "s3://path/my_table",
    mode="append",
    partitionBy=['my_date_string'])

job.commit()

我得到了例外：

IllegalArgumentException: u"requirement failed: The number of columns doesn't match.\nOld column names (2): code" \
                          u", my_date"
New column names (0): "
End of LogType:stdout

似乎根本没有添加新列！！！:(

但是，当我删除重新分区语句时，它会写入所有列（例如所有新旧列）。如何解决这个问题？如何在重新分区之前执行列“添加”？

4

0 回答 0