我正在尝试使用 spark-csv_2.10-1.3.0 将 csv 文件加载到数据帧
sqlContext.read.format('com.databricks.spark.csv')
.options(header='true',dateFormat='dd/MM/YYYY hh:mm')
.load('test.csv',schema = s_acr_rv_derivative_schema)
架构:
s_acr_rv_derivative_schema = StructType([
StructField("COUNTERPARTY", StringType(), True),
StructField("ORIG_COUNTERPARTY", StringType(), True),
StructField("CUSTOMER_NAME", StringType(), True),
StructField("TRADE_ID", StringType(), True),
StructField("TRADE_DATE", DateType(), True)]
问题:
Caused by: java.lang.IllegalArgumentException
at java.sql.Date.valueOf(Date.java:140)
at com.databricks.spark.csv.util.TypeCast$.castTo(TypeCast.scala:64)