我目前有一个从 S3 存储桶获取数据并生成数据框的脚本。我想将数据帧发送到雪花。目前我正在这样做
df_write = df \
.write\
.format(SNOWFLAKE_SOURCE_NAME)\
.options(**sfOptions)\
.option("dbtable","test_table")\
.mode("append")\
.save()
产生的错误是
An error occurred while calling o72.save.
: java.lang.NoSuchMethodError: scala.Product.$init$(Lscala/Product;)V
at net.snowflake.spark.snowflake.Parameters$MergedParameters.<init>(Parameters.scala:294)
at net.snowflake.spark.snowflake.Parameters$.mergeParameters(Parameters.scala:288)
at net.snowflake.spark.snowflake.DefaultSource.createRelation(DefaultSource.scala:97)
at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
我的想法是这可能是不匹配的版本错误。目前我正在使用 hadoop-aws-2.7.3.jar、spark-snowflake_2.12-2.8.1-spark_3.0.jar、snowflake-jdbc-3.12.8.jar