自 Spark 2.3.0 ( https://issues.apache.org/jira/browse/SPARK-19112 ) 起， Apache Spark 据称支持 Facebook 的 Zstandard 压缩算法，但我实际上无法读取 Zstandard 压缩文件：
$ spark-shell
...
// Short name throws an exception
scala> val events = spark.read.option("compression", "zstd").json("data.zst")
java.lang.IllegalArgumentException: Codec [zstd] is not available. Known codecs are bzip2, deflate, uncompressed, lz4, gzip, snappy, none.
// Codec class can be imported
scala> import org.apache.spark.io.ZStdCompressionCodec
import org.apache.spark.io.ZStdCompressionCodec
// Fully-qualified code class bypasses error, but results in corrupt records
scala> spark.read.option("compression", "org.apache.spark.io.ZStdCompressionCodec").json("data.zst")
res4: org.apache.spark.sql.DataFrame = [_corrupt_record: string]
我需要做什么才能读取这样的文件？
环境是 AWS EMR 5.14.0。