0

我正在尝试获取使用 DataFrame 创建的 Delta Lake 表的元数据信息。有关版本、时间戳的信息。

尝试过：spark.sql("describe deltaSample").show(10,false)-这没有提供与版本和时间戳相关的信息：

我想知道有多少个带有时间戳信息的版本 

+--------+---------+-------+
|col_name|data_type|comment|
+--------+---------+-------+
|_c0     |string   |null   |
|_c1     |string   |null   |
+--------+---------+-------+

下面是代码： // 在 spark-shell 中下载 delta

spark2-shell --packages io.delta:delta-core_2.11:0.2.0
val data = spark.read.csv("/xyz/deltaLake/deltaLakeSample.csv")

// 保存数据框

data.write.format("delta").save("/xyz/deltaLake/deltaSample")

//创建delta湖表

spark.sql("create table deltaSample using delta location '/xyz/deltaLake/deltaSample'")
val updatedInfo = data.withColumn("_c1",when(col("_c1").equalTo("right"), "updated").otherwise(col("_c1")) )

//更新delta湖表

updatedInfo.write.format("delta").mode("overwrite").save("/xyz/deltaLake/deltaSample")
spark.read.format("delta").option("versionAsOf", 0).load("/xyz/deltaLake/deltaSample/").show(10,false)
+---+-----+
|_c0|_c1  |
+---+-----+
|rt |right|
|lt |left |
|bk |back |
|frt|front|

+---+-----+

spark.read.format("delta").option("versionAsOf", 1).load("/xyz/deltaLake/deltaSample/").show(10,false)
+---+-------+
|_c0|_c1    |
+---+-------+
|rt |updated|
|lt |left   |
|bk |back   |
|frt|front  |
+---+-------+

// 获取创建的表的元数据。带有版本，时间戳信息。

spark.sql("describe history deltaSample") -- not working

org.apache.spark.sql.AnalysisException: Table or view was not found: history;
  at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:47)
  at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveRelations$$lookupTableFromCatalog(Analyzer.scala:733)
  at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.resolveRelation(Analyzer.scala:685)

预期的表格显示（例如：添加的列版本、时间戳）：

+--------+---------+-------+-------+------------
|_c0     |_c1      |Version|timestamp           |
+--------+---------+-------+-------+------------
|rt      |right    |0      |2019-07-22 00:24:00|
|lt      |left     |0      |2019-07-22 00:24:00|
|rt      |updated  |1      |2019-08-22 00:25:60|
|lt      |left     |1      |2019-08-22 00:25:60|
+--------+---------+-------+------------------+
4

1 回答 1

2

查看 Delta Lake 表历史记录的功能包含在最近宣布的 0.3.0 中，每个宣布 Delta Lake 0.3.0 版本。

目前，您可以使用 Scala API 执行此操作；有能力在 SQL 中做到这一点目前在路线图上。对于 Scala API 示例，使用0.3.0,

import io.delta.tables._

val deltaTable = DeltaTable.forPath(spark, pathToTable)

val fullHistoryDF = deltaTable.history()    // get the full history of the table.

val lastOperationDF = deltaTable.history(1) // get the last operation.

结果fullHistoryDF类似于：

+-------+-------------------+------+--------+---------+--------------------+----+--------+---------+-----------+--------------+-------------+
|version|          timestamp|userId|userName|operation| operationParameters| job|notebook|clusterId|readVersion|isolationLevel|isBlindAppend|
+-------+-------------------+------+--------+---------+--------------------+----+--------+---------+-----------+--------------+-------------+
|      5|2019-07-29 14:07:47|  null|    null|   DELETE|[predicate -> ["(...|null|    null|     null|          4|          null|        false|
|      4|2019-07-29 14:07:41|  null|    null|   UPDATE|[predicate -> (id...|null|    null|     null|          3|          null|        false|
|      3|2019-07-29 14:07:29|  null|    null|   DELETE|[predicate -> ["(...|null|    null|     null|          2|          null|        false|
|      2|2019-07-29 14:06:56|  null|    null|   UPDATE|[predicate -> (id...|null|    null|     null|          1|          null|        false|
|      1|2019-07-29 14:04:31|  null|    null|   DELETE|[predicate -> ["(...|null|    null|     null|          0|          null|        false|
|      0|2019-07-29 14:01:40|  null|    null|    WRITE|[mode -> ErrorIfE...|null|    null|     null|       null|          null|         true|
+-------+-------------------+------+--------+---------+--------------------+----+--------+---------+-----------+--------------+-------------+
于 2019-08-02T14:25:28.747 回答