scala - Spark 2.2 数据框 [scala]

Question

OrderNo    Status1    Status2     Status3
123    Completed      Pending     Pending
456    Rejected   Completed   Completed
789    Pending    In Progress     Completed

上面是输入数据集的表格，下面是预期的输出。这里的问题是我们应该根据订单号而不是状态出现次数来计算。我们可以在使用 scala 的 spark 数据帧的帮助下做到这一点吗？提前感谢您的帮助。

Pending     2
Rejected    1
Completed   3
In Progress 2

Answer 1

你可以试试下面的代码。它计算所有状态的不同 OrderNo 的数量。我希望它有所帮助。

val rawDF = Seq(
  ("123", "Completed", "Pending", "Pending"),
  ("456", "Rejected", "Completed", "Completed"),
  ("789", "Pending", "In Progress", "Completed")
).toDF("OrderNo", "Status1", "Status2", "Status3")

val newDF = rawDF.withColumn("All_Status",  array($"Status1", $"Status2", $"Status3"))
    .withColumn("Status", explode($"All_Status"))
    .groupBy("Status").agg(size(collect_set($"OrderNo")).as("DistOrderCnt"))

这是结果。（注意：In Progress 仅在测试数据中出现一次。）

+-----------+------------+ | Status|DistOrderCnt| +-----------+------------+ | Completed| 3| |In Progress| 1| | Pending| 2| | Rejected| 1| +-----------+------------+