0

这是我正在使用的示例数据框（df）：

+---+----+--------+
| id|orig|scrubbed|
+---+----+--------+
|  1|   a|       a|
|  2|   B|       b|
|  3|   c|       c|
|  4|   D|       d|
|  5|   *|      XX|
|  6|   $|      XX|
|  7|  ZZ|      ZZ|
|  8|  XX|      XX|
|  9|   y|       y|
| 10|   Z|       z|
+---+----+--------+

我想执行一项检查，告诉我擦洗后“填充”（不包含“XX”或“ZZ”）的项目比例是否至少为 80%。（此检查应该失败。）我可以添加一个合规性分析器VerificationRunBuilder来计算指标，如下所示：

val myVerificationResult: VerificationResult = new VerificationRunBuilder(df).
    addRequiredAnalyzer(
        Compliance(
            "populatedAfterScrubbing",
            "`scrubbed` NOT IN ('ZZ', 'XX') AND `scrubbed` IS NOT NULL",
            Some("`orig` NOT IN ('ZZ', 'XX') AND `orig` IS NOT NULL")
        )
    ).
    addCheck(
        Check(CheckLevel.Error, "Review Check").
            hasSize(_ >= 1)
    ).
    run()

此代码运行并使用约束成功检查数据hasSize，但我无法弄清楚如何根据我的自定义合规性分析器添加约束。这可能吗？

4

2 回答 2

0

我找到了一个似乎可行的解决方案，以防万一有人感兴趣。答案在于创建自定义约束而不是自定义分析器。这是工作代码：

val myConstraint = Constraint.complianceConstraint(
    "my constraint",
    "`scrubbed` NOT IN ('ZZ', 'XX') AND `scrubbed` IS NOT NULL",
    (fraction:Double)=>fraction>=0.8,
    Some("`orig` NOT IN ('ZZ', 'XX') AND `orig` IS NOT NULL"),
    Some("no peeking")
)

val myVerificationResult: VerificationResult = { VerificationSuite()
    .onData(df)
    .addCheck(
        Check(CheckLevel.Error, "Review Check") 
            .addConstraint(myConstraint)
    )
    .run()
}
val result = checkResultsAsDataFrame(spark, myVerificationResult)
result.show(truncate=true)

结果完全符合预期：

+------------+-----------+------------+--------------------+-----------------+--------------------+
|       check|check_level|check_status|          constraint|constraint_status|  constraint_message|
+------------+-----------+------------+--------------------+-----------------+--------------------+
|Review Check|      Error|       Error|ComplianceConstra...|          Failure|Value: 0.75 does ...|
+------------+-----------+------------+--------------------+-----------------+--------------------+
于 2020-05-15T16:14:37.847 回答
0

这不能通过检查来完成，仅使用统计https://github.com/awslabs/deequ/blob/master/src/main/scala/com/amazon/deequ/checks/Check.scala#L667之类的

Check(CheckLevel.Warning, "Statisfies TEST Constraint")      
      .satisfies("`scrubbed` NOT IN ('ZZ', 'XX') AND `scrubbed` IS NOT NULL",
                 "my constraint",
                 "fraction:Double",(fraction:Double)=>fraction>=0.8,
                  Some("..."))
      ))

我认为这是OOB，而不是通过complianceconstraint 进行定义，但如果您有相关的逻辑，这也是一种食物的想法。

于 2020-08-27T18:16:34.087 回答