azure-data-explorer - Kusto - 最新版本行的物化视图

Question

我有一个名为“元数据”的表，其中包含由 TestId 分区的 Parameter 和 ParamterValue 列表。每次更改测试时，测试都将重新引入到 Azure 数据资源管理器中，并且没有版本。

我的总体目标是：

定义一个函数（GetTestsFromSearch），它采用键值对的参数（动态），让我查询所有测试（最新版本）以查找键/值对的匹配：

( {{"Search param1", "Search value1"},{"Search param2", "Search value2"}}

例子

GetTestsFromSearch({{"ProjectId", "SturnProject"},{"Product Name", "Nacelle "}})

应该返回

测试ID	版本
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0	3
ea5b688c-b61f-4c5b-bb87-af2eac94d454	1

从下面的示例元数据表中

另一个目标是为每个测试创建一个仅包含最新元数据的物化视图（解释如下表）

元数据表示例

测试ID	测试名称	范围	参数值	版本
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0	我的测试	项目编号	Sturn 项目	1
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0	我的测试	产品分类	2MW	1
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0	我的测试	项目开始日期	2022-02-03	1
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0	我的测试	项目编号	Sturn 项目	2
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0	我的测试	产品分类	2MW	2
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0	我的测试	项目开始日期	2022-02-03	2
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0	我的测试	项目编号	Sturn 项目	3
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0	我的测试	产品分类	2MW	3
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0	我的测试	项目开始日期	2022-02-03	3
ea5b688c-b61f-4c5b-bb87-af2eac94d454	我的测试	项目编号	Sturn 项目	1
ea5b688c-b61f-4c5b-bb87-af2eac94d454	我的测试	项目状态	打开	1
ea5b688c-b61f-4c5b-bb87-af2eac94d454	我的测试	产品名称	机舱	1

随着时间的推移，在几个不同的版本中会有数千个测试，因此我预计，创建一个仅维护每个测试的最新版本的物化视图是一个好主意- 我试图将视图创建为：

metadata
| summarize arg_max(Version,*) by TestId

但这仅给了我每个 TestId/Version 的一个参数和参数值，而不是测试的整个结果集。

任何人都可以为这个物化视图指出正确的方向吗？

我已经包含了一个元数据表的示例作为 DataTable，可以直接在 kusto 中使用。

元数据表作为 DataTable

datatable (TestId: string, Name: string, Parameter: string, ParameterValue: string, Version: int) [
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test Report, DMS number","1234-231",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Project name","Thor3",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","GTRS reference","gtrs",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Product Category","2MW",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Project number","TE-12321",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","DUT responsible person","ANFRB3",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test execution person","ANFRB3",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Project Manager","ANFRB3",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","DVPR, DMS number","1234-1234",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","DVRE, DMS number","1231-1213",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test Start Date","2022-02-23T00:00:00.0000000Z",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test Category","Verification safety",int(3),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","GTRS reference","gtrs",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Project number","TE-12321",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","DUT responsible person","ANFRB2",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test execution person","ANFRB2",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Project Manager","ANFRB2",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","DVPR, DMS number","1234-1234",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","DVRE, DMS number","1231-1213",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test Start Date","2022-02-23T00:00:00.0000000Z",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test Category","Verification safety",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Product Category","2MW",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Project name","Thor3",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test Report, DMS number","1234-231",int(2),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test Category","Verification safety",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Project Manager","ANFRB",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","GTRS reference","gtrs",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Product Category","2MW",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Project name","Thor3",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Project number","TE-12321",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","DUT responsible person","ANFRB",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test execution person","ANFRB",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","DVPR, DMS number","1234-1234",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test Report, DMS number","1234-231",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","DVRE, DMS number","1231-1213",int(1),
    "fc76aa10-5cf8-447e-95f6-3bd801ef2ed0","ANFRB-FILEVIEW-TEST","Test Start Date","2022-02-23T00:00:00.0000000Z",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","GTRS reference","gtrs232",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Product Category","4MW",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Project name","Myproject",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Project number","43324534",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","DUT responsible person","ANFRB",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Test execution person","ANFRB",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Project Manager","ANFRB",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","DVPR, DMS number","435123454",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Test Report, DMS number","123123123",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","DVRE, DMS number","12312312312",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Test Start Date","2022-03-01T00:00:00.0000000Z",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Test Category","Verification functionality",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Test facility","CHE",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Test rig","rig23",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Sample ID","1",int(1),
    "ea5b688c-b61f-4c5b-bb87-af2eac94d454","ANFRB2-TEST","Link to test data","asdfsafdsdfa",int(1)
]

谢谢

score 1 · Accepted Answer

如果大多数测试包含相同的属性（如您的示例中所示），您可以考虑将架构更改为宽架构，其中a和的每次运行 ( Version)都是一条记录。结果架构将类似于以下输出：TestIdName

**datatable**
| extend pack(Parameter, ParameterValue)
| summarize make_bag(Column1) by TestId, Name, Version
| evaluate bag_unpack(bag_Column1)

然后，您可以使用以下聚合设置物化视图，该聚合将提供您正在寻找的 IIUC：

T | summarize arg_max(Version, *) by TestId, Name

要从示例中的架构切换到建议的架构，您可以将摄取管道更改为以新的架构格式摄取，或使用更新策略进行转换。如果选择后者，请避免在更新策略功能中使用bag_unpack插件。相反，明确投影您需要的列，以避免非确定性模式。

另一种选择是将所有属性保留在单个动态列中，结果如下：

**datatable**
| extend pack(Parameter, ParameterValue)
| summarize make_bag(Column1) by TestId, Name, Version

并使用与上述相同的物化视图定义。

对于第一个问题 - 使用第二个建议的架构，您可以尝试以下操作：

let GetTestsFromSearch = (Filter:dynamic)
{
    T
    | extend pack(Parameter, ParameterValue)
    | summarize Properties = make_bag(Column1) by TestId, Name, Version  
    | summarize arg_max(Version, *) by TestId, Name
    | extend Filter
    | mv-apply Filter on 
    ( 
        extend key=tostring(bag_keys(Filter)[0]) 
        | extend expected = tostring(Filter[key]), actual = tostring(Properties[key])
        | summarize count(), countif(actual == expected)
        | where count_  == countif_
    )
};
GetTestsFromSearch(dynamic({"Test Category" : "Verification safety", "Project name" : "Thor3"}));

测试ID	姓名	版本	特性	数数_	计数
fc76aa10-5cf8-447e-95f6-3bd801ef2ed0	ANFRB-文件查看-测试	3	{ “测试报告，DMS 编号”：“1234-231”， “项目名称”：“Thor3”， “GTRS 参考”：“gtrs”， “产品类别”：“2MW”， “项目编号”：“TE- 12321”、 “DUT负责人”：“ANFRB3”、 “测试执行人”：“ANFRB3”、 “项目经理”：“ANFRB3”、 “DVPR、DMS编号”：“1234-1234”、 “DVRE、DMS编号": "1231-1213", "测试开始日期": "2022-02-23T00:00:00.0000000Z", "测试类别": "验证安全性" }	2	2

azure-data-explorer - Kusto - 最新版本行的物化视图

1 回答 1

Related

Reference