2

我在 ES 5 中映射了以下文档：

{
   "appName" : {
      "mappings" : {
         "market_audit" : {
            "properties" : {
               "generation_date": {
                  "type": "date"
               },
               "customers" : {
                  "type" : "nested",
                  "properties" : {
                     "customer_id" : {
                        "type" : "integer"
                     },
   [... other properties ...]
}

“客户”节点中的多个条目可能具有相同的 customer_id，我试图仅检索具有特定 customer_id（即“1”）的条目以及顶级文档的“generation_date”（仅最新文件将被处理）。

我能够提出以下查询：

{
  "query": {},
  "sort": [
    { "generation_date": "desc" }
  ],
  "size": 1,
  "aggregations": {
    "nested": {
      "nested": {
        "path": "customers"
      },
      "aggregations": {
        "filter": {
          "filter": {
            "match": {
              "customers.customer_id": {
                "query": "1"
              }
            }
          },
          "aggregations": {
            "tophits_agg": {
              "top_hits": {}
            }
          }
        }
      }
    }
  }
}

此查询获取我感兴趣的数据，位于“聚合”数组中（以及包含整个文档的“命中”数组）。这里的问题是我使用的框架（ONGR 的 ElasticSearch 包以及 DSL 包，使用 Symfony3）每次尝试访问没有可用存储桶的实际数据时都会抱怨。

我已阅读 ES 文档，但无法提出添加存储桶的有效查询。我确定我遗漏了一些东西，一点帮助将非常受欢迎。如果您对如何适当地修改查询有一个想法，我想我可以想出 PHP 代码来生成它。

编辑：由于这个问题得到了一些意见并且没有答案（而且我仍然被困住），我会接受任何允许我从生成的最新文档中检索有关特定“客户”（使用 customer_id）信息的查询（根据“generation_date”字段）。我给出的查询正是我能够提出的，我很确定有更好的方法来做到这一点。建议可能吗？

编辑 2：这是发送到 ES 的数据：

{
    "index": {
    "_type": "market_data_audit_document"
    }
}
{
    "customers": [
    {
        "customer_id": 1,
        "colocation_name": "colo1",
        "colocation_id": 26,
        "device_name": "device 1",
        "channels": [
        {
            "name": "channel1-5",
            "multicast":"1.2.1.5",
            "sugar_state":4,
            "network_state":1
        }
        ]
    },
    {
        "customer_id":2,
        "colocation_name":"colo2",
        "colocation_id":27,
        "device_name":"device 2",
        "channels": [
        {
            "name":"channel2-5",
            "multicast":"1.2.2.5",
            "sugar_state":4,
            "network_state":1
        }
        ]
    },
    {
        "customer_id":3,
        "colocation_name":"colo3",
        "colocation_id":28,
        "device_name":"device 3",
        "channels": [
        {
            "name":"channel3-5",
            "multicast":"1.2.3.5",
            "sugar_state":4,
            "network_state":1
        }
        ]
    },
    {
        "customer_id":4,
        "colocation_name":"colo4",
        "colocation_id":29,
        "device_name":"device 4"
        ,"channels": [
        {
            "name":"channel4-5",
            "multicast":"1.2.4.5",
            "sugar_state":4,
            "network_state":1
        }
        ]
    },
    {
        "customer_id":5,
        "colocation_name":"colo5",
        "colocation_id":30,
        "device_name":"device 5",
        "channels": [
        {
            "name":"channel5-5",
            "multicast":"1.2.5.5",
            "sugar_state":4,
            "network_state":1
        }
        ]
    }
    ],
    "generation_date":"2017-02-27T10:55:45+0100"
}

不幸的是，当我尝试发送这篇文章中列出的查询时，我发现聚合并没有像我预期的那样：它返回“好”数据，但来自所有存储的文档！这是一个输出示例：

{
   "timed_out" : false,
   "took" : 60,
   "hits" : {
      "total" : 2,
      "hits" : [
         {
            "_source" : {
               "customers" : [
                  {
                     "colocation_id" : 26,
                     "channels" : [
                        {
                           "name" : "channel1-5",
                           "sugar_state" : 4,
                           "network_state" : 1,
                           "multicast" : "1.2.1.5"
                        }
                     ],
                     "customer_id" : 1,
                     "colocation_name" : "colo1",
                     "device_name" : "device 1"
                  },
                  {
                     "colocation_id" : 27,
                     "channels" : [
                        {
                           "multicast" : "1.2.2.5",
                           "network_state" : 1,
                           "name" : "channel2-5",
                           "sugar_state" : 4
                        }
                     ],
                     "customer_id" : 2,
                     "device_name" : "device 2",
                     "colocation_name" : "colo2"
                  },
                  {
                     "device_name" : "device 3",
                     "colocation_name" : "colo3",
                     "customer_id" : 3,
                     "channels" : [
                        {
                           "multicast" : "1.2.3.5",
                           "network_state" : 1,
                           "sugar_state" : 4,
                           "name" : "channel3-5"
                        }
                     ],
                     "colocation_id" : 28
                  },
                  {
                     "channels" : [
                        {
                           "sugar_state" : 4,
                           "name" : "channel4-5",
                           "multicast" : "1.2.4.5",
                           "network_state" : 1
                        }
                     ],
                     "customer_id" : 4,
                     "colocation_id" : 29,
                     "colocation_name" : "colo4",
                     "device_name" : "device 4"
                  },
                  {
                     "device_name" : "device 5",
                     "colocation_name" : "colo5",
                     "colocation_id" : 30,
                     "channels" : [
                        {
                           "sugar_state" : 4,
                           "name" : "channel5-5",
                           "multicast" : "1.2.5.5",
                           "network_state" : 1
                        }
                     ],
                     "customer_id" : 5
                  }
               ],
               "generation_date" : "2017-02-27T11:45:37+0100"
            },
            "_type" : "market_data_audit_document",
            "sort" : [
               1488192337000
            ],
            "_index" : "mars",
            "_score" : null,
            "_id" : "AVp_LPeJdrvi0cWb8CrL"
         }
      ],
      "max_score" : null
   },
   "aggregations" : {
      "nested" : {
         "doc_count" : 10,
         "filter" : {
            "doc_count" : 2,
            "tophits_agg" : {
               "hits" : {
                  "max_score" : 1,
                  "total" : 2,
                  "hits" : [
                     {
                        "_nested" : {
                           "offset" : 0,
                           "field" : "customers"
                        },
                        "_score" : 1,
                        "_source" : {
                           "channels" : [
                              {
                                 "name" : "channel1-5",
                                 "sugar_state" : 4,
                                 "multicast" : "1.2.1.5",
                                 "network_state" : 1
                              }
                           ],
                           "customer_id" : 1,
                           "colocation_id" : 26,
                           "colocation_name" : "colo1",
                           "device_name" : "device 1"
                        }
                     },
                     {
                        "_source" : {
                           "colocation_id" : 26,
                           "customer_id" : 1,
                           "channels" : [
                              {
                                 "multicast" : "1.2.1.5",
                                 "network_state" : 1,
                                 "name" : "channel1-5",
                                 "sugar_state" : 4
                              }
                           ],
                           "device_name" : "device 1",
                           "colocation_name" : "colo1"
                        },
                        "_nested" : {
                           "offset" : 0,
                           "field" : "customers"
                        },
                        "_score" : 1
                     }
                  ]
               }
            }
         }
      }
   },
   "_shards" : {
      "total" : 13,
      "successful" : 1,
      "failures" : [
         {
            "reason" : {
               "index" : ".kibana",
               "index_uuid" : "bTkwoysSQ0y8Tt9yYFRStg",
               "type" : "query_shard_exception",
               "reason" : "No mapping found for [generation_date] in order to sort on"
            },
            "shard" : 0,
            "node" : "4ZUgOm4VRry6EtUK15UH3Q",
            "index" : ".kibana"
         },
         {
            "reason" : {
               "index_uuid" : "lN2mVF9bRjuDtiBF2qACfA",
               "index" : "archiv1_log",
               "type" : "query_shard_exception",
               "reason" : "No mapping found for [generation_date] in order to sort on"
            },
            "shard" : 0,
            "node" : "4ZUgOm4VRry6EtUK15UH3Q",
            "index" : "archiv1_log"
         },
         {
            "index" : "archiv1_session",
            "shard" : 0,
            "node" : "4ZUgOm4VRry6EtUK15UH3Q",
            "reason" : {
               "type" : "query_shard_exception",
               "index" : "archiv1_session",
               "index_uuid" : "cmMAW04YTtCb0khEqHpNyA",
               "reason" : "No mapping found for [generation_date] in order to sort on"
            }
         },
         {
            "shard" : 0,
            "node" : "4ZUgOm4VRry6EtUK15UH3Q",
            "reason" : {
               "reason" : "No mapping found for [generation_date] in order to sort on",
               "index" : "archiv1_users_dev",
               "index_uuid" : "AH48gIf5T0CXSQaE7uvVRg",
               "type" : "query_shard_exception"
            },
            "index" : "archiv1_users_dev"
         }
      ],
      "failed" : 12
   }
}
4

2 回答 2

1

根据您的描述：

  • 您将文档存储在具有一堆属性的弹性搜索上
  • 每个文档都包含数组中的客户列表（嵌套文档）
  • 您只想提取与 customer.id 相关的嵌套文档
  • 您的库不管理没有存储桶的 Elasticsearch 响应
  • 您期望 Elasticsearch 返回嵌套文档

问题

它存在两种聚合：

  • 水桶
  • 指标

在您的情况下，您在 Nested Agg 下有 2 个聚合：过滤器和指标。筛选 ：

解决方法：

我怀疑您的 PHP 库能否正确处理嵌套聚合结果，但您可以使用 Filter代替Filter Aggregations 来获取存储桶列表

{
  "aggregations": {
    "nested": {
      "nested": {
        "path": "customers"
      },
      "aggregations": {
        "filters_customer": {
          "filters": {
            "filters": [
              {
                "match": {
                  "customers.customer_id": "1"
                }
              }
            ]
          },
          "aggregations": {
            "top_hits_customer": {
              "top_hits": {}
            }
          }
        }
      }
    }
  }
}

将提供类似：

{
  "aggregations": {
    "nested": {
      "doc_count": 15,
      "filters_customer": {
        "buckets": [
          {
            "doc_count": 3,
            "top_hits_customer": {
              "hits": {
                "total": 3,
                "max_score": 1,
                "hits": [
                  {
                    "_nested": {
                      "field": "customers",
                      "offset": 0
                    },
                    "_score": 1,
                    "_source": {
                      "customer_id": 1,
                      "foo": "bar"
                    }
                  },
                  {
                    "_nested": {
                      "field": "customers",
                      "offset": 0
                    },
                    "_score": 1,
                    "_source": {
                      "customer_id": 1,
                      "foo": "bar"
                    }
                  },
                  {
                    "_nested": {
                      "field": "customers",
                      "offset": 0
                    },
                    "_score": 1,
                    "_source": {
                      "customer_id": 1,
                      "foo": "bar"
                    }
                  }
                ]
              }
            }
          }
        ]
      }
    }
  }
}

请注意您的 EDIT 2

Elasticsearch 将根据您的报告日期搜索所有文档，而不是“TOP 1”文档。按报告拆分结果的一种方法是在报告日期使用术语桶：

{
  "query": {},
  "size": 0,
  "aggregations": {
    "grp_report": {
      "terms": {
        "field": "generation_date"
      },
      "aggregations": {
        "nested_customers": {
          "nested": {
            "path": "customers"
          },
          "aggregations": {
            "filters_customer": {
              "filters": {
                "filters": [
                  {
                    "match": {
                      "customers.customer_id": "1"
                    }
                  }
                ]
              },
              "aggregations": {
                "top_hits_customer": {
                  "top_hits": {}
                }
              }
            }
          }
        }
      }
    }
  }
}

建议：

避免复杂的文档，更喜欢将您的报告拆分为具有相关键（例如reportId）的小文档。您将能够轻松过滤和聚合，而无需任何嵌套文档。添加关于女巫的客户文档信息，您将过滤所有类型（在这种情况下，冗余不是问题）。

用例示例：

  • 报告列表
  • 显示每个报告的客户信息
  • 跨多个报告显示客户的历史记录

当前文档示例：/indexName/market_audit

{
  "generation_date": "...",
  "customers": [
    {
      "id": 1,
      "foo": "bar 1"
    },
    {
      "id": 2,
      "foo": "bar 2"
    },
    {
      "id": 3,
      "foo": "bar 3"
    }
  ]
}

改版文件：

/indexName/market_audit_report

{
  "report_id" : "123456"
  "generation_date": "...",
  "foo":"bar"
}

/indexName/market_audit_customer 文件

{
  "report_id" : "123456"
  "customer_id": 1,
  "foo": "bar 1"
}


{
  "report_id" : "123456"
  "customer_id": 2,
  "foo": "bar 2"
}


{
  "report_id" : "123456"
  "customer_id": 3,
  "foo": "bar 3"
}

如果您知道您的报告 ID，您将能够在一个请求中获取所有数据：

  • 报告 ID 过滤器
  • 类型的术语聚合
    • 类型报告的过滤器
      • 获取报告的 top_hit 聚合
    • 过滤器聚合，仅获取类型客户和客户 ID 1
      • 客户 1 信息的 top_hit 聚合

或者

  • 报告 ID 过滤器
  • 类型的术语聚合
    • 类型报告的过滤器
      • 获取报告的 top_hit 聚合
    • 客户 ID 的术语聚合
      • 一个 top_hit 聚合来检索每个客户的信息

热门点击聚合大小

不要忘记size在你的 top_hits 中提供一个，否则你只会得到前 3 个

于 2017-02-27T11:39:11.593 回答
0

阅读 elasticsearch 第一行聚合定义我认为您不太了解它的工作原理：

聚合框架有助于提供基于搜索查询的聚合数据

由于您的查询根本没有任何过滤器，因此返回对象中所有存储的文档是hits.hits预期的结果。然后您使用filter聚合来帮助您获取所需的文档，但它们属于aggs返回的属性dict
如果我是对的，我建议您尽可能简单，所以这是我猜测的查询 

{
  "query": {
    "filtered": {
        "filter": {
            "nested": {
                "path" : "customers",
                "filter": {
                    "bool": {
                        "must" : [
                            "term": {"customer_id" : "1"}
                        ]
                    }
                }
            }
        }
    }
  },
  "aggregations": {
    "tophits_agg": {
      "top_hits": {}
    }
  }
}
于 2017-02-27T12:57:23.183 回答