elasticsearch - 使用 bunyan/logstash/elasticsearch 记录请求时出现 MapperParsingException

Question

我们使用 Logstash 和 Bunyan 在 ElasticSearch 中记录 Restify 请求。但是，当在日志记录中包含 JSON 正文时，索引会发生冲突，因为具有相同名称的字段有时具有不同的类型。

One example is when req.body is sometimes a string and sometimes an object. We've worked around that by always setting body to an object (since our restify API is not supposed to receive strings for any valid requests).

However the problem keeps occurring for fields in the body object. We can't really control what a client sends in, and if a request contains a string where a number is expected the elasticsearch index has already typed that field to number.

Is there any way of continually fixing this apart from checking and potentially replacing every field posted in the body? Converting the body from an object to a string before logging it would work, but that would seriously reduce its usability in Kibana when making visualizations.

Answer 1

如果字段需要同时包含字符串或数字，则必须将其定义为字符串。否则，当出现不匹配时，elasticsearch 会很乐意放弃该事件。

Elastic 计划向 logstash 添加一个“死信”功能，因此您可以将该字段保留为一个数字，并将作为字符串传递的任何事件定向到那里，而不是使用 elasticsearch。

为了防止弹性搜索根据收到的第一个数据将字段设为数字，您可能需要禁用动态映射或设置模板以更好地帮助映射。