0

我对logstash还是很陌生，解析出多行日志消息似乎仍然有点吓人，但我希望我正在尝试做的事情（也就是解析logstash日志）这是经过验证和测试的模式，希望有人也许可以为我指出一个可行的解决方案。

基本上我有这样的错误：

[2014-10-15 22:34:36,958][DEBUG][action.search.type       ] [kafka] [logstash-2014.10.15][1], node[RBRz8xNgQQKsAnEgbjqVTw], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@4511ca8d] lastShard [true]
org.elasticsearch.search.SearchParseException: [logstash-2014.10.15][1]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"facets":{"stats":{"statistical":{"field":"host"},"facet_filter":{"fquery":{"query":{"filtered":{"query":{"bool":{"should":[{"query_string":{"query":"status:200"}},{"query_string":{"query":"status:304"}},{"query_string":{"query":"_missing_:status"}},{"query_string":{"query":"type:\"nginx-error\""}},{"query_string":{"query":"type:(-\"nginx-*\")"}},{"query_string":{"query":"status:[308 TO 999]"}}]}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1413411577151,"to":1413412477151}}}]}}}}}}},"stats_Success":{"statistical":{"field":"host"},"facet_filter":{"fquery":{"query":{"filtered":{"query":{"bool":{"should":[{"query_string":{"query":"status:200"}}]}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1413411577151,"to":1413412477151}}}]}}}}}}},"stats_Cache Hits":{"statistical":{"field":"host"},"facet_filter":{"fquery":{"query":{"filtered":{"query":{"bool":{"should":[{"query_string":{"query":"status:304"}}]}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1413411577151,"to":1413412477151}}}]}}}}}}},"stats_No Status":{"statistical":{"field":"host"},"facet_filter":{"fquery":{"query":{"filtered":{"query":{"bool":{"should":[{"query_string":{"query":"_missing_:status"}}]}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1413411577152,"to":1413412477152}}}]}}}}}}},"stats_NGINX error":{"statistical":{"field":"host"},"facet_filter":{"fquery":{"query":{"filtered":{"query":{"bool":{"should":[{"query_string":{"query":"type:\"nginx-error\""}}]}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1413411577152,"to":1413412477152}}}]}}}}}}},"stats_Not Nginx":{"statistical":{"field":"host"},"facet_filter":{"fquery":{"query":{"filtered":{"query":{"bool":{"should":[{"query_string":{"query":"type:(-\"nginx-*\")"}}]}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1413411577152,"to":1413412477152}}}]}}}}}}},"stats_Failure":{"statistical":{"field":"host"},"facet_filter":{"fquery":{"query":{"filtered":{"query":{"bool":{"should":[{"query_string":{"query":"status:[308 TO 999]"}}]}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1413411577152,"to":1413412477152}}}]}}}}}}}},"size":0}]]
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:660)
    at org.elasticsearch.search.SearchService.createContext(SearchService.java:516)
    at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:488)
    at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:257)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:206)
    at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:203)
    at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.search.facet.FacetPhaseExecutionException: Facet [stats]: field [host] isn't a number field, but a string
    at org.elasticsearch.search.facet.statistical.StatisticalFacetParser.parse(StatisticalFacetParser.java:132)
    at org.elasticsearch.search.facet.FacetParseElement.parse(FacetParseElement.java:93)
    at org.elasticsearch.search.SearchService.parseSource(SearchService.java:644)
    ... 9 more

这似乎对我在 Kibana 仪表板中使用“主机”作为数字感到犹豫。我实际上对实际错误并不感兴趣，而是对能够以一种有效的 logstash 方式解析它更感兴趣，以便我可以查看 Kibana 并查看合理的错误消息。

我假设它会在multiline { ... }我的文件输入中使用一些编解码器，但看看那条消息，它看起来很毛茸茸......肯定有一个众所周知的“轮子”在某个地方我不需要发明吗？

4

2 回答 2

1

每个例外都以日期开头，因此请关闭它。这是一个“过滤器”示例，但它适用于编解码器：

filter {
    multiline {
      negate => 'true'
      pattern => "^\[%{YEAR}"
      what => 'previous'
    }
  }
}
于 2014-10-16T02:48:20.610 回答
1

multiline {...}是要走的路。尝试以下file类型的代码段input。

codec => multiline
{
    pattern => "(^\d+\serror)|(^.+Exception: .+)|(^\s+at .+)|(^\?s+... \d+ more)|(^\s*Caused by:.+)|(^.+ \d+ common frames omitted)
    what => "previous"
    multiline_tag => "multi_tagged"
}
于 2014-10-16T00:05:09.950 回答