30

我有一台远程机器,它结合了多行事件并通过伐木工人协议发送它们。

进来的是看起来像这样的东西:

{
     "message" => "2014-10-20T20:52:56.133+0000 host 2014-10-20 15:52:56,036 [ERROR   ][app.logic     ] Failed to turn message into JSON\nTraceback (most recent call last):\n  File \"somefile.py", line 249, in _get_values\n    return r.json()\n  File \"/path/to/env/lib/python3.4/site-packages/requests/models.py\", line 793, in json\n    return json.loads(self.text, **kwargs)\n  File \"/usr/local/lib/python3.4/json/__init__.py\", line 318, in loads\n    return _default_decoder.decode(s)\n  File \"/usr/local/lib/python3.4/json/decoder.py\", line 343, in decode\n    obj, end = self.raw_decode(s, idx=_w(s, 0).end())\n  File \"/usr/local/lib/python3.4/json/decoder.py\", line 361, in raw_decode\n    raise ValueError(errmsg(\"Expecting value\", s, err.value)) from None\nValueError: Expecting value: line 1 column 1 (char 0), Failed to turn message into JSON"
}

当我尝试将消息与

grok {         
    match => [ "message", "%{TIMESTAMP_ISO8601:timestamp} \[%LOGLEVEL:loglevel}%{    SPACE}\]\[%{NOTSPACE:module}%{SPACE}\]%{GREEDYDATA:message}" ]
}

GREEDYDATA并不像我想的那么贪婪。

所以然后我尝试使用gsub:

mutate {
    gsub => ["message", "\n", "LINE_BREAK"]
}
# Grok goes here
mutate {
    gsub => ["message", "LINE_BREAK", "\n"]
}

但那个没有工作,而不是

The Quick brown fox
jumps over the lazy
groks

我有

The Quick brown fox\njumps over the lazy\ngroks

所以...

如何将换行符添加回我的数据,使GREEDYDATA匹配成为我的换行符,或者以其他方式获取我的消息的相关部分?

4

3 回答 3

83

一切GREEDYDATA都是 is .*,但.不匹配换行符,因此您可以替换%{GREEDYDATA:message}(?<message>(.|\r|\n)*)并使其真正贪婪。

于 2014-10-20T21:33:42.067 回答
23

将正则表达式标志添加到开头允许匹配换行符:

match => [ "message", "(?m)%{TIMESTA...
于 2014-10-20T21:45:45.390 回答
1

我使用 (?m) 和 [^\n]+ 对 Vertica 日志的最终探索

match => ["message","(?m)%{TIMESTAMP_ISO8601:ClientTimestamp}%{SPACE}(%{DATA:Action}:)?(%{DATA:ThreadID} )?(\[%{DATA:Module}\] )?(\<%{DATA:Level}\> )?(\[%{DATA:SubAction}\] )?(@%{DATA:Nodename}:)?( (?<Session>(\{.*?\} )?.*?/.*?): )?(?<message>[^\n]+)((\n)?(\t)?(?<StackTrace>[^\n]+))?"]

感谢 asperla

https://github.com/elastic/logstash/issues/2282

于 2020-04-30T10:13:17.377 回答