2

我正在尝试设置 logstash 来解析日志并将其发送到另一台服务器。几乎每次 GROK 解析日志失败时,都会导致以下错误:

{:timestamp=>"2013-07-08T02:20:07.390000-0400",
:message=>"thread watchdog timeout",
:thread=>#<Thread:0x133b92c run>,
:backtrace=>["file:/opt/logstash/logstash.jar!/logstash/filterworker.rb:46:in `backtrace'",
"file:/opt/logstash/logstash.jar!/logstash/filterworker.rb:46:in `run'",
"file:/opt/logstash/logstash.jar!/logstash/agent.rb:785:in `each'",
"file:/opt/logstash/logstash.jar!/logstash/agent.rb:785:in `run_filter'",
"file:/opt/logstash/logstash.jar!/logstash/agent.rb:492:in `run_with_config'"],
:thread_watchdog=>2013-07-08 02:19:57 -0400,
:age=>10.006,
:cutoff=>10,
:state=>{:event=>#<LogStash::Event:0xa02ea9 @data={"@source"=>"file://clstaging12//home/xyz/xyz.com/apps/logs/mailerjob_log_2013_07_04.txt",
"@tags"=>[],
"@fields"=>{},
"@timestamp"=>"2013-07-08T06:19:50.114Z",
"@source_host"=>"clstaging12",
"@source_path"=>"//home/xyz/xyz.com/apps/logs/mailerjob_log_2013_07_04.txt",
"@message"=>"PID:31730 2013-07-04T13:59:03-05:00 DEBUG :[property_listing_contact.php] Inside getBrokersContactInfoReceivedUserCount of the PropertyListing",
"@type"=>"zend_log"},
@cancelled=false>,
:filter=><LogStash::Filters::Grok type=>"zend_log",
patterns_dir=>["/home/xyz/xyz.com/conf/patterns"],
pattern=>["%{ZEND_LOG}"],
match=>{"@message"=>["%{ZEND_LOG}"]},
tag_on_failure=>["_grokparsefailure"]>},
:level=>:fatal}

模式 ZEND_LOG 是:

ZEND_LOG_SIGNATURE (?:IP\:(?<clientip>(?:\%ip\%|%{IP})) )?PID\:%{NUMBER:pid}
ZEND_CONTEXT [^\]]+
ZEND_LOG %{ZEND_LOG_SIGNATURE} %{TIMESTAMP_ISO8601:timestamp} %{WORD:level} \:\[%{ZEND_CONTEXT:context}\] %{GREEDYDATA:message}

这会导致 logstash 代理每隔几分钟就崩溃一次,使其几乎无法使用。我查看了许多在 JIRA 上提交的现有错误,但没有任何运气。这里有几个链接:

https://logstash.jira.com/browse/LOGSTASH-508

https://logstash.jira.com/browse/LOGSTASH-525

https://logstash.jira.com/browse/LOGSTASH-765

4

1 回答 1

2

我无法完全解决这个问题,但经过进一步调查后,我发现当 GROK 无法与整行匹配时发生错误,即它仍在等待更多输入。

这可能完全是因为我试图解析的日志是多行的,而且我使用了一个贪婪的正则表达式。我通过使我的正则表达式更加强类型来解决这个问题。

于 2013-08-26T09:55:20.320 回答