0

我正在尝试使用 grok 过滤器在结构上过滤我的日志logstash。

这是一个示例日志：

5d563f04-b5d8-4b8d-b3ac-df26028c3719 SoapRequest CheckUserPassword <?xml version=\"1.0\" encoding=\"utf-8\"?><soap:Envelope xmlns:soap=\"http://schemas.xmlsoap.org/soap/envelope/\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\"><soap:Body><CheckUserPassword xmlns=\"http://users.tvinci.com/\"><sWSUserName>users_199</sWSUserName><sWSPassword>11111</sWSPassword><sUserName>test</sUserName><sPassword>123456</sPassword><bPreventDoubleLogins>false</bPreventDoubleLogins></CheckUserPassword></soap:Body></soap:Envelope>

这是我的过滤器 grok 匹配模式：

%{DATA:method_id} %{WORD:method_type} %{WORD:method} %{GREEDYDATA:data}

我收到的结构是：

  "method_id" => "963ad634-92d6-4a6c-9e6b-ef57e6bcd374",
      "method_type" => "SoapRequest",
      "method" => "CheckUserPassword",
      "data" => " <?xml version=\"1.0\" encoding=\"utf-8\"?><soap:Envelope xmlns:soap=\"http://schemas.xmlsoap.org/soap/envelope/\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\"><soap:Body><CheckUserPassword"

除了数据字段之外，这是正确的结构，在这里我希望看到整个 SOAP XML（如您所见，它在中间被剪掉了）

有什么建议么？

4

3 回答 3

2

使用以下过滤器： 

mutate {
  gsub => [

    "message", "\n", "",  # Unix newline

    "message", "\r", "",  # OS X newline

    "message", "\r\n", "" # Windows newline

  ]
}
于 2014-09-30T01:19:52.900 回答
0

实际上，您的模式应该可以正常工作。您的配置中一定有其他问题。试试下面：

logstash.config

input {
  stdin {
  }
}

filter {
  grok {
    match => [ "message", "%{DATA:method_id} %{WORD:method_type} %{WORD:method} %{GREEDYDATA:data}" ]
  } 
}

output {
  stdout { debug => true }
} 

$ java -jar logstash-1.2.1-flatjar.jar agent -f logstash.conf
5d563f04-b5d8-4b8d-b3ac-df26028c3719 SoapRequest CheckUserPassword <?xml version="1.0" encoding="utf-8"?><soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"><soap:Body><CheckUserPassword xmlns="http://users.tvinci.com/"><sWSUserName>users_199</sWSUserName><sWSPassword>11111</sWSPassword><sUserName>test</sUserName><sPassword>123456</sPassword><bPreventDoubleLogins>false</bPreventDoubleLogins></CheckUserPassword></soap:Body></soap:Envelope>
{
        "message" => "5d563f04-b5d8-4b8d-b3ac-df26028c3719 SoapRequest CheckUserPassword <?xml version=\"1.0\" encoding=\"utf-8\"?><soap:Envelope xmlns:soap=\"http://schemas.xmlsoap.org/soap/envelope/\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\"><soap:Body><CheckUserPassword xmlns=\"http://users.tvinci.com/\"><sWSUserName>users_199</sWSUserName><sWSPassword>11111</sWSPassword><sUserName>test</sUserName><sPassword>123456</sPassword><bPreventDoubleLogins>false</bPreventDoubleLogins></CheckUserPassword></soap:Body></soap:Envelope>\r",
     "@timestamp" => "2013-10-26T04:01:19.386Z",
       "@version" => "1",
           "host" => "NYCL530",
      "method_id" => "5d563f04-b5d8-4b8d-b3ac-df26028c3719",
    "method_type" => "SoapRequest",
         "method" => "CheckUserPassword",
           "data" => "<?xml version=\"1.0\" encoding=\"utf-8\"?><soap:Envelope xmlns:soap=\"http://schemas.xmlsoap.org/soap/envelope/\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\"><soap:Body><CheckUserPassword xmlns=\"http://users.tvinci.com/\"><sWSUserName>users_199</sWSUserName><sWSPassword>11111</sWSPassword><sUserName>test</sUserName><sPassword>123456</sPassword><bPreventDoubleLogins>false</bPreventDoubleLogins></CheckUserPassword></soap:Body></soap:Envelope>\r"
}
于 2013-10-26T04:09:45.490 回答
0

如果您遇到另一张海报建议的换行问题

从我在互联网上看到的一篇文章看来，使用输入文件字段上的多行编解码器选项也解决了这个问题。

在下面的示例中，每次我在一行的开头看到 TIMESTAMP_ISO8601 时，我都声明它是一个新的“条目”，并且直到行开头的下一个时间戳的所有内容都是该条目的一部分。

input {
  file {
    path => "/var/elasticsearch-input/Log.log"
    type => "log4netLog"
    codec => multiline {
      pattern => "^%{TIMESTAMP_ISO8601} "
      negate => true
      what => previous
    }
  }
}

在您的情况下，您需要为 GUID 编写正则表达式并将其作为模式放入其中。它可能看起来像这样，但我不是 100% 确定，因为我还没有测试过。

input {
  file {
    path => "/var/elasticsearch-input/Log.log"
    type => "log4netLog"
    codec => multiline {
      pattern => "^%{UUID} "
      negate => true
      what => previous
    }
  }
}

文档：https ://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html 我从中提取的文章（在配置 logstash - intputs 下）：http://www.ben-morris .com/using-logstash-elasticsearch-and-log4net-for-centralized-logging-in-windows/

于 2017-03-07T18:07:05.640 回答