1

有人可以告诉我我做错了什么，或者为什么 Logstash 不想解析 ISO8601 时间戳？

我得到的错误信息是

操作失败...“error”=>{“type”=>“mapper_parsing_exception”，“reason”=>“无法解析 [timestamp]”，“caused_by”=>{“type”=>“illegal_argument_exception”，“ reason"=>"Invalid format: \"2017-03-24 12:14:50\" 格式错误在 \"17-03-24 12:14:50\""}}

示例日志文件行（IP 地址中的最后一个字节故意替换为 000）

2017-03-24 12:14:50 87.123.123.000 12345678.domain.com GET /smil:stream_17.smil/chunk_ctvideo_ridp0va0r600115_cs211711500_mpd.m4s - HTTP/1.1 200 750584 0.714 "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36" https://referrer.domain.com/video/2107 https fra1 "HIT, MISS" 12345678.domain.com

GROK 模式（使用http://grokconstructor.appspot.com/do/match进行验证）

RAW %{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{IPV4:clientip}%{SPACE}%{HOSTNAME:http_host}%{SPACE}%{WORD:verb}%{SPACE}\/(.*:)?%{WORD:stream}%{NOTSPACE}%{SPACE}%{NOTSPACE}%{SPACE}%{WORD:protocol}\/%{NUMBER:httpversion}%{SPACE}%{NUMBER:response}%{SPACE}%{NUMBER:bytes}%{SPACE}%{SECOND:request_time}%{SPACE}%{QUOTEDSTRING:agent}%{SPACE}%{URI:referrer}%{SPACE}%{WORD}%{SPACE}%{WORD:location}%{SPACE}%{QUOTEDSTRING:cache_status}%{SPACE}%{WORD:account}%{GREEDYDATA}

Logstash 配置（输入端）：

input {
    file {
      path => "/subfolder/logs/*"
      type => "access_logs"
      start_position => "beginning"
    }
}
filter {
    # skip first two lines in log file with comments
    if [message] =~ /^#/ {
        drop { }
    }

    grok {
        patterns_dir => ["/opt/logstash/patterns"]
        match => { "message" => "%{RAW}" }
    }

    date {
        match => [ "timestamp" , "yyyy-MM-dd HH:mm:ss" ]
        locale => "en"
    }

    # ... (rest of the config omitted for readability)
}
4

1 回答 1

1

所以我很确定这是由于字段timestamp被映射到 Elasticsearch 中它不解析的类型引起的。如果您发布索引映射，我很乐意查看它。

注意：您可以通过添加快速解决此问题，remove_field因为如果date过滤器成功，该字段的值将被拉入@timestamp. 现在，您在两个字段中存储了相同的值。然后您不必担心该字段的映射。:)

date {
    match => [ "timestamp" , "yyyy-MM-dd HH:mm:ss" ]
    locale => "en"
    remove_field => [ "timestamp" ]
}
于 2017-03-27T17:17:05.263 回答