0

如何更新 logstash 或 elasticsearch 模板，以便避免在 logstash 服务器上显示此 mapper_parsing_exception 错误。尝试 elk 进行 aws cloudtrail 日志记录，但似乎每一步都遇到障碍。

我也明白用 grok 完成但不知道怎么做！

[2018-10-06T00:28:19,948][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"cloudtrail-2018.10.06", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x73d7cfa2>], :response=>{"index"=>{"_index"=>"cloudtrail-2018.10.06", "_type"=>"doc", "_id"=>"pIPGRmYBZjQ3shBzZDlQ", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [Records.requestParameters.ebsOptimized] tried to parse field [ebsOptimized] as object, but found a concrete value"}}}}

第二个错误

2018-10-06T00:28:33,047][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"cloudtrail-2018.10.06", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x7c5a7518>], :response=>{"index"=>{"_index"=>"cloudtrail-2018.10.06", "_type"=>"doc", "_id"=>"JIPGRmYBZjQ3shBzmTqd", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [Records.apiVersion]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: \"2017_03_25\" is malformed at \"_03_25\

这是我的模板和默认动态映射

{
  "cloudtrail": {
    "order": 0,
    "version": 60001,
    "index_patterns": [
      "cloudtrail-*"
    ],
    "settings": {
      "index": {
        "mapping": {
          "total_fields": {
            "limit": "6000"
          }
        },
        "refresh_interval": "5s"
      }
    },
    "mappings": {
      "_default_": {
        "dynamic_templates": [
          {
            "message_field": {
              "path_match": "message",
              "match_mapping_type": "string",
              "mapping": {
                "type": "text",
                "norms": false
              }
            }
          },
          {
            "string_fields": {
              "match": "*",
              "match_mapping_type": "string",
              "mapping": {
                "type": "text",
                "norms": false,
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }
            }
          }
        ],
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "@version": {
            "type": "keyword"
          },
          "geoip": {
            "dynamic": true,
            "properties": {
              "ip": {
                "type": "ip"
              },
              "location": {
                "type": "geo_point"
              },
              "latitude": {
                "type": "half_float"
              },
              "longitude": {
                "type": "half_float"
              }
            }
          }
        }
      }
    },
    "aliases": {}
  }
}

映射，不知道为什么这么大

"apiVersion": {
                "type": "date"
              },
              "awsRegion": {
                "type": "text",
                "norms": false,
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              }

更新 1：

@Amir masud zarebidaki，谢谢。我认为 elasticsearch 会动态地进行所有映射并进行优化，这对于 elasticsearch 概念来说还是很新的。但是，我确实通过在我的过滤器中添加以下内容来消除他的日期格式错误

mutate {
             gsub => [              "eventSource", "\.amazonaws\.com$", "",
              "apiVersion", "_", "-"        ]

但不确定这样做是否正确，因为我现在丢失了一些数据，并且我在日志中出现了新的错误，形式为

[2018-10-08T13:26:16,000][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"cloudtrail-2018.10.08", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x30ef00f4>], :response=>{"index"=>{"_index"=>"cloudtrail-2018.10.08", "_type"=>"doc", "_id"=>"aknbU2YBeGi09oGfWDua", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [responseElements.role]", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:230"}}}}}
4

0 回答 0