logstash - Logstash 输入文件名作为输出弹性搜索索引

Question

有没有办法让logstash读取文件的文件名作为输出到ElasticSearch的索引名？

我正在为logstash使用以下配置。

input{
    file{
        path => "/logstashInput/*"
    }
}
output{
    elasticsearch{
        index => "FromfileX"
    }
}

我希望能够放置一个文件e.g. log-from-20.10.2016.log并将其编入索引log-from-20.10.2016中。logstash 输入插件“文件”是否产生任何用于过滤器或输出的变量？

Answer 1

是的，您可以使用该path字段grok并将文件名提取到该index字段中

  input {
     file {
         path => "/logstashInput/*"
     }
  }
  filter {
     grok {
        match => ["path", "(?<index>log-from-\d{2}\.\d{2}\.\d{4})\.log$" ]
     }
  }
  output{
     elasticsearch {
        index => "%{index}"
     }
  }

Answer 2

input {
    file {
        path => "/home/ubuntu/data/gunicorn.log"
        start_position => "beginning"
    }
}

filter { 
    grok {
        match => {
        "message" => "%{USERNAME:u1} %{USERNAME:u2} \[%{HTTPDATE:http_date}\] \"%{DATA:http_verb} %{URIPATHPARAM:api} %{DATA:http_version}\" %{NUMBER:status_code} %{NUMBER:byte} \"%{DATA:external_api}\" \"%{GREEDYDATA:android_client}\""
        remove_field => ["message"]
       }
    }

    date {
        match => ["http_date", "dd/MMM/yyyy:HH:mm:ss +ssss"]
    } 

    ruby {
        code => "event.set('index_name',event.get('path').split('/')[-1].gsub('.log',''))"
    } 
}
output {
    elasticsearch {
        hosts => ["0.0.0.0:9200"]
        index => "%{index_name}-%{+yyyy-MM-dd}"
        user => "*********************"
        password => "*****************"
    }

    stdout { codec => rubydebug }
}