尝试将多个 AWS 帐户的 cloudtrail 日志从 s3 获取到 elasticsearch 中,直到现在一切都停止了。错误显示如下所示
[2018-10-16T21:33:42,096][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError", :will_retry_in_seconds=>8}
[2018-10-16T21:33:44,406][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://vpc-sec-dummytext.eu-west-1.es.amazonaws.com:443/, :path=>"/"}
[2018-10-16T21:33:44,430][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"https://vpc-sec-dummytext.eu-west-1.es.amazonaws.com:443/"}
[2018-10-16T21:33:51,426][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff {:code=>413, :url=>"https://vpc-sec-dummytext.eu-west-1.es.amazonaws.com:443/_bulk"}
这也是我的 logstash 配置,因为我正在使用 logstash 进行摄取
```
input {
s3 {
bucket => "dummy-s3"
region => "eu-west-1"
type => "cloudtrail"
sincedb_path => "/tmp/logstash/cloudtrail"
exclude_pattern => "/CloudTrail-Digest/"
interval => 120
codec => "json"
}
}
filter {
if [type] == "cloudtrail" {
json {
source => "message"
}
split {
field => "Records"
add_tag => "splitted"
}
if ("splitted" in [tags]) {
date {
match => ["eventTime", "ISO8601"]
remove_tag => ["splitted"]
remove_field => ["timestamp"]
}
}
geoip {
source => "[Records][sourceIPAddress]"
target => "geoip"
add_tag => ["cloudtrail-geoip"]
}
mutate {
gsub => [
"eventSource", "\.amazonaws\.com$", "",
"apiVersion", "_", "-"
]
}
}
}
output {
elasticsearch {
hosts => ["vpc-sec-dummytext.eu-west-1.es.amazonaws.com:443"]
ssl => true
index => "cloudtrail-%{+YYYY.MM.dd}"
doc_as_upsert => true
template_overwrite => true
}
stdout {
codec => rubydebug
}
}
}
当 log-stash 从 ubuntu ec2 启动或重新启动时,日志被摄取几分钟然后停止
任何帮助将不胜感激。