3

我已经安装了带有 elasticcsearch-0.20.6 的 logstash 1.1.13，logstash.conf 的以下配置

input {
tcp {
port => 524
type => rsyslog
}
udp {
port => 524
type => rsyslog
}
}
filter {
grok {
type => "rsyslog"
pattern => [ "<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{PROG:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" ]
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{@source_host}" ]
}
syslog_pri {
type => "rsyslog"
}
date {
type => "rsyslog"
syslog_timestamp => [ "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
mutate {
type => "rsyslog"
exclude_tags => "_grokparsefailure"
replace => [ "@source_host", "%{syslog_hostname}" ]
replace => [ "@message", "%{syslog_message}" ]
}
mutate {
type => "rsyslog"
remove => [ "syslog_hostname", "syslog_message", "syslog_timestamp" ]
}
}

output {
elasticsearch {
 host => "127.0.0.1"
 port => 9300
 node_name => "sysloG33r-1"
 bind_host => "localhost"
 }
}

和

弹性搜索.yml

cluster: 
    name: syslogcluster
node:
    name: "sysloG33r-1"
path:
    data: /var/lib/elasticsearch
path:
    logs: /var/log/elasticsearch
network:
    host: "0.0.0.0"

并使用命令启动logstash

    [root@clane elasticsearch]# java -jar /usr/local/bin/logstash/bin/logstash.jar agent -f /etc/logstash/logstash.conf
Using experimental plugin 'syslog_pri'. This plugin is untested and may change in the future. For more information about plugin statuses, see http://logstash.net/docs/1.1.13/plugin-status  {:level=>:warn}
date: You used a deprecated setting 'syslog_timestamp => ["MMM d HH:mm:ss", "MMM dd HH:mm:ss"]'. You should use 'match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]' {:level=>:warn}
PORT SETTINGS 127.0.0.1:9300
log4j, [2013-06-21T14:40:08.013]  WARN: org.elasticsearch.discovery: [sysloG33r-1] waited for 30s and no initial state was set by the discovery
Failed to index an event, will retry {:exception=>org.elasticsearch.discovery.MasterNotDiscoveredException: waited for [1m], :event=>{"@source"=>"tcp://10.66.59.35:34662/", "@tags"=>[], "@fields"=>{"syslog_pri"=>["78"], "syslog_program"=>["crond"], "syslog_pid"=>["6511"], "received_at"=>["2013-06-21T13:40:01.845Z"], "received_from"=>["10.66.59.35"], "syslog_severity_code"=>6, "syslog_facility_code"=>9, "syslog_facility"=>"clock", "syslog_severity"=>"informational"}, "@timestamp"=>"2013-06-21T12:40:01.000Z", "@source_host"=>"kent", "@source_path"=>"/", "@message"=>"(root) CMD (/opt/bin/firewall-state.sh)", "@type"=>"rsyslog"}, :level=>:warn}

和弹性搜索

/usr/local/bin/elasticsearch start

我可以看到 elasticsearch(9200,9300) 和 logstash(524) 的所有正确 Java 端口

tcp        0      0 :::524                      :::*                        LISTEN      12557/java          
tcp        0      0 :::9200                     :::*                        LISTEN      10782/java          
tcp        0      0 :::9300                     :::*                        LISTEN      10782/java          
tcp        0      0 ::ffff:127.0.0.1:9301       :::*                        LISTEN      12557/java          
udp        0      0 :::524                      :::*                                    12557/java          
udp        0      0 :::54328                    :::*                                    10782/java

但是我在logstash上看到这个错误，有什么想法吗？

Failed to index an event, will retry {:exception=>org.elasticsearch.discovery.MasterNotDiscoveredException: waited for [1m], :event=>{"@source"=>"tcp://10.66.59.35:33598/", "@tags"=>[], "@fields"=>{"syslog_pri"=>["78"], "syslog_program"=>["crond"], "syslog_pid"=>["12983"], "received_at"=>["2013-06-21T12:07:01.541Z"], "received_from"=>["10.66.59.35"], "syslog_severity_code"=>6, "syslog_facility_code"=>9, "syslog_facility"=>"clock", "syslog_severity"=>"informational"}, "@timestamp"=>"2013-06-21T11:07:01.000Z", "@source_host"=>"kent", "@source_path"=>"/", "@message"=>"(root) CMD (/opt/bin/firewall-state.sh)", "@type"=>"rsyslog"}, :level=>:warn}
4

3 回答 3

8

我假设您已经检查了明显的事情，例如“ElasticSearch 是否正在运行？” 和“我可以在 localhost 上打开到端口 9300 的 TCP 连接吗？”

即使您host在输出中使用了参数，但elasticsearch可能发生的情况是 Logstash 中的 ElasticSearch 客户端正在尝试通过多播（这是新安装通常默认配置的方式）发现集群成员，并且失败了。这在 EC2 以及防火墙配置可能会干扰多播发现的许多其他环境中很常见。如果这是您集群中的唯一成员，则在您的集群中设置以下内容elasticsearch.yml应该可以解决问题：

discovery:
  zen:
    ping:
      multicast:
        enabled: false
      unicast:
        hosts: <your_ip>[9300-9400]

在 AWS 上，还有一个 EC2 发现插件可以为您解决这个问题。

顺便说一句，这个问题确实属于服务器故障而不是堆栈溢出。

于 2013-07-11T02:59:08.367 回答
1

我有一个类似的问题，它来自我的 ip 配置。简而言之，检查您在 logstash 主机上是否只有一个 IP 地址。如果没有，它可以选择错误的。

在这里发布了相同的答案：Logstash with Elasticsearch

于 2013-12-20T14:31:50.767 回答
0

我遇到了同样的问题，并通过在 logstash 的 elasticsearch conf 中添加集群选项来解决。由于您在 elasticsearch.yml 中修改了集群名称，logstash 客户端将无法使用默认值找到集群。

也尝试这样做

于 2015-03-23T09:22:25.943 回答