4

我有一个流利的 + 弹性搜索 + kibana 设置。Fluentd 配置为使用此插件输出到弹性搜索： https ://github.com/uken/fluent-plugin-elasticsearch 。问题是 fluentd 在运行时永远不会将其缓冲区刷新到弹性搜索，它只是将数据存储在内存缓冲区中。发出关闭后，缓冲区被刷新并更新弹性搜索。

尽管设置flush_interval 60s 和 ，仍会出现此问题buffer_chunk_limit 25k。默认值也会产生同样的问题。

Fluentd 收集器日志文件输出和 cfg：

    2014-07-11 11:45:44 +0100 [info]: fluent/supervisor.rb:202:supervise: starting fluentd-0.10.48
2014-07-11 11:45:44 +0100 [info]: fluent/supervisor.rb:304:read_config: reading config file path="/etc/fluent/fluent.conf"
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered buffer plugin 'file'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered buffer plugin 'memory'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered input plugin 'debug_agent'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered input plugin 'exec'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered input plugin 'forward'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered input plugin 'gc_stat'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered input plugin 'http'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered input plugin 'monitor_agent'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered input plugin 'object_space'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered input plugin 'status'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered input plugin 'tcp'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered input plugin 'unix'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered input plugin 'syslog'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered input plugin 'tail'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered output plugin 'copy'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered output plugin 'exec'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered output plugin 'exec_filter'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered output plugin 'file'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered output plugin 'forward'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered output plugin 'null'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered output plugin 'roundrobin'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered output plugin 'stdout'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered output plugin 'tcp'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered output plugin 'unix'
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered output plugin 'test'
2014-07-11 11:45:44 +0100 [info]: fluent/engine.rb:87:block in configure: gem 'fluent-plugin-elasticsearch' version '0.3.0'
2014-07-11 11:45:44 +0100 [info]: fluent/engine.rb:87:block in configure: gem 'fluentd' version '0.10.48'
2014-07-11 11:45:44 +0100 [info]: fluent/engine.rb:91:configure: using configuration file: <ROOT>
  <source>
    type forward
    port 24224
    bind 0.0.0.0
  </source>
  <source>
    type syslog
    port 5544
    protocol_type tcp
    bind 0.0.0.0
    tag index.syslog
  </source>
  <match index.**>
    type elasticsearch
    logstash_format true
    host localhost
    port 9200
    include_tag_key true
    tag_key tag
  </match>
</ROOT>
2014-07-11 11:45:44 +0100 [info]: fluent/engine.rb:101:block in configure: adding source type="forward"
2014-07-11 11:45:44 +0100 [info]: fluent/engine.rb:101:block in configure: adding source type="syslog"
2014-07-11 11:45:44 +0100 [info]: fluent/engine.rb:117:block in configure: adding match pattern="index.**" type="elasticsearch"
2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered output plugin 'elasticsearch'
2014-07-11 11:45:44 +0100 [info]: plugin/in_forward.rb:75:listen: listening fluent socket on 0.0.0.0:24224
2014-07-11 11:45:45 +0100 [debug]: plugin/in_syslog.rb:183:listen: listening syslog socket on 0.0.0.0:5544 with tcp
2014-07-11 11:45:45 +0100 [trace]: plugin/in_forward.rb:189:initialize: accepted fluent socket from '10.18.80.20:57119': object_id=70254255494700
2014-07-11 11:45:45 +0100 [trace]: plugin/in_forward.rb:245:on_close: closed fluent socket object_id=70254255494700
2014-07-11 11:45:46 +0100 [trace]: plugin/in_forward.rb:189:initialize: accepted fluent socket from '10.18.80.20:57120': object_id=70254256323480
2014-07-11 11:45:46 +0100 [trace]: plugin/in_forward.rb:245:on_close: closed fluent socket object_id=70254256323480
2014-07-11 11:45:47 +0100 [trace]: plugin/in_forward.rb:189:initialize: accepted fluent socket from '10.18.80.20:57121': object_id=70254256319280
2014-07-11 11:45:47 +0100 [trace]: plugin/in_forward.rb:245:on_close: closed fluent socket object_id=70254256319280
.......

谁能帮助解释为什么 fluentd 没有刷新到弹性搜索？谢谢。

4

3 回答 3

2

经过多次故障排除后，我发现安装的特定版本的 ruby​​ 中存在一个错误，导致启动时线程死锁。关闭时未出现此问题，因为它被称为不同。这是重新 ruby​​ 2.0.0-p353 的特定错误；下一个版本 ruby​​ 2.0.0-p451 解决了这个问题。

https://groups.google.com/forum/#!msg/fluentd/t2uKfttiYCw/P8zy5kZpEIIJ

于 2014-09-08T14:52:47.373 回答
1

/var/spool/fluentd 下是否有 fluentd 缓冲区？他们长大了吗？

你能检查一下 ElasticSearch 日志吗？

grep --color FrameException /var/log/elasticsearch/elasticsearch.log
org.elasticsearch.common.netty.handler.codec.frame.TooLongFrameException: HTTP content length exceeded 655360000 bytes.
于 2014-09-05T19:15:17.513 回答
0

查看您粘贴的标准输出，它看起来不像为 out_elasticsearch 插件设置了“flush_interval”：

<match index.**>
  type elasticsearch
  logstash_format true
  host localhost
  port 9200
  include_tag_key true
  tag_key tag
</match>

您可以尝试在上述配置中插入一行“flush_interval 60s”，等待一分钟左右，然后查看 Elasticsearch 中是否显示数据？

于 2014-07-11T22:16:50.260 回答