1

在 Amazon Linux 上进行 yum 更新后，Td-agent 不工作。

Td-agent 1.1.20-0 工作，但 1.1.21-0 不工作，s3 中没有日志数据。

错误信息是

2015-01-15 06:36:40 +0900 [error]: failed to configure sub output s3: Other '' plugin already use same buffer_path: type = , buffer_path = /var/log/td-agent/buffer/s3
2015-01-15 06:36:40 +0900 [error]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/buf_file.rb:94:in `configure'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/output.rb:188:in `configure'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/output.rb:471:in `configure'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-mixin-config-placeholders-0.3.0/lib/fluent/mixin/config_placeholders.rb:105:in `configure'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-s3-0.4.1/lib/fluent/plugin/out_s3.rb:47:in `configure'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-forest-0.2.4/lib/fluent/plugin/out_forest.rb:132:in `block in plant'
<internal:prelude>:10:in `synchronize'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-forest-0.2.4/lib/fluent/plugin/out_forest.rb:128:in `plant'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-forest-0.2.4/lib/fluent/plugin/out_forest.rb:168:in `emit'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/match.rb:36:in `emit'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/engine.rb:160:in `emit_stream'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:227:in `receive_lines'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:318:in `call'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:318:in `wrap_receive_lines'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:511:in `call'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:511:in `on_notify'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:343:in `on_notify'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:444:in `call'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:444:in `on_change'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.1.1/lib/cool.io/loop.rb:96:in `run_once'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.1.1/lib/cool.io/loop.rb:96:in `run'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:212:in `run'
2015-01-15 06:36:40 +0900 [error]: Cannot output messages with tag 'apache.access'

和 td-agent.conf（整个配置）如下所示：

####
## Output descriptions:
##


## match tag=debug.** and dump to console
<match debug.**>
  type stdout
</match>

#httpd logs
<match apache.access>
  type forest
  subtype s3
  <template>
    aws_key_id MY_AWS_KEY
    aws_sec_key MY_AWS_SECRET_KEY
    s3_endpoint s3.amazonaws.com
    s3_bucket my_s3_bucket_name
    path httpd/logs/
    buffer_path /var/log/td-agent/buffer/s3
    time_slice_format %Y-%m-%d/${hostname}-%H
    time_slice_wait 10m
    buffer_chunk_limit 256m
  </template>
</match>

#not matched
<match app.**>
  type stdout
</match>

####
## Source descriptions:
##

## built-in TCP input
## @see http://docs.fluentd.org/articles/in_forward
<source>
  type forward
</source>

## built-in UNIX socket input
#<source>
#  type unix
#</source>

# HTTP input
# POST http://localhost:8888/<tag>?json=<json>
# POST http://localhost:8888/td.myapp.login?json={"user"%3A"me"}
# @see http://docs.fluentd.org/articles/in_http
#<source>
#  type http
#  port 8888
#</source>

## live debugging agent
<source>
  type debug_agent
  bind 127.0.0.1
  port 24230
</source>

## File input
## read apache logs continuously and tags td.apache.access
<source>
  type tail
  #format apache
  format ltsv
  time_format [%d/%b/%Y:%H:%M:%S %z]
  time_key time
  pos_file /var/log/td-agent/apache.access.pos
  path /path/to/apache_logfile.log
  tag apache.access
</source>

请告诉我如何解决

4

2 回答 2

2

fluent-plugin-forest 版本不相关。这是时间和标签问题的数量。

Fluentd 在配置阶段检查 buffer_path 重复。但是 fluent-plugin-forest 在运行时创建插件，所以这个奇怪的重复错误发生在运行时。

请将 ${tag} 或其他占位符设置为 buffer_path。

http://docs.fluentd.org/articles/buf_file

此外，fluent-plugin-forest 现在有以下限制。所以我建议flush_at_shutdown true在你的配置中设置。

https://github.com/tagomoris/fluent-plugin-forest/issues/15

于 2015-02-27T05:42:27.917 回答
1

不幸的是，td-agent 并未正式支持 Amazon Linux。原因是 Amazon Linux 不断更改 ABI（应用程序二进制接口）并且与 CentOS/REHL 不兼容。

您可以使用此脚本在 Amazon Linux 上创建包并构建您的包。> https://github.com/treasure-data/omnibus-td-agent

于 2015-02-26T22:17:35.353 回答