logging - Fluentd copy output plugin not sending same data to both destinations?

Question

I have a web server stack with multiple nodes (auto scaling group), and each web server is configured to use Fluentd to forward log files to a central collector which saves the logs in an S3 bucket.

I am testing a new collector, and configured the web servers to send a copy of an access log to both the old and the new collector, but there is a huge discrepancy in the aggregated files.

The old collector is saving 2x the log data. I can't figure out for the life of me what the issue is, unless the data is getting lost somewhere on the way to the new collector (and how could I best test this?).

Web server td-agent.conf

<source>
  type tail
  path /var/log/some/access.log
  tag some-access.log
  format tsv
  keys time,userId,position,userAgent,platform
  pos_file /tmp/fluent.some-access.log.pos
  rotate_wait
</source>

<match some-access.log>
  type copy
  <store>
    type forward
    <server>
      host log-collector-1.lan
      port 24224
    </server>
  </store>
  <store>
    type forward
    <server>
      host log-collector-2.lan
      port 24224
    </server>
  </store>
</match>

The log collectors have the exact same config... so no need to look there.

Answer 1

我第一次听到这样的行为。我假设 log-collector-1.lan 指的是与 log-collector-2.lan 相同的服务器。

您没有粘贴聚合器端配置，所以只是猜测。如何检查“保存 2 倍的日志数据”？

您可以通过以下方式检查 fluentd 是否接收日志。

• 使用 monitor_agent 检查 S3 的缓冲区大小：http ://docs.fluentd.org/articles/monitoring#monitoring-agent
• fluent-tail 命令检查通过的事件：https ://github.com/choplin/fluent-tail
• 使用标准输出和复制插件转储实际记录。