syslog - Remove duplicate remote syslog messages

Question

For redundancy, every host in our distributed network sends its syslog messages to two dedicated rsyslog-nodes. These in turn send syslogs to a central graylog instance:

    / rsyslog \
host           --> graylog  
    \ rsyslog /

Now every log-message gets duplicated!

Question: How can we keep the redundancy but remove duplicates? Does fluentd have a way to deal with this? Or any other opensource software designed to aggregate log-messages? We do not want to include much more complexity to the whole setup, but inserting one additional component is fine.

Answer 1

AKAIK，没有内置重复数据删除消息的开源软件，因为这些要求是特定于应用程序的。我们可以用主节点实现这样的消息处理，但它存在性能/可扩展性问题。

我们有几种方法来放松问题。

• 添加唯一 id 以记录并在查询中区分。我不知道如何在graylog中区分...
• 一个流用于实时，另一个流用于备份。一个灰色日志中没有合并流
• 在 fluentd 上，您可以使用 fluent-plugin-suppress( https://github.com/fujiwara/fluent-plugin-suppress ) 或 fluent-plugin-dedup( https://github.com/edvakf/fluent-plugin-dedup ) 对于这种情况。