3

我试图通过 Fluent Bit 转发插件解决将 Java 堆栈跟踪消息合并到一个日志事件中的问题。原始日志写成带有分隔符的纯文本（不是 JSON 格式），所以我必须处理多行问题。日志由 dockerized 服务提供，例如：

10:20:52.049 [http-nio-8080-exec-10] DEBUG r.a.s.w.a.w.CustomBasicAuthenticationFilter - Basic Authentication Authorization header found for user 'ui_client'
10:20:57.905 [http-nio-8080-exec-1] WARN  r.a.s.o.provider.CustomTokenEndpoint - Handling error: InvalidGrantException, Bad credentials
    org.springframework.security.oauth2.common.exceptions.InvalidGrantException: Bad credentials
            at org.springframework.security.oauth2.provider.password.ResourceOwnerPasswordTokenGranter.getOAuth2Authentication(ResourceOwnerPasswordTokenGranter.java:79)
            at org.springframework.security.oauth2.provider.token.AbstractTokenGranter.getAccessToken(AbstractTokenGranter.java:72)
            at org.springframework.security.oauth2.provider.token.AbstractTokenGranter.grant(AbstractTokenGranter.java:67)
            at org.springframework.security.oauth2.provider.CompositeTokenGranter.grant(CompositeTokenGranter.java:38)

当我使用从文件系统读取某些服务日志的tail插件时，我得到的消息合并得很好

解析器.conf：

[PARSER]
    Name   regular_line
    Format regex
    Regex  (?<time>\d{1,2}\:\d{1,2}\:\d{1,2}\.\d{1,23}) \[(?<thread>.*)\] (?<level>\S+) (?<_>(?:\s*))(?<class>([a-zA-Z_$][a-zA-Z\d_$]*\.)*[a-zA-Z_$][a-zA-Z\d_$]*) - (?<msg>.*)

[PARSER]
    Name   stacktrace
    Format regex
    Regex  /^\s+\S.*/

流利的bit.conf

[INPUT]
    Name              tail
    path              /fluent-bit/logs/ibin/*.log
    Multiline         On
    Parser_Firstline  regular_line
    Parser_1          stacktrace

然后我将事件正确合并到记录标准输出中

[75] tail.1: [1605597502.905885300, {"time"=>"12:12:37.687", "thread"=>"http-nio-8080-exec-8", "level"=>"WARN", "_"=>" ", "class"=>"r.a.s.o.provider.CustomTokenEndpoint", "msg"=>"Handling error: InvalidGrantException, Bad credentials
        org.springframework.security.oauth2.common.exceptions.InvalidGrantException: Bad credentials
                at org.springframework.security.oauth2.provider.password.ResourceOwnerPasswordTokenGranter.getOAuth2Authentication(ResourceOwnerPasswordTokenGranter.java:79)
                at org.springframework.security.oauth2.provider.token.AbstractTokenGranter.getAccessToken(AbstractTokenGranter.java:72)
                at org.springframework.security.oauth2.provider.token.AbstractTokenGranter.grant(AbstractTokenGranter.java:67)
                at org.springframework.security.oauth2.provider.CompositeTokenGranter.grant(CompositeTokenGranter.java:38)
        ...
                at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
                at java.base/java.lang.Thread.run(Thread.java:830)"}]

[76] tail.1: [1605597502.906165800, {"time"=>"12:12:40.791", "thread"=>"http-nio-8080-exec-9", "level"=>"DEBUG", "class"=>"r.a.s.w.a.w.CustomBasicAuthenticationFilter", "msg"=>"Basic Authentication Authorization header found for user 'ui_client'"}]

但是当我尝试使用具有类似配置的转发插件来获得相同的结果时 - 我将每个堆栈跟踪行作为单独的记录

流利的bit.conf

[INPUT]
    Name   forward
    Listen 0.0.0.0
    Port   24224
    Multiline         On
    Parser_Firstline  regular_line
    Parser_1          stacktrace

标准输出

[2] auth-service: [1605598215.000000000, {"container_id"=>"9ed6bb", "container_name"=>"/auth-service", "source"=>"stdout", "log"=>"07:30:15.970 [http-nio-8080-exec-5] WARN  r.a.s.o.provider.CustomTokenEndpoint - Handling error: InvalidGrantException, Bad credentials"}]
[3] auth-service: [1605598215.000000000, {"container_id"=>"9ed6bb", "container_name"=>"/auth-service", "source"=>"stdout", "log"=>"org.springframework.security.oauth2.common.exceptions.InvalidGrantException: Bad credentials"}]
[4] auth-service: [1605598215.000000000, {"container_id"=>"9ed6bb", "container_name"=>"/auth-service", "source"=>"stdout", "log"=>"      at org.springframework.security.oauth2.provider.password.ResourceOwnerPasswordTokenGranter.getOAuth2Authentication(ResourceOwnerPasswordTokenGranter.java:79)"}]
[5] auth-service: [1605598215.000000000, {"container_name"=>"/auth-service", "source"=>"stdout", "log"=>"  at org.springframework.security.oauth2.provider.token.AbstractTokenGranter.getAccessToken(AbstractTokenGranter.java:72)", "container_id"=>"9ed6bb"}]
[6] auth-service: [1605598215.000000000, {"container_id"=>"9ed6bb", "container_name"=>"/auth-service", "source"=>"stdout", "log"=>"      at org.springframework.security.oauth2.provider.token.AbstractTokenGranter.grant(AbstractTokenGranter.java:67)"}]
[7] auth-service: [1605598215.000000000, {"source"=>"stdout", "log"=>"   at org.springframework.security.oauth2.provider.CompositeTokenGranter.grant(CompositeTokenGranter.java:38)", "container_id"=>"9ed6bb", "container_name"=>"/auth-service"}]

我猜前向插件不支持多行，尽管它没有报告此配置的任何错误/警告。

问题是：如何配置fluent bit input forward插件将多行日志消息合并为一条记录？ 这个问题有什么解决办法吗？

4

0 回答 0