2

我使用 helm 安装了 fluentbit，Fluent-bit Verison 是 1.13.11，fluentbit pod 运行良好，但仍然无法向 Amazon ES 发送数据，以下是错误和 yamls 文件。

请提供任何可以帮助我轻松安装的 URL。

错误：- 出现两种错误：-

1st -
[2020/10/12 12:05:06] [error] [out_es] could not pack/validate JSON response
{"took":0,"errors":true,"items":[{"index":{"_index":"log-test-2020.10.12","_type":"flb_type","_id":null,"status":400,"error":{"type":"validation_exception","reason":"Validation Failed: 1: this action would add [10] total shards, but this cluster currently has [991]/[1000] maximum shards open;"}}},{"index":{"_index":"log-test-2020.10.12","_type":"flb_type","_id":null,"status":400,"error":{"type":"validation_exception","reason":"Validation Failed: 1: this action would add [10] total shards, but this cluster currently has [991]/[1000] maximum shards open;"}}},{"index":{"_index":"log-test-2020.10.12","_type":"flb_type","_id":null,"status":400,"error"{"type":"validat```

2nd :- 
[2020/10/12 12:05:06] [ warn] [engine] failed to flush chunk '1-1602504304.544264456.flb', retry in 6 seconds: task_id=23, input=tail.0 > output=es.0
[2020/10/12 12:05:06] [ warn] [engine] failed to flush chunk '1-1602504304.79518090.flb', retry in 10 seconds: task_id=21, input=tail.0 > output=es.0
[2020/10/12 12:05:07] [ warn] [engine] failed to flush chunk '1-1602504295.264072662.flb', retry in 81 seconds: task_id=8, input=tail.0 > out```




fluentbit config file :- 
[INPUT]
    Name              tail
    Tag               kube.*
    Path              /var/log/containers/*.log
    Parser            docker
    DB                /var/log/flb_kube.db
    Mem_Buf_Limit     30MB
    Skip_Long_Lines   On
    Refresh_Interval  10
[OUTPUT]
    Name            es
    Match           *
    Host            ${FLUENT_ELASTICSEARCH_HOST}
    Port            ${FLUENT_ELASTICSEARCH_PORT}
    Logstash_Format On
    Logstash_Prefix log-test
    Time_Key        @timestamp
    tls             On
    Retry_Limit     False

customParsers: |

[PARSER]
    Name   apache
    Format regex
    Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
    Time_Key time
    Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
    Name   apache2
    Format regex
    Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
    Time_Key time
    Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
    Name   apache_error
    Format regex
    Regex  ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$
[PARSER]

[PARSER]
    Name   json
    Format json
    Time_Key time
    Time_Format %d/%b/%Y:%H:%M:%S %z

[PARSER]
    Name        docker
    Format      json
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%L
    Time_Keep   On
4

2 回答 2

0

更改 OUTPUT: Retry_Limit 10 或更低，并与 INTPUT: Buffer_Max_Size 平衡，这应该有助于保持缓冲区充满要重试的项目

于 2020-10-13T02:30:01.333 回答
0

您必须增加 kibana 中的碎片数，因为它清楚地说明了错误日志中打开的最大碎片数：-

validation_exception","re​​ason":"验证失败：1：此操作将添加 [10] 个总分片，但此集群当前有 [991]/[1000] 个最大分片打开；"}

在 kibana 开发工具 UI 中使用下面的 cmd 来增加分片数量：-

PUT /_cluster/settings {“持久”：{“cluster.max_shards_per_node”：}}

于 2021-01-21T09:40:35.853 回答