1

我正在使用 fluentd(v1.0 td-agent3) 和 fluent-plugin-mongo。虽然设置了 time_format、time_key 和 include_time_key 参数，但 MongoDB 中的日期数据没有毫秒。如何配置以在 MongoDB 中节省毫秒？

log sender 上的 td-agent.conf 如下。我也配置了输出文件。

<source>
  @type tail
  tag esb
  <parse>
    @type regexp
    expression /^(?<logtime>[^\]]*) \| (?<category>(TRACE|DEBUG|INFO|WARN|ERROR)) (?<message>.*)$/ 
    time_key logtime
    time_type string
    time_format %Y-%m-%d %H:%M:%S,%L
    keep_time_key true
  </parse>
  path /opt/esb.log
  path_key filename
  read_from_head true
  pos_file /var/log/td-agent/tmp/esb.log.pos
</source>

<match esb>
  @type record_reformer
  tag hostname.esb
  <record>
    hostname ${hostname}
  </record>
</match>

<match hostname.**>
  @type copy
  <inject>
    time_key logtime
    time_type string
    time_format %Y-%m-%d %H:%M:%S,%L
  </inject>
  <store>
    @type file
    path /var/log/td-agent/fuse
    append true
    <buffer>
      timekey 2m
      timekey_wait 1m
      flush_at_shutdown true
    </buffer>
  </store>
  <store>
    @type forward
    <server>
      host 10.xx.xx.xx
      port 24224
    </server>
    <secondary>
      @type file
      path /var/log/td-agent/esb_bk.log
    </secondary>
  </store>
</match>

日志发送方的输出文件（部分）如下所示。Fluentd 可以在“logtime”中捕获毫秒。

2018-04-06T08:58:29+09:00   hostname.esb    {"logtime":"2018-04-06 08:58:29,391","category":"DEBUG","message":"| 9-107 - /healthy | ServletHandler                   | 94 - org.eclipse.jetty.util - 9.2.21.v20170120 | call servlet org.apache.camel.component.jetty.CamelContinuationServlet-58c526a1@b930b0cf==org.apache.camel.component.jetty.CamelContinuationServlet,-1,true","filename":"/opt/esb.log","hostname":"esb"}
2018-04-06T08:58:29+09:00   hostname.esb    {"logtime":"2018-04-06 08:58:29,391","category":"DEBUG","message":"| 9-107 - /healthy | DefaultHttpBinding               | 294 - org.apache.camel.camel-http-common - 2.17.0.redhat-630310 | Streaming response in chunked mode with buffer size 32768","filename":"/opt/esb.log","hostname":"esb"}

使用 MongoDB 的日志接收器上的 td-agent.conf 如下。

<source>
  @type forward
  port 24224
</source>

<match hostname.esb>
  @type copy
  <inject>
    time_key logtime
    time_type string
    time_format %Y-%m-%d %H:%M:%S,%L
  </inject>
  <store>
    @type file
    path /var/log/td-agent/fuse
    append true
    <buffer>
      timekey 2m
      timekey_wait 1m
      flush_at_shutdown true
    </buffer>
  </store>
  <store>
    @type mongo
    host localhost
    port 27017
    database fluentd
    collection esb
    capped
    capped_size 1024m
    include_time_key true
    time_key logtime
    <buffer>
      flush_interval 10s
    </buffer>
    user fluentd
    password password
  </store>
</match>

日志接收器上的输出文件（部分）如下所示。我猜 Fluentd 可以正确接收“logtime”。

2018-04-06T08:58:29+09:00   hostname.esb    {"logtime":"2018-04-06 08:58:29,391","category":"DEBUG","message":"| 9-107 - /healthy | ServletHandler                   | 94 - org.eclipse.jetty.util - 9.2.21.v20170120 | call servlet org.apache.camel.component.jetty.CamelContinuationServlet-58c526a1@b930b0cf==org.apache.camel.component.jetty.CamelContinuationServlet,-1,true","filename":"/opt/esb.log","hostname":"esb"}
2018-04-06T08:58:29+09:00   hostname.esb    {"logtime":"2018-04-06 08:58:29,391","category":"DEBUG","message":"| 9-107 - /healthy | DefaultHttpBinding               | 294 - org.apache.camel.camel-http-common - 2.17.0.redhat-630310 | Streaming response in chunked mode with buffer size 32768","filename":"/opt/esb.log","hostname":"esb"}

但是MongoDB的结果是这样的。毫秒（值 ,391）被删除。由于我住在 GMT+9:00 时区，MongoDB 中的“logtime”并不完全匹配。

> db.issesb.find({"hostname":"esb","logtime":ISODate("2018-04-05T23:58:29Z"), "message":/294 - org.apache.camel.came/});
{ "_id" : ObjectId("5ac6b86f4411493241af1f48"), "logtime" : ISODate("2018-04-05T23:58:29Z"), "category" : "DEBUG", "message" : "| 9-107 - /healthy | DefaultHttpBinding               | 294 - org.apache.camel.camel-http-common - 2.17.0.redhat-630310 | Streaming response in chunked mode with buffer size 32768", "filename" : "/opt/fuse/data/log/fuse.log", "hostname" : "esb" }

任何帮助表示赞赏。提前致谢。

4

0 回答 0