0

我试图解析自定义日志。Loki 不接受解析的时间戳原因：

level=error ts=2020-09-18T07:17:52.655628313Z msg="final error sending batch" status=400 error="server returned HTTP status 400 Bad Request (400): entry for stream has timestamp too new: 2020-09-18 10:01:52.935 +0000 UTC"

但旧日志随时间变化，例如：2020-09-09 10:01:52.935 甚至 2020-09-17 10:01:52.935 loki 解析正常

我的日志：

2020-09-18 10:01:52.935;message

重现行为的步骤：grafana/loki:latest grafana/promtail:latest

我的 promtail 配置：

...
    scrape_configs:
  - job_name: myjob
    pipeline_stages:
    - regex:
        expression: '^(?P<timestamp>[\d-\s:.]{20,30});(?P<message>.*)'
    - timestamp:
        source: timestamp
        format: 2006-01-02 15:04:05.999
    static_configs:
    - targets:
        - localhost
      labels:
        job: myjob
        _path_: /logs/*log

我的 loki 配置：

auth_enabled: false

server:
  http_listen_port: 3100

ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 5m
  chunk_retain_period: 30s
  max_transfer_retries: 0

schema_config:
  configs:
    - from: 2018-04-15
      store: boltdb
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 792h

storage_config:
  boltdb:
    directory: /tmp/loki/index

  filesystem:
    directory: /tmp/loki/chunks

limits_config:
  enforce_metric_name: false
  reject_old_samples: false

chunk_store_config:
  max_look_back_period: 0s

table_manager:
  retention_deletes_enabled: true
  retention_period: 792h
4

1 回答 1

1

在 promtail 管道阶段尝试这种格式：

pipeline_stages:
    - regex:
        expression: '^(?P<timestamp>\d{4}-\d{2}-\d{2}\S\d{2}:\d{2}:\d{2}.\d+\+\d{2}:\d{2}).*'
    - timestamp:
        source: timestamp
        format: RFC3339Nano
        action_on_failure: skip
于 2020-12-22T09:03:06.227 回答