kubernetes - Fluentbit 在 Kubernetes 中获取 Docker Logs(Systemd) 无法正常工作

Question

我正在尝试在 Kubernetes 中配置 Fluentbit 以从应用程序 POD/Docker 容器中获取日志，并使用 GELF 格式将此日志消息发送到 Graylog，但这不起作用。

请参阅下面的堆栈：

输入

Docker 版本 1.13.1
- Docker 日志格式 => JSON
- Docker 日志驱动程序 => Journald => systemd
Fluent-bit 1.3 在 Kubernetes 中作为 Daemonset 运行
Kubernetes 1.17
操作系统主机：CentOS 7

输出

消息输出格式：GELF 1.1
集中式日志 => Graylog 3

问题是fluentbit没有从systemd读取日志我在两个输出（Systemd，Stdout）中都没有得到任何日志，STDOUT只是为了帮助进行故障排除。

我不知道为什么我无法从 systemd 读取。我完全按照文档 https://docs.fluentbit.io/manual/input/systemd

我的 K8S 配置：

流利的位configmap.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: log
  labels:
    k8s-app: fluent-bit
data:
  # Configuration files: server, input, filters and output
  # ======================================================
  fluent-bit.conf: |
    [SERVICE]
      Flush         1
      Log_Level     debug
      Daemon        off

    @INCLUDE input-systemd.conf
    @INCLUDE output-stdout.conf

  input-systemd.conf: |
    [INPUT]
      Name            systemd
      Tag             host.*
      Parser          json
      Systemd_Filter  _SYSTEMD_UNIT=docker.service

  output-graylog.conf: |
    [OUTPUT]
      Name          gelf
      Match         *
      Host          10.142.15.214
      Port          12201
      Mode          tcp
      Gelf_Short_Message_Key log

  output-stdout.conf: |
    [OUTPUT]
      Name   stdout
      Match  *

流利的位ds.yaml

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: log
  labels:
    k8s-app: fluent-bit-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  selector:
    matchLabels:
      k8s-app: fluent-bit-logging
      version: v1
      kubernetes.io/cluster-service: "true"
  template:
    metadata:
      labels:
        k8s-app: fluent-bit-logging
        version: v1
        kubernetes.io/cluster-service: "true"
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "2020"
        prometheus.io/path: /api/v1/metrics/prometheus
    spec:
      containers:
      - name: fluent-bit
        image: fluent/fluent-bit:1.3.5
        imagePullPolicy: Always
        ports:
          - containerPort: 2020
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: fluent-bit-config
          mountPath: /fluent-bit/etc/
      terminationGracePeriodSeconds: 10
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: fluent-bit-config
        configMap:
          name: fluent-bit-config
      serviceAccountName: fluent-bit
      tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      - operator: "Exists"
        effect: "NoExecute"
      - operator: "Exists"
        effect: "NoSchedule"

流利的位角色绑定.yaml

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: fluent-bit-read
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluent-bit-read
subjects:
- kind: ServiceAccount
  name: fluent-bit
  namespace: log

流利的位角色.yaml

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: fluent-bit-read
rules:
- apiGroups: [""]
  resources:
  - namespaces
  - pods
  verbs: ["get", "list", "watch"]

fluent-bit-service-account.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluent-bit
  namespace: log

我的 Fluentbit OUTPUT(STDOUT) 仅用于调试：

$ kubectl logs -f fluent-bit-2bzxb -n log


[2020/02/20 18:54:23] [Warning] [config] I cannot open /fluent-bit/etc/..2020_02_20_18_54_22.252769193/parsers_custom.conf file
[2020/02/20 18:54:23] [ info] [storage] initializing...
[2020/02/20 18:54:23] [ info] [storage] in-memory
[2020/02/20 18:54:23] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2020/02/20 18:54:23] [ info] [engine] started (pid=1)
[2020/02/20 18:54:23] [ info] [filter_kube] https=1 host=kubernetes.default.svc port=443
[2020/02/20 18:54:23] [ info] [filter_kube] local POD info OK
[2020/02/20 18:54:23] [ info] [filter_kube] testing connectivity with API server...
[2020/02/20 18:54:23] [ info] [filter_kube] API server connectivity OK
[2020/02/20 18:54:23] [ info] [sp] stream processor started

问题是我没有从 systemd 使用此配置获取任何日志

score 2 · Accepted Answer

谢谢@edsiper 我修复了我的守护进程添加“路径：/运行/日志”

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: log
  labels:
    k8s-app: fluent-bit-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  selector:
    matchLabels:
      k8s-app: fluent-bit-logging
      version: v1
      kubernetes.io/cluster-service: "true"
  template:
    metadata:
      labels:
        k8s-app: fluent-bit-logging
        version: v1
        kubernetes.io/cluster-service: "true"
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "2020"
        prometheus.io/path: /api/v1/metrics/prometheus
    spec:
      containers:
      - name: fluent-bit
        image: fluent/fluent-bit:1.3.5
        imagePullPolicy: Always
        ports:
          - containerPort: 2020
        env:
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: systemdlog
          mountPath: /run/log
        - name: fluent-bit-config
          mountPath: /fluent-bit/etc/
      terminationGracePeriodSeconds: 10
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: systemdlog
        hostPath:
          path: /run/log
      - name: fluent-bit-config
        configMap:
          name: fluent-bit-config
      serviceAccountName: fluent-bit
      tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      - operator: "Exists"
        effect: "NoExecute"
      - operator: "Exists"
        effect: "NoSchedule"

score 1 · Accepted Answer

1

您的 Fluent Bit 容器是否可以访问 Systemd 日志路径？

于 2020-02-20T20:29:17.530 回答

score 1 · Accepted Answer

没有足够的业力发表评论，所以发布作为@edsiper 的答案：“您的 Fluent Bit 容器是否可以访问 Systemd 日志路径？” 在默认设置 - 不 - 它没有。当我试图解决这个问题时，我偶然发现了这个线程：https ://github.com/fluent/fluent-bit/issues/497

长话短说：

您需要以 root 身份运行 fluent-bit 容器，因为访问日志需要 root 权限
将 docker 中的机器 id 设置为与根机器中的相同

绑定 /run/log/journal:/run/log/journal 所以：

fluent-bit:
   image: 'bitnami/fluent-bit:latest'
   restart: always
   user: root        #give root access
   network_mode: host
   command: /fluent-bit/bin/fluent-bit -c /fluent-bit/etc/fluent-bit.conf
   volumes:
      - ./service/config/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf
      - /etc/machine-id:/etc/machine-id:ro       #set the machine id
      - /run/log/journal:/run/log/journal        #give access to logs

然后，在 fluent-bit.conf 中，您需要编辑 INPUT Path：

 [INPUT]
     Name            systemd
     Tag             *
     Path            /run/log/journal
     Systemd_Filter    _SYSTEMD_UNIT=docker.service
     Systemd_Filter    _SYSTEMD_UNIT=kubelet.service

score 0 · Accepted Answer

使用相同的配置，我得到 [info] [filter_kube] 测试与 API 服务器的连接性... [warn] [filter_kube] 无法获取 POD 的元数据我使用 Azure Kubernetes 服务。为什么我收到此错误的任何指示

kubernetes - Fluentbit 在 Kubernetes 中获取 Docker Logs(Systemd) 无法正常工作

输入

输出

4 回答 4

Related

Reference