我正在获取 Docker(Systemd) 日志并尝试以 GELF 格式将其发送到 Graylog 3 输出，但日志格式不正确，并且 Graylog 将其丢弃。
我关注这个参考：
- https://docs.fluentbit.io/manual/output/gelf
- https://docs.fluentbit.io/manual/input/systemd
- https://fluentbit.io/kubernetes/
请参阅下面的堆栈：
输入
- Docker 版本 1.13.1
- Docker 日志格式 => JSON
- Docker 日志驱动程序 => Journald => systemd
- Fluent-bit 1.3 在 Kubernetes 中作为 Daemonset 运行
- Kubernetes 1.17
- 操作系统主机：CentOS 7
输出
- 消息输出格式：GELF 1.1
- 集中式日志 => Graylog 3
我的 Kubernetes 配置：
流利的位configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: fluent-bit-config
namespace: log
labels:
k8s-app: fluent-bit
data:
# Configuration files: server, input, filters and output
# ======================================================
fluent-bit.conf: |
[SERVICE]
Flush 1
Log_Level info
Daemon off
Parsers_File parser-docker.conf
[INPUT]
Name tail
Tag kube.*
Path /var/log/messages
Parser docker
DB /var/log/flb_kube.db
Mem_Buf_Limit 5MB
Refresh_Interval 10
[FILTER]
Name kubernetes
Match kube.*
Merge_Log_Key log
Merge_Log On
Keep_Log Off
Annotations Off
Labels Off
[FILTER]
Name nest
Match *
Operation lift
Nested_under log
[OUTPUT]
Name gelf
Match kube.*
Host 10.142.15.214
Port 12201
Mode tcp
Gelf_Short_Message_Key data
[OUTPUT]
Name stdout
Match *
parser-docker.conf: |
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep Off
流利的位ds.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluent-bit
namespace: log
labels:
k8s-app: fluent-bit-logging
version: v1
kubernetes.io/cluster-service: "true"
spec:
selector:
matchLabels:
k8s-app: fluent-bit-logging
version: v1
kubernetes.io/cluster-service: "true"
template:
metadata:
labels:
k8s-app: fluent-bit-logging
version: v1
kubernetes.io/cluster-service: "true"
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "2020"
prometheus.io/path: /api/v1/metrics/prometheus
fluentbit.io/exclude: "true"
spec:
containers:
- name: fluent-bit
image: fluent/fluent-bit:1.3.5
imagePullPolicy: Always
ports:
- containerPort: 2020
env:
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: systemdlog
mountPath: /run/log
- name: fluent-bit-config
mountPath: /fluent-bit/etc/
terminationGracePeriodSeconds: 10
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: systemdlog
hostPath:
path: /run/log
- name: fluent-bit-config
configMap:
name: fluent-bit-config
serviceAccountName: fluent-bit
tolerations:
- key: node-role.kubernetes.io/master
operator: Exists
effect: NoSchedule
- operator: "Exists"
effect: "NoExecute"
- operator: "Exists"
effect: "NoSchedule"
流利的位角色绑定.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: fluent-bit-read
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: fluent-bit-read
subjects:
- kind: ServiceAccount
name: fluent-bit
namespace: log
流利的位角色.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: fluent-bit-read
rules:
- apiGroups: [""]
resources:
- namespaces
- pods
verbs: ["get", "list", "watch"]
fluent-bit-service-account.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: fluent-bit
namespace: log
我的 Fluentbit OUTPUT(STDOUT) 仅用于调试：
$ kubectl logs -f fluent-bit-2bzxb -n log
[9] host.docker.service: [1582317069.020005000, {"PRIORITY"=>"6", "_TRANSPORT"=>"journal", "_PID"=>"1486", "_UID"=>"0", "_GID"=>"0", "_COMM"=>"dockerd-current", "_EXE"=>"/usr/bin/dockerd-current", "_CMDLINE"=>"/usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --init-path=/usr/libexec/docker/docker-init-current --seccomp-profile=/etc/docker/seccomp.json --selinux-enabled --log-driver=journald --signature-verification=false --storage-driver overlay2", "_CAP_EFFECTIVE"=>"1fffffffff", "_SYSTEMD_CGROUP"=>"/system.slice/docker.service", "_SYSTEMD_UNIT"=>"docker.service", "_SYSTEMD_SLICE"=>"system.slice", "_BOOT_ID"=>"18d81c7e97e6419f999af50af13060c8", "_MACHINE_ID"=>"b5447343d5617cb5f7fd428164927298", "_HOSTNAME"=>"k8s-worker-1", "CONTAINER_TAG"=>"d72b414a9bcc", "CONTAINER_ID"=>"d72b414a9bcc", "CONTAINER_ID_FULL"=>"d72b414a9bccef31dbaa4c8473b06d63583195ebde9a8b729a06a81b68233144", "CONTAINER_NAME"=>"k8s_fluent-bit_fluent-bit-zg7pz_log_43007dd0-ce10-4c6d-a97f-e7369f866879_0", "MESSAGE"=>"
[9] host.docker.service: [1582317068.864240000, {"_TRANSPORT"=>"journal", "_PID"=>"1486", "_UID"=>"0", "_GID"=>"0", "_COMM"=>"dockerd-current", "_EXE"=>"/usr/bin/dockerd-current", "_CMDLINE"=>"/usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --init-path=/usr/libexec/docker/docker-init-current --seccomp-profile=/etc/docker/seccomp.json --selinux-enabled --log-driver=journald --signature-verification=false --storage-driver overlay2", "_CAP_EFFECTIVE"=>"1fffffffff", "_SYSTEMD_CGROUP"=>"/system.slice/docker.service", "_SYSTEMD_UNIT"=>"docker.service", "_SYSTEMD_SLICE"=>"system.slice", "_BOOT_ID"=>"18d81c7e97e6419f999af50af13060c8", "_MACHINE_ID"=>"b5447343d5617cb5f7fd428164927298", "_HOSTNAME"=>"k8s-worker-1", "PRIORITY"=>"3", "CONTAINER_TAG"=>"d3383915a884", "CONTAINER_ID"=>"d3383915a884", "CONTAINER_ID_FULL"=>"d3383915a884fb0e2b40189e7db1a1131161e57dba39a40824accf1b2aa59f22", "CONTAINER_NAME"=>"k8s_demo-app_demo-app-6c79ffd869-trstt_default_cd6c5f4d-ec44-4f43-9b92-d7bb28a5f676_1", "MESSAGE"=>"2020/02/21 20:31:08 10.142.15.231:48012 GET /", "_SOURCE_REALTIME_TIMESTAMP"=>"1582317068863691"}]", "_SOURCE_REALTIME_TIMESTAMP"=>"1582317069000509"}]"
问题是如何将日志正确格式化为 GELF 格式以发送到 Graylog 3