kubernetes - promtail "发现接收器的通道已满，因此将重试下一个周期"

Question

所以我是 loki 的新手，对输出感到困惑。这是我在promtail 的一个POD 中看到的。

ts=2021-08-09T01:08:56.295063689Z caller=log.go:124 level=info msg="Seeked /var/log/pods/ci_ygm-backend-dev-kaniko-build-pipeline-run-vxqd6-kustomize-mn7tw_a8ed5fd8-b079-4e0c-ad30-21792024cc0c/step-push/0.log - &{Offset:1285 Whence:0}"
level=info ts=2021-08-09T01:08:56.295135916Z caller=tailer.go:126 component=tailer msg="tail routine: started" path=/var/log/pods/ci_ygm-backend-dev-kaniko-build-pipeline-run-vxqd6-kustomize-mn7tw_a8ed5fd8-b079-4e0c-ad30-21792024cc0c/step-push/0.log
level=info ts=2021-08-09T01:08:56.595534324Z caller=filetargetmanager.go:255 msg="Adding target" key="{app=\"ygm-backend-dev-kaniko-build-pipeline-run-vxqd6-kustomize-d9zrj\", container=\"place-tools\", job=\"ci/ygm-backend-dev-kaniko-build-pipeline-run-vxqd6-kustomize-d9zrj\", namespace=\"ci\", node_name=\"ip-10-110-2-224.us-west-2.compute.internal\", pod=\"ygm-backend-dev-kaniko-build-pipeline-run-vxqd6-kustomize-mn7tw\"}"
ts=2021-08-09T01:08:57.200181022Z caller=log.go:168 component=discovery level=debug msg="Discovery receiver's channel was full so will retry the next cycle"
ts=2021-08-09T01:09:02.21837369Z caller=log.go:168 component=discovery level=debug msg="Discovery receiver's channel was full so will retry the next cycle"
ts=2021-08-09T01:09:07.19746437Z caller=log.go:168 component=discovery level=debug msg="Discovery receiver's channel was full so will retry the next cycle"
ts=2021-08-09T01:09:10.097408968Z caller=log.go:124 level=info msg="Seeked /var/log/pods/ci_ygm-backend-dev-kaniko-build-pipeline-run-vxqd6-kustomize-mn7tw_a8ed5fd8-b079-4e0c-ad30-21792024cc0c/place-scripts/0.log - &{Offset:0 Whence:0}"

通过 HELM 图表安装

values.promtail.yaml

resources:
  limits:
    cpu: 200m
    memory: 512Mi
  requests:
    cpu: 100m
    memory: 128Mi

config:
  lokiAddress: http://loki.monitoring:3100/loki/api/v1/push

values.loki.yaml

nodeSelector:
  ops: "true"

rbac:
  create: true
  pspEnabled: true

然后

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

helm upgrade --install promtail grafana/promtail \
    --create-namespace \
    --namespace monitoring \
    --values cluster/production/charts/loki/values.promtail.yaml 

helm upgrade --install loki grafana/loki \
    --create-namespace \
    --namespace monitoring \
    --values cluster/production/charts/loki/values.loki.yaml 

✗  kgsvc     
NAME            TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
loki            ClusterIP   172.20.158.151   <none>        3100/TCP   31m
loki-headless   ClusterIP   None             <none>        3100/TCP   31m

➜ kgpo 
NAME             READY   STATUS    RESTARTS   AGE
loki-0           1/1     Running   0          29m
promtail-4trn9   1/1     Running   0          14m
promtail-4vndq   1/1     Running   0          17m
promtail-7wpv9   1/1     Running   1          17m
promtail-8fqnx   1/1     Running   0          10m
promtail-95fr4   1/1     Running   0          8m41s
promtail-f8mv2   1/1     Running   0          9m42s
promtail-fxs9b   1/1     Running   0          16m
promtail-lk82m   1/1     Running   0          15m
promtail-pddr8   1/1     Running   0          13m
promtail-pn6qz   1/1     Running   0          5m56s
promtail-tvgs2   1/1     Running   0          6m56s
promtail-wcs62   1/1     Running   0          11m
promtail-wfxxm   1/1     Running   0          12m
promtail-xlbhm   1/1     Running   0          7m49s
utils            1/1     Running   0          26m