我希望将来自远程 EKS 集群的日志转发到托管 ECK 的集中式 EKS 集群。
正在使用的版本：
- EKS v1.20.7
- 弹性搜索 v7.7.0
- Kibana v7.7.0
- 文件节拍 v7.10.0
该设置使用 AWS NLB 使用基于主机的路由将请求转发到 Nginx 入口。
在 Filebeat 上测试 Elasticsearch 的 DNS 查找（filebeat 测试输出）时，它会验证请求。
但 Filebeat 的日志却讲述了一个不同的故事。
2021-10-05T10:39:00.202Z ERROR [publisher_pipeline_output]
pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://elasticsearch.dev.example.com:9200)):
Get "https://elasticsearch.dev.example.com:9200": Bad Request
当使用 curl 请求时，Filebeat 代理可以通过 NLB 连接到远程 Elasticsearch。
配置如下。注意：dev.example.com 是托管 ECK 的远程集群。
app:
name: "filebeat"
configmap:
enabled: true
filebeatConfig:
filebeat.yml: |-
filebeat.autodiscover:
providers:
- type: kubernetes
node: ${NODE_NAME}
hints.enabled: true
templates:
- config:
- type: container
paths:
- /var/lib/docker/containers/*/${data.kubernetes.container.id}-json.log
exclude_lines: ["^\\s+[\\-`('.|_]"]
processors:
- drop_event.when.not.or:
- contains.kubernetes.namespace: "apps-"
- equals.kubernetes.namespace: "cicd"
- decode_json_fields:
fields: ["message"]
target: ""
process_array: true
overwrite_keys: true
- add_fields:
fields:
kubernetes.cluster.name: dev-eks-cluster
target: ""
processors:
- add_cloud_metadata: ~
- add_host_metadata: ~
cloud:
id: '${ELASTIC_CLOUD_ID}'
cloud:
auth: '${ELASTIC_CLOUD_AUTH}'
output:
elasticsearch:
enabled: true
hosts: "elasticsearch.dev.example.com"
username: '${ELASTICSEARCH_USERNAME}'
password: '${ELASTICSEARCH_PASSWORD}'
protocol: https
ssl:
verification_mode: "none"
headers:
Host: "elasticsearch.dev.example.com"
proxy_url: "https://example.elb.eu-west-2.amazonaws.com"
proxy_disable: false
daemonset:
enabled: true
version: 7.10.0
image:
repository: "docker.elastic.co/beats/filebeat"
tag: "7.10.0"
pullPolicy: Always
extraenvs:
- name: ELASTICSEARCH_HOST
value: "https://elasticsearch.dev.example.com"
- name: ELASTICSEARCH_PORT
value: "9200"
- name: ELASTICSEARCH_USERNAME
value: "elastic"
- name: ELASTICSEARCH_PASSWORD
value: "remote-cluster-elasticsearch-es-elastic-user-password"
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
clusterrolebinding:
enabled: true
namespace: monitoring
clusterrole:
enabled: true
serviceaccount:
enabled: true
namespace: monitoring
deployment:
enabled: false
configmap:
enabled: false
任何有关如何启用 Filebeat 转发的提示或建议，将不胜感激:-)
#1 缺少端口：
即使按照建议添加了端口。Filebeat 出错：
2021-10-06T08:34:41.355Z 错误 [publisher_pipeline_output] pipeline/output.go:154 无法连接到退避（elasticsearch（https://elasticsearch.dev.example.com:9200））：获取“https:// /elasticsearch.dev.example.com:9200"：错误请求