我在使用 Elasticsearch 方面有很好的经验,我使用过 2.4 版本,现在正在尝试学习新的 Elasticsearch。我正在尝试实现 Filebeat 以将我的 apache 和系统日志发送到我的 Elasticsearch 端点。为了节省我的时间,我更喜欢在公共域下的 AWS Elasticsearch Service 上启动一个 t2.medium 单节点实例,并且我附加了访问策略以允许每个人访问集群。AWS Elasticsearch 实例已启动并正常运行。我启动了一个 Ubuntu(18.04) 服务器,下载了 filebeat tar 并在以下配置中进行了以下配置filebeat.yml
:
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["https://my-public-test-domain.ap-southeast-1.es.amazonaws.com:443"]
18.04- # Optional protocol and basic auth credentials.
#protocol: "https"
#username: "elastic"
#password: "changeme"
我启用了所需的模块:
filebeat modules enable system apache
然后根据 filebeat 文档,我更改了 filebeat 文件的所有权,并使用以下命令启动了 filebeat:
sudo chown root filebeat.yml
sudo ./filebeat -e
当我启动 filebeat 时,我遇到了以下权限和所有权问题:
Error loading config from file '/home/ubuntu/beats/filebeat-7.2.0-linux-x86_64/modules.d/system.yml', error invalid config: config file ("/home/ubuntu/beats/filebeat-7.2.0-linux-x86_64/modules.d/system.yml") must be owned by the user identifier (uid=0) or root
为了解决这个问题,我更改了引发错误的文件的所有权。当我重新启动 filebeat 服务时,我开始面临以下问题:
Connection marked as failed because the onConnect callback failed: cannot retrieve the elasticsearch license: unauthorized access, could not connect to the xpack endpoint, verify your credentials
通过这个链接,我发现要使用 AWS Elasticsearch,我需要 Beats OSS 版本。
所以我再次从这个链接下载了beat的OSS版本,并按照与上面相同的过程,但仍然没有运气。现在我面临以下错误:
错误一:
Attempting to reconnect to backoff(elasticsearch(https://my-public-test-domain.ap-southeast-1.es.amazonaws.com:443)) with 12 reconnect attempt(s)
错误2:
Failed to connect to backoff(elasticsearch(https://my-public-test-domain.ap-southeast-1.es.amazonaws.com:443)): Connection marked as failed because the onConnect callback failed: 1 error: Error loading pipeline for fileset system/auth: This module requires an Elasticsearch plugin that provides the geoip processor. Please visit the Elasticsearch documentation for instructions on how to install this plugin. Response body: {"error":{"root_cause":[{"type":"parse_exception","reason":"No processor type exists with name [geoip]","header":{"processor_type":"geoip"}}],"type":"parse_exception","reason":"No processor type exists with name [geoip]","header":{"processor_type":"geoip"}},"status":400}
从第二个错误中,我可以理解 geoip 插件不可用,因此我遇到了这个错误。
还需要做什么才能使其正常工作?
有没有人成功将 Beats 连接到 AWS Elasticsearch?
我还可以采取哪些其他步骤来缓解上述问题?
环境细节:
- AWS 弹性搜索版本:6.7
- 文件节拍:7.2.0