amazon-web-services - 在 AWS-ec2 上安装 Beats 以发送到本地 ELK

Question

我必须在 AWS-EC2-Windows 服务器上设置 jboss，这将根据要求进行扩展。我们正在使用 ELK 进行基础设施监控，将在此处安装beats，将数据发送到本地logstash。在那里，我们使用主机名和 IP 登录服务器。现在的问题是：在自动缩放的情况下，我们如何实现这一点。

请指教。

谢谢， 阿布舍克

Answer 1

如果您要创建一个 EC2 实例并为其创建一个 AMI，以便基于该实例自动扩展，那么配置可以成为其中的一部分。

如果您的意思是将其添加到允许列表中，您可以使用直接连接或定义了自定义 CIDR 块的 VPC，并将该子网添加到允许列表中。

Answer 2

AFAIK You need to change the logstash config file on disk to add new hosts, and it should notice the updated config automatically and "just work".

I would suggest a local script that can read/write the config file and that polls an SQS queue "listening" for autoscaling events. You can have your ASG send SNS messages when it scales and then subscribe an SQS queue to receive them. Messages will be retained for upto 14 days and theres options to add delays if required. The message you receive from SQS will indicate the region, instance-id and operation (launched or terminated) from which you can lookup the IP address/hostname to add/remove from the config file (and the message should be deleted from the queue when processed successfully). Editing the config file is just simple string operations to locate the right line and insert the new one. This approach only requires outbound HTTPS access for your local script to work and some IAM permissions, but there is (a probably trivial) cost implication.

Another option is a UserData script thats executed on each instance at startup (part of the Launch Template of your AutoScale group). Exactly how it might communicate with your on-prem depends on your architecture/capabilities - anythings possible. You could write a simple webservice to manage the config file and have the instances call it but thats a lot more effort and somewhat risky in my opinion.

FYI - if you use SQS look at Long Polling if your checking the queues frequently/want the message to propagate as quickly as possible (TLDR - faster & cheaper than polling any more than twice a minute). Its good practice to use a dead-letter queue with SQS - messages that get retrieved but not removed from the queue end up here. Setup alarms on the queue and deadletter queue to alert you via email if there are messages failing to be processed or not getting picked up in sensible time (ie your script has crashed etc).