amazon-web-services - 使用 EFK 堆栈从 kibana 上的 S3 获取 AWS 日志

Question

我有一个在 EKS 上运行的 EFK 堆栈。如何从 S3 存储桶收集日志并通过 Kibana 查看它们？

Answer 1

you can use this plugin to add a new source for the s3 https://github.com/tomohisaota/fluent-plugin-forward-aws

first, you need to add the source configs

<source>
  type forward_aws
  aws_access_key_id     XXXXXXXXXXXXXXXXXXXX
  aws_secret_access_key XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

  aws_s3_endpoint       s3-ap-northeast-1.amazonaws.com
  aws_s3_bucketname     XXXXXXXXXXXXXXXXXXXX

  aws_sqs_endpoint      sqs.ap-northeast-1.amazonaws.com
  aws_sqs_queue_url     https://sqs.ap-northeast-1.amazonaws.com/XXXXXXXXXXXXXXXXXXXX
</source>

then you need to parse them with a match and forward them to elasticsearch