我按照本教程为 cloudwatch 日志组配置和部署了 functionbeat。我还添加了弹性搜索和 kibana 端点。这是输出
MacBook-Pro:functionbeat-7.10.2-darwin-x86_64 user$ ./functionbeat setup -e
2021-10-20T22:57:35.097-0400 INFO instance/beat.go:645 Home path: [/Users/user/functionbeat-7.10.2-darwin-x86_64] Config path: [/Users/user/functionbeat-7.10.2-darwin-x86_64] Data path: [/tmp] Logs path: [/tmp/logs]
2021-10-20T22:57:35.098-0400 INFO instance/beat.go:653 Beat ID: c9cbbe8c-319a-4577-be8e-de223fba4f6e
2021-10-20T22:57:35.100-0400 INFO [beat] instance/beat.go:981 Beat info {"system_info": {"beat": {"path": {"config": "/Users/user/functionbeat-7.10.2-darwin-x86_64", "data": "/tmp", "home": "/Users/user/functionbeat-7.10.2-darwin-x86_64", "logs": "/tmp/logs"}, "type": "functionbeat", "uuid": "c9cbbe8c-319a-4577-be8e-de223fba4f6e"}}}
2021-10-20T22:57:35.100-0400 INFO [beat] instance/beat.go:990 Build info {"system_info": {"build": {"commit": "aacf9ecd9c494aa0908f61fbca82c906b16562a8", "libbeat": "7.10.2", "time": "2021-01-12T22:39:38.000Z", "version": "7.10.2"}}}
2021-10-20T22:57:35.100-0400 INFO [beat] instance/beat.go:993 Go runtime info {"system_info": {"go": {"os":"darwin","arch":"amd64","max_procs":8,"version":"go1.14.12"}}}
2021-10-20T22:57:35.101-0400 INFO [beat] instance/beat.go:997 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-10-13T02:09:16.051008-04:00","name":"Jijos-MacBook-Pro.local","ip":["127.0.0.1/8","::1/128","fe80::1/64","fe80::1ce5:e86b:6d5c:88d9/64","192.168.1.11/24","fe80::84d7:afff:fe5e:107c/64","fe80::84d7:afff:fe5e:107c/64","fe80::46cf:bba0:8bb3:5b62/64","fe80::99c:e0d2:dbd5:e7aa/64","fe80::cdce:ea46:e3c:3feb/64","fe80::76aa:a49d:6061:d997/64","fe80::aede:48ff:fe00:1122/64"],"kernel_version":"20.6.0","mac":["a6:83:e7:8a:24:01","a4:83:e7:8a:24:01","86:d7:af:5e:10:7c","86:d7:af:5e:10:7c","82:ea:c5:62:c4:05","82:ea:c5:62:c4:04","82:ea:c5:62:c4:01","82:ea:c5:62:c4:00","82:ea:c5:62:c4:01","ac:de:48:00:11:22"],"os":{"family":"darwin","platform":"darwin","name":"Mac OS X","version":"10.16","major":10,"minor":16,"patch":0,"build":"20G165"},"timezone":"EDT","timezone_offset_sec":-14400,"id":"BEEB9A65-6CAE-51E9-B16E-BCE2FBB9EED9"}}}
2021-10-20T22:57:35.101-0400 INFO [beat] instance/beat.go:1026 Process info {"system_info": {"process": {"cwd": "/Users/user/functionbeat-7.10.2-darwin-x86_64", "exe": "./functionbeat", "name": "functionbeat", "pid": 13218, "ppid": 8720, "start_time": "2021-10-20T22:57:34.835-0400"}}}
2021-10-20T22:57:35.101-0400 INFO instance/beat.go:299 Setup Beat: functionbeat; Version: 7.10.2
2021-10-20T22:57:35.102-0400 INFO [index-management] idxmgmt/std.go:184 Set output.elasticsearch.index to 'functionbeat-7.10.2' as ILM is enabled.
2021-10-20T22:57:35.103-0400 INFO eslegclient/connection.go:99 elasticsearch url: https://l-es.xyz.io:443
2021-10-20T22:57:35.106-0400 INFO [publisher] pipeline/module.go:113 Beat name: Jijos-MacBook-Pro.local
2021-10-20T22:57:35.107-0400 INFO eslegclient/connection.go:99 elasticsearch url: https://l-es.xyz.io:443
2021-10-20T22:57:35.316-0400 INFO [esclientleg] eslegclient/connection.go:314 Attempting to connect to Elasticsearch version 7.10.2
2021-10-20T22:57:35.366-0400 INFO template/load.go:183 Existing template will be overwritten, as overwrite is enabled.
2021-10-20T22:57:35.420-0400 INFO template/load.go:117 Try loading template functionbeat-7.10.2 to Elasticsearch
2021-10-20T22:57:35.496-0400 INFO template/load.go:109 template with name 'functionbeat-7.10.2' loaded.
2021-10-20T22:57:35.496-0400 INFO [index-management] idxmgmt/std.go:298 Loaded index template.
Index setup finished.
在部署functionbeat时，我也能够成功部署它
MacBook-Pro:functionbeat-7.10.2-darwin-x86_64 user$ ./functionbeat -v -e -d "*" deploy fn-cloudwatch-logs
2021-10-20T22:58:30.531-0400 INFO instance/beat.go:645 Home path: [/Users/user/functionbeat-7.10.2-darwin-x86_64] Config path: [/Users/user/functionbeat-7.10.2-darwin-x86_64] Data path: [/tmp] Logs path: [/tmp/logs]
2021-10-20T22:58:30.532-0400 DEBUG [beat] instance/beat.go:697 Beat metadata path: /tmp/meta.json
2021-10-20T22:58:30.532-0400 INFO instance/beat.go:653 Beat ID: c9cbbe8c-319a-4577-be8e-de223fba4f6e
2021-10-20T22:58:30.535-0400 DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:126 add_cloud_metadata: starting to fetch metadata, timeout=3s
2021-10-20T22:58:33.536-0400 DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:162 add_cloud_metadata: received disposition for azure after 3.000470363s. result=[provider:azure, error=failed requesting azure metadata: Get "http://169.254.169.254/metadata/instance/compute?api-version=2017-04-02": dial tcp 169.254.169.254:80: i/o timeout, metadata={}]
2021-10-20T22:58:33.536-0400 DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:162 add_cloud_metadata: received disposition for aws after 3.000674614s. result=[provider:aws, error=failed requesting aws metadata: Get "http://169.254.169.254/2014-02-25/dynamic/instance-identity/document": dial tcp 169.254.169.254:80: i/o timeout, metadata={}]
2021-10-20T22:58:33.536-0400 DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:162 add_cloud_metadata: received disposition for openstack after 3.000710413s. result=[provider:openstack, error=failed requesting openstack metadata: Get "http://169.254.169.254/2009-04-04/meta-data/instance-id": dial tcp 169.254.169.254:80: i/o timeout, metadata={}]
2021-10-20T22:58:33.536-0400 DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:162 add_cloud_metadata: received disposition for gcp after 3.000737886s. result=[provider:gcp, error=failed requesting gcp metadata: Get "http://169.254.169.254/computeMetadata/v1/?recursive=true&alt=json": dial tcp 169.254.169.254:80: i/o timeout, metadata={}]
2021-10-20T22:58:33.536-0400 DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:162 add_cloud_metadata: received disposition for digitalocean after 3.000758996s. result=[provider:digitalocean, error=failed requesting digitalocean metadata: Get "http://169.254.169.254/metadata/v1.json": dial tcp 169.254.169.254:80: i/o timeout, metadata={}]
2021-10-20T22:58:33.536-0400 DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:129 add_cloud_metadata: fetchMetadata ran for 3.000778492s
2021-10-20T22:58:33.536-0400 INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:89 add_cloud_metadata: hosting provider type not detected.
2021-10-20T22:58:33.536-0400 DEBUG [processors] processors/processor.go:120 Generated new processors: add_host_metadata=[netinfo.enabled=[true], cache.ttl=[5m0s]], add_cloud_metadata={}
2021-10-20T22:58:33.538-0400 DEBUG [cli-handler] cmd/cli_handler.go:52 Starting deploy for: fn-cloudwatch-logs
2021-10-20T22:58:33.539-0400 DEBUG [aws] aws/cli_manager.go:119 Deploying function: fn-cloudwatch-logs
2021-10-20T22:58:33.539-0400 DEBUG [provider] aws/template_builder.go:90 Compressing all assets into an artifact
2021-10-20T22:58:35.284-0400 DEBUG [provider] aws/template_builder.go:96 Compression is successful (zip size: 22046351 bytes)
2021-10-20T22:58:35.339-0400 INFO [provider] aws/template_builder.go:155 No role is configured for function fn-cloudwatch-logs, creating a custom role.
2021-10-20T22:58:35.342-0400 DEBUG [aws] aws/cli_manager.go:69 Using cloudformation template:
{
"AWSTemplateFormatVersion": "2010-09-09",
"Resources": {
"fnbfncloudwatchlogs": {
"Properties": {
"Code": {
"S3Bucket": "functionbeat-deploy-bucket-test-poc",
"S3Key": "functionbeat-deployment/fn-cloudwatch-logs/eMPnb_aKewcOO0XW-fgVUvvN0PXWxEDkvxFexJEI-zY/functionbeat.zip"
},
"Description": "lambda function for cloudwatch logs",
"Environment": {
"Variables": {
"BEAT_STRICT_PERMS": "false",
"ENABLED_FUNCTIONS": "fn-cloudwatch-logs"
}
},
"FunctionName": "fn-cloudwatch-logs",
"Handler": "functionbeat-aws",
"MemorySize": 128,
"ReservedConcurrentExecutions": 5,
"Role": {
"Fn::GetAtt": [
"fnbfncloudwatchlogsIAMRoleLambdaExecution",
"Arn"
]
},
"Runtime": "go1.x",
"Timeout": 3
},
"Type": "AWS::Lambda::Function"
},
"fnbfncloudwatchlogsIAMRoleLambdaExecution": {
"Properties": {
"AssumeRolePolicyDocument": {
"Statement": [
{
"Action": "sts:AssumeRole",
"Effect": "Allow",
"Principal": {
"Service": {
"Fn::Join": [
"",
[
"lambda.",
{
"Ref": "AWS::URLSuffix"
}
]
]
}
}
}
]
},
"Path": "/",
"Policies": [
{
"PolicyDocument": {
"Statement": [
{
"Action": [
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Effect": "Allow",
"Resource": [
{
"Fn::Sub": "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/fn-cloudwatch-logs:*"
}
]
}
]
},
"PolicyName": {
"Fn::Join": [
"-",
[
"fnb",
"lambda",
"fn-cloudwatch-logs"
]
]
}
}
],
"RoleName": "functionbeat-lambda-fn-cloudwatch-logs"
},
"Type": "AWS::IAM::Role"
},
"fnbfncloudwatchlogsLogGroup": {
"Properties": {
"LogGroupName": "/aws/lambda/fn-cloudwatch-logs"
},
"Type": "AWS::Logs::LogGroup"
},
"fnbfncloudwatchlogsPermission0": {
"Properties": {
"Action": "lambda:InvokeFunction",
"FunctionName": {
"Fn::GetAtt": [
"fnbfncloudwatchlogs",
"Arn"
]
},
"Principal": {
"Fn::Join": [
"",
[
"logs.",
{
"Ref": "AWS::Region"
},
".",
{
"Ref": "AWS::URLSuffix"
}
]
]
},
"SourceArn": {
"Fn::Join": [
"",
[
"arn:",
{
"Ref": "AWS::Partition"
},
":logs:",
{
"Ref": "AWS::Region"
},
":",
{
"Ref": "AWS::AccountId"
},
":log-group:",
"/aws/containerinsights/translator-eks-ci-blue-cluster/application",
":*"
]
]
}
},
"Type": "AWS::Lambda::Permission"
},
"fnbfncloudwatchlogsSFawscontainerinsightstranslatoreksciblueclusterapplication": {
"Properties": {
"DestinationArn": {
"Fn::GetAtt": [
"fnbfncloudwatchlogs",
"Arn"
]
},
"FilterPattern": "mylog_",
"LogGroupName": "/aws/containerinsights/translator-eks-ci-blue-cluster/application"
},
"Type": "AWS::Logs::SubscriptionFilter"
}
}
}
2021-10-20T22:58:35.343-0400 DEBUG [aws.executor] executor/executor.go:53 The executor is executing '6' operations for converging state
2021-10-20T22:58:35.343-0400 DEBUG [aws] aws/op_ensure_bucket.go:33 Verifying presence of S3 bucket: functionbeat-deploy-bucket-test-poc
2021-10-20T22:58:35.548-0400 DEBUG [aws] aws/op_upload_to_bucket.go:44 Uploading file 'functionbeat-deployment/fn-cloudwatch-logs/eMPnb_aKewcOO0XW-fgVUvvN0PXWxEDkvxFexJEI-zY/functionbeat.zip' to bucket 'functionbeat-deploy-bucket-test-poc' with size 22046351 bytes
2021-10-20T22:58:36.286-0400 DEBUG [aws] aws/op_upload_to_bucket.go:57 Upload successful
2021-10-20T22:58:36.287-0400 DEBUG [aws] aws/op_upload_to_bucket.go:44 Uploading file 'functionbeat-deployment/fn-cloudwatch-logs/2_AZvlkBMEoQfFeV_dSW5B2VD927AWycnifwEPnFtcI/cloudformation-template-create.json' to bucket 'functionbeat-deploy-bucket-test-poc' with size 4231 bytes
2021-10-20T22:58:36.328-0400 DEBUG [aws] aws/op_upload_to_bucket.go:57 Upload successful
2021-10-20T22:58:36.328-0400 DEBUG [aws] aws/op_cloudformation.go:48 Creating CloudFormation create stack request
2021-10-20T22:58:36.848-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::CloudFormation::Stack, LogicalResourceId: fnb-fn-cloudwatch-logs-stack, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: User Initiated
2021-10-20T22:58:40.973-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: fnbfncloudwatchlogsIAMRoleLambdaExecution, ResourceStatus: CREATE_IN_PROGRESS
2021-10-20T22:58:40.973-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbfncloudwatchlogsLogGroup, ResourceStatus: CREATE_IN_PROGRESS
2021-10-20T22:58:43.035-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: fnbfncloudwatchlogsIAMRoleLambdaExecution, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2021-10-20T22:58:43.035-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbfncloudwatchlogsLogGroup, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2021-10-20T22:58:45.091-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::Logs::LogGroup, LogicalResourceId: fnbfncloudwatchlogsLogGroup, ResourceStatus: CREATE_COMPLETE
2021-10-20T22:58:55.375-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::IAM::Role, LogicalResourceId: fnbfncloudwatchlogsIAMRoleLambdaExecution, ResourceStatus: CREATE_COMPLETE
2021-10-20T22:58:57.429-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbfncloudwatchlogs, ResourceStatus: CREATE_IN_PROGRESS
2021-10-20T22:59:01.542-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbfncloudwatchlogs, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2021-10-20T22:59:05.670-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::Lambda::Function, LogicalResourceId: fnbfncloudwatchlogs, ResourceStatus: CREATE_COMPLETE
2021-10-20T22:59:07.727-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbfncloudwatchlogsPermission0, ResourceStatus: CREATE_IN_PROGRESS
2021-10-20T22:59:07.728-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbfncloudwatchlogsSFawscontainerinsightstranslatoreksciblueclusterapplication, ResourceStatus: CREATE_IN_PROGRESS
2021-10-20T22:59:07.728-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbfncloudwatchlogsPermission0, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2021-10-20T22:59:07.728-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbfncloudwatchlogsSFawscontainerinsightstranslatoreksciblueclusterapplication, ResourceStatus: CREATE_IN_PROGRESS, ResourceStatusReason: Resource creation Initiated
2021-10-20T22:59:07.728-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::Logs::SubscriptionFilter, LogicalResourceId: fnbfncloudwatchlogsSFawscontainerinsightstranslatoreksciblueclusterapplication, ResourceStatus: CREATE_COMPLETE
2021-10-20T22:59:18.052-0400 INFO [aws] aws/op_cloudformation.go:97 Stack event received, ResourceType: AWS::Lambda::Permission, LogicalResourceId: fnbfncloudwatchlogsPermission0, ResourceStatus: CREATE_COMPLETE
2021-10-20T22:59:19.735-0400 DEBUG [aws] aws/op_delete_file_bucket.go:38 Removing file 'functionbeat-deployment/fn-cloudwatch-logs/eMPnb_aKewcOO0XW-fgVUvvN0PXWxEDkvxFexJEI-zY/functionbeat.zip' on bucket 'functionbeat-deploy-bucket-test-poc'
2021-10-20T22:59:19.808-0400 DEBUG [aws] aws/op_delete_file_bucket.go:51 Remove successful
2021-10-20T22:59:19.808-0400 DEBUG [aws.executor] executor/executor.go:68 All operations successful
2021-10-20T22:59:19.808-0400 DEBUG [aws] aws/cli_manager.go:125 Successfully created function 'fn-cloudwatch-logs'
2021-10-20T22:59:19.808-0400 DEBUG [aws] aws/cli_manager.go:126 Deploy finish for function 'fn-cloudwatch-logs'
Function: fn-cloudwatch-logs, deploy successful
2021-10-20T22:59:19.810-0400 DEBUG [cli-handler] cmd/cli_handler.go:64 Deploy execution ended
在 aws 控制台中，我看到fn-cloudwatch-logs正在创建该函数，并且我还在 s3 存储桶中看到了 cloudformation 模板。我还看到一个触发器被添加到 cloudwatch 日志组中，但是 kibana 中的 functionbeat status 说No Data is being received。
我还尝试将示例日志数据提供给 lambda 函数，它最终Could not parse events from cloudwatch在 cloudwatch 中引发错误
2021-10-21T19:17:32.605Z ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://******.io:443)): Connection marked as failed because the onConnect callback failed: 169.254.40.221 requires the default distribution of Elasticsearch. Please update to the default distribution of Elasticsearch for full access to all free features, or switch to the OSS distribution of 169.254.40.221.
2021-10-21T19:17:32.605Z ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://******.io:443)): Connection marked as failed because the onConnect callback failed: 169.254.40.221 requires the default distribution of Elasticsearch. Please update to the default distribution of Elasticsearch for full access to all free features, or switch to the OSS distribution of 169.254.40.221.
我在配置中缺少什么吗？