amazon-web-services - 如何在 CodePipeline 中处理多个环境？

Question

我正在使用代码管道来部署我的基础架构，并且我希望能够将它部署在不同的环境中（开发、登台、产品......）。

我目前有一个 buildspec.yml 文件，其中包含一些“pip install”指令和“aws cloudformation package”命令。我还创建了 2 个管道，一个用于生产，另一个用于开发，指向 github 上的 2 个不同分支。我遇到的问题是，由于两个分支中的文件都包含相似的资源，例如，我在 S3 存储桶上存在名称冲突。

使用 AWS CLI 和 cloudformation 创建或更新堆栈时，您可以使用 --parameters 选项传递参数。我想在我创建的 2 个管道中做类似的事情。

解决这个问题的最佳解决方案是什么？

最终目标是自动化我们基础设施的部署。我们的基础设施由用户、KMS 密钥、Lamdbas（在 python 中）、组和存储桶组成。

我按照教程创建了两个管道：http: //docs.aws.amazon.com/lambda/latest/dg/automating-deployment.html

第一个管道链接到包含代码的 repo 的主分支，第二个管道链接到暂存分支。我的目标是使用第一个管道在生产环境中自动部署主分支，并使用第二个管道在暂存环境中自动部署暂存分支。

我的 buildspec.yml 文件如下所示：

version: 0.1
phases:
    install:
        commands:
            - pip install requests -t .
            - pip install simplejson -t .
            - pip install Image -t .
            - aws cloudformation package --template-file image_processing_sam.yml --s3-bucket package-bucket --output-template-file new_image_processing_sam.yml
artifacts:
    type: zip
    files:
        - new_image_processing_sam.yml

image_processing_sam.yml 文件如下所示：

AWSTemplateFormatVersion: "2010-09-09"
Transform: "AWS::Serverless-2016-10-31"
Description: Create a thumbnail for an image uploaded to S3
Resources:

  ThumbnailFunction:
    Type: "AWS::Serverless::Function"
    Properties:
      Role: !GetAtt LambdaExecutionRole.Arn
      Handler: create_thumbnail.handler
      Runtime: python2.7
      Timeout: 30
      Description: "A function computing the thumbnail for an image."

  LambdaSecretEncryptionKey:
    Type: "AWS::KMS::Key"
    Properties:
      Description: "A key used to encrypt secrets used in the Lambda functions"
      Enabled: True
      EnableKeyRotation: False
      KeyPolicy:
        Version: "2012-10-17"
        Id: "lambda-secret-encryption-key"
        Statement:
          -
            Sid: "Allow administration of the key"
            Effect: "Allow"
            Principal:
              AWS: "arn:aws:iam::xxxxxxxxxxxxx:role/cloudformation-lambda-execution-role"
            Action:
              - "kms:Create*"
              - "kms:Describe*"
              - "kms:Enable*"
              - "kms:List*"
              - "kms:Put*"
              - "kms:Update*"
              - "kms:Revoke*"
              - "kms:Disable*"
              - "kms:Get*"
              - "kms:Delete*"
              - "kms:ScheduleKeyDeletion"
              - "kms:CancelKeyDeletion"
            Resource: "*"
          -
            Sid: "Allow use of the key"
            Effect: "Allow"
            Principal:
              AWS:
                - !GetAtt LambdaExecutionRole.Arn
            Action:
              - "kms:Encrypt"
              - "kms:Decrypt"
              - "kms:ReEncrypt*"
              - "kms:GenerateDataKey*"
              - "kms:DescribeKey"
            Resource: "*"

  LambdaExecutionRole:
    Type: "AWS::IAM::Role"
    Properties:
      RoleName: "LambdaExecutionRole"
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - "lambda.amazonaws.com"
          Action:
          - "sts:AssumeRole"
      Policies:
        -
          PolicyName: LambdaKMS
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              -
                Effect: "Allow"
                Action:
                  - "kms:Decrypt"
                Resource: "*"
              -
                Effect: "Allow"
                Action:
                  - "lambda:InvokeFunction"
                Resource: "*"
      ManagedPolicyArns:
      - "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"

  UserGroup:
      Type: "AWS::IAM::Group"

  LambdaTriggerUser:
    Type: "AWS::IAM::User"
    Properties:
      UserName: "LambdaTriggerUser"

  LambdaTriggerUserKeys:
    Type: "AWS::IAM::AccessKey"
    Properties:
      UserName:
        Ref: LambdaTriggerUser

  Users:
    Type: "AWS::IAM::UserToGroupAddition"
    Properties:
      GroupName:
        Ref: UserGroup
      Users:
        - Ref: LambdaTriggerUser

  Policies:
    Type: "AWS::IAM::Policy"
    Properties:
      PolicyName: UserPolicy
      PolicyDocument:
        Statement:
          -
            Effect: "Allow"
            Action:
              - "lambda:InvokeFunction"
            Resource:
              - !GetAtt DispatcherFunction.Arn
      Groups:
        - Ref: UserGroup

  PackageBucket:
    Type: "AWS::S3::Bucket"
    Properties:
      BucketName: "package-bucket"
      VersioningConfiguration:
        Status: "Enabled"

Outputs:
  LambdaTriggerUserAccessKey:
    Value:
      Ref: "LambdaTriggerUserKeys"
    Description: "AWSAccessKeyId of LambdaTriggerUser"

  LambdaTriggerUserSecretKey:
    Value: !GetAtt LambdaTriggerUserKeys.SecretAccessKey
    Description: "AWSSecretKey of LambdaTriggerUser"

我在两个管道中添加了一个部署操作，以执行在 beta 操作期间计算的更改集。

第一个管道就像一个魅力，可以完成我期望它做的所有事情。每次我在主分支中推送代码时，它都会被部署。

我面临的问题是，当我在暂存分支中推送代码时，一切都在管道中运行，直到达到部署操作。部署操作尝试创建一个新堆栈，但由于它与 buildspec.yml 和已处理的 image_processing_sam.yml 完全相同，因此我会遇到如下所示的名称冲突。

package-bucket already exists in stack arn:aws:cloudformation:eu-west-1:xxxxxxxxxxxx:stack/master/xxxxxx-xxxx-xxx-xxxx
LambdaTriggerUser already exists in stack arn:aws:cloudformation:eu-west-1:xxxxxxxxxxxx:stack/master/xxxxxx-xxxx-xxx-xxxx
LambdaExecutionRole already exists in stack arn:aws:cloudformation:eu-west-1:xxxxxxxxxxxx:stack/master/xxxxxx-xxxx-xxx-xxxx
...

有没有办法对 buildspec.yml 进行参数化，以便能够为 image_processing_sam.yml 中的资源名称添加后缀？欢迎任何其他实现这一目标的想法。

此致。

score 10 · Accepted Answer

http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/continuous-delivery-codepipeline-basic-walkthrough.html

模板配置文件通过如下参数文件应用于 CodePipeline 中的 CloudFormation：

{
  "Parameters" : {
    "DBName" : "TestWordPressDB",
    "DBPassword" : "TestDBRootPassword",
    "DBRootPassword" : "TestDBRootPassword",
    "DBUser" : "TestDBuser",    
    "KeyName" : "TestEC2KeyName"
    }
}

将这些文件放在 repo 的根目录中，至少可以通过两种方式引用它们。

在您的 CodePipeline CloudFormation 中：

Configuration:
    ActionMode: REPLACE_ON_FAILURE
    RoleArn: !GetAtt [CFNRole, Arn]
    StackName: !Ref TestStackName
    TemplateConfiguration: !Sub "TemplateSource::${TestStackConfig}"
    TemplatePath: !Sub "TemplateSource::${TemplateFileName}"

或者在模板配置字段的控制台中：

值得注意的是，配置文件格式与通过 cli 使用的 CloudFormation 不同

-- parameters

--parameters 使用这种格式：

[
  {
    "ParameterKey": "team",
    "ParameterValue": "AD-Student Life Applications"
  },
  {
    "ParameterKey": "env",
    "ParameterValue": "dev"
  },
  {
    "ParameterKey": "dataSensitivity",
    "ParameterValue": "public"
  },
  {
    "ParameterKey": "app",
    "ParameterValue": "events-list-test"
  }
]

CodePipeline Cloudformation 模板配置文件使用以下格式：

{
  "Parameters" : {
    "DBName" : "TestWordPressDB",
    "DBPassword" : "TestDBRootPassword",
    "DBRootPassword" : "TestDBRootPassword",
    "DBUser" : "TestDBuser",    
    "KeyName" : "TestEC2KeyName"
  }
}

score 4 · Accepted Answer

检查 Eric Nord 的答案。这是您正在寻找的那个。

我也在 AWS 论坛上问过这个问题。

以下是 AWS 提供的解决方案：

你好，

如果您的目标是为 staging 和 master 使用不同的存储桶名称，那么另一种选择是使用 CloudFormation 参数。

在编辑现有管道时，如果您编辑操作，您可以展开“高级”面板并输入参数覆盖，以便为每个阶段指定不同的存储桶前缀。您还可以在工件中输入参数作为单独的 .json 文件。

这里有更多关于这样做的详细信息：http: //docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/continuous-delivery-codepipeline-parameter-override-functions.html

以下是用于测试和生产的不同堆栈配置的完整演练：http: //docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/continuous-delivery-codepipeline-basic-walkthrough.html

蒂姆。

您绝对应该遵循提供的文档。这是我在下面提出的解决方案。

这是我自己不满意的解决方案。

我添加了一个在构建时运行的脚本，并根据构建项目的 CodeBuild 代理的 ARN 修改了我的模板。

我添加了可能发生命名冲突的“BRANCH_NAME”。image_processing_sam.yml 现在是：

AWSTemplateFormatVersion: "2010-09-09"
Transform: "AWS::Serverless-2016-10-31"
Description: Create a thumbnail for an image uploaded to S3
Resources:

  ThumbnailFunction:
    Type: "AWS::Serverless::Function"
    Properties:
      Role: !GetAtt LambdaExecutionRole.Arn
      Handler: create_thumbnail.handler
      Runtime: python2.7
      Timeout: 30
      Description: "A function computing the thumbnail for an image."

  LambdaSecretEncryptionKey:
    Type: "AWS::KMS::Key"
    Properties:
      Description: "A key used to encrypt secrets used in the Lambda functions"
      Enabled: True
      EnableKeyRotation: False
      KeyPolicy:
        Version: "2012-10-17"
        Id: "lambda-secret-encryption-keyBRANCH_NAME"
        Statement:
          -
            Sid: "Allow administration of the key"
            Effect: "Allow"
            Principal:
              AWS: "arn:aws:iam::xxxxxxxxxxxxx:role/cloudformation-lambda-execution-role"
            Action:
              - "kms:Create*"
              - "kms:Describe*"
              - "kms:Enable*"
              - "kms:List*"
              - "kms:Put*"
              - "kms:Update*"
              - "kms:Revoke*"
              - "kms:Disable*"
              - "kms:Get*"
              - "kms:Delete*"
              - "kms:ScheduleKeyDeletion"
              - "kms:CancelKeyDeletion"
            Resource: "*"
          -
            Sid: "Allow use of the key"
            Effect: "Allow"
            Principal:
              AWS:
                - !GetAtt LambdaExecutionRole.Arn
            Action:
              - "kms:Encrypt"
              - "kms:Decrypt"
              - "kms:ReEncrypt*"
              - "kms:GenerateDataKey*"
              - "kms:DescribeKey"
            Resource: "*"

  LambdaExecutionRole:
    Type: "AWS::IAM::Role"
    Properties:
      RoleName: "LambdaExecutionRoleBRANCH_NAME"
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - "lambda.amazonaws.com"
          Action:
          - "sts:AssumeRole"
      Policies:
        -
          PolicyName: LambdaKMSBRANCH_NAME
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              -
                Effect: "Allow"
                Action:
                  - "kms:Decrypt"
                Resource: "*"
              -
                Effect: "Allow"
                Action:
                  - "lambda:InvokeFunction"
                Resource: "*"
      ManagedPolicyArns:
      - "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"

  UserGroup:
      Type: "AWS::IAM::Group"

  LambdaTriggerUser:
    Type: "AWS::IAM::User"
    Properties:
      UserName: "LambdaTriggerUserBRANCH_NAME"

  LambdaTriggerUserKeys:
    Type: "AWS::IAM::AccessKey"
    Properties:
      UserName:
        Ref: LambdaTriggerUser

  Users:
    Type: "AWS::IAM::UserToGroupAddition"
    Properties:
      GroupName:
        Ref: UserGroup
      Users:
        - Ref: LambdaTriggerUser

  Policies:
    Type: "AWS::IAM::Policy"
    Properties:
      PolicyName: UserPolicyBRANCH_NAME
      PolicyDocument:
        Statement:
          -
            Effect: "Allow"
            Action:
              - "lambda:InvokeFunction"
            Resource:
              - !GetAtt DispatcherFunction.Arn
      Groups:
        - Ref: UserGroup

  PackageBucket:
    Type: "AWS::S3::Bucket"
    Properties:
      BucketName: "package-bucketBRANCH_NAME"
      VersioningConfiguration:
        Status: "Enabled"

Outputs:
  LambdaTriggerUserAccessKey:
    Value:
      Ref: "LambdaTriggerUserKeys"
    Description: "AWSAccessKeyId of LambdaTriggerUser"

  LambdaTriggerUserSecretKey:
    Value: !GetAtt LambdaTriggerUserKeys.SecretAccessKey
    Description: "AWSSecretKey of LambdaTriggerUser"

替换模板中“BRANCH_NAME”的 script.sh 是：

#!/bin/bash
echo $CODEBUILD_AGENT_ENV_CODEBUILD_BUILD_ARN
if [[ "$CODEBUILD_AGENT_ENV_CODEBUILD_BUILD_ARN" == *"master"* ]]; then
    sed "s/BRANCH_NAME//g" image_processing_sam.yml > generated_image_processing_sam.yml;
fi
if [[ "$CODEBUILD_AGENT_ENV_CODEBUILD_BUILD_ARN" == *"staging"* ]]; then
    sed "s/BRANCH_NAME/staging/g" image_processing_sam.yml > generated_image_processing_sam.yml;
fi

buildspec.yml 现在是：

version: 0.1
phases:
    install:
        commands:
            # Install required module for python
            - pip install requests -t .
            - pip install simplejson -t .
            - pip install Image -t .
            - bash ./script.sh
            # To be able to see any issue in the generated template
            - cat generated_image_processing_sam.yml
            # Package the generated cloudformation template in order to deploy
            - aws cloudformation package --template-file generated_image_processing_sam.yml --s3-bucket piximate-package-bucket --output-template-file new_image_processing_sam.yml
artifacts:
    type: zip
    files:
        - new_image_processing_sam.yml

我希望它能以某种方式帮助你。如果有人可以提供任何可以帮助的改进或文档，我会很高兴。

amazon-web-services - 如何在 CodePipeline 中处理多个环境？

2 回答 2

Related

Reference