Using CodePipeline For ECS Continuous Deployment

In the Launching A WordPress Multisite App on ECS article the process for launching a Docker container with our WordPress app in the AWS cloud. Now we want to automatically deploy updates to the image definition hosted in a CodeCommit repository to the ECS Cluster. CodePipeline is the tool we are going to be using to assist in this process. If all goes well, CodePipeline will monitor the Docker images repository for changes then fire up the “build image” process and when that finishes run a deployment to the ECS Cluster.

The Environment

An image definition repository has been created and is hosted on CodeCommit. It contains the Dockerfile and buildspec.yml file along with various helper files and other supporting elements that are needed to build the Docker images. We can run and build these locally on Docker using Docker Desktop on and M1 based Mac via Docker compose files.

The Docker images are hosted in an ECR repository, giving use quick and easy access to a private images repository that can provided images to our local dev box or on the cloud.

Elastic Container Service has been configured with a Task Definition that runs the ECR image with the :develop tag and sets up the container definition with virtual CPU and memory for the Docker container. This is pushed into a Cluster which runs some EC2 servers and is wired up with a Service definition that uses the Task to bring in the proper image on ECR and put it on the Cluster EC2 instances behind an application load balancer (ALB) with scaling policies.

We have also used AWS CodeBuild to setup a project that builds docker from within a temporary Docker container using the AWS CodeCommit repo noted above. This replicates the process we can run locally on our Docker CLI on our local dev box to build and push images to the ECR noted above. This makes for a web-based one-click operation to build new dev server images for our Store Locator Plus® SaaS service.

Now onto getting updates in the Docker image definition repo to update our cluster automatically via CodePipeline using the components above. Here are the notes.

Notes On Using CodePipeline for ECS Continuous Deployment

Our pipeline starts with 3 stages: Source, Build, and Deploy using a CodePipeline v1 type.

CodePipeline Stage 1: Source

We got this wrong on the first go assuming our source would be when the ECR image changes. But, guess what? The CodePipline is supposed to help BUILD that image, so that is not where we want to start. What we want CodePipeline to monitor is our image definition source in CodeCommit. Our source needs to be the CodeCommit ECS Image repo.

Action name: Source
Provider: AWS CodeCommit
Repo: myslp_aws_ecs_kit
Branch: develop
Change detection: AWS CodePipeline
Output Artifact: CodePipeline default
Output Artifacts (JSON var name): SourceArtifacts

CodePipeline Stage 2 : Build

Action name: Build
Provider: AWS CodeBuild
Region: us-east-1
Input Artifacts: SourceArtifacts (from above)
Project name (from CodeBuild): myslp_image_builder_dev
Built Type: single build
Variable Namespace: BuildVariables
Output Artifacts: BuildArtifact

CodePipeline Stage 3: Deploy

Deploy Stage Attempt 1

Action name: Deploy
Action provider: Amazon ECS
Region: us-east-1
Input Artifacts: BuildArtifact (from prior stage)
Cluster: myslp-cluster-dev
Service: myslp-webserer-service-dev
Variable namespace: DeployVariables

This did not work, CodePipeline kept trying to read artifacts from a non-existent AWS S3 bucket. The CodePipeline role has full access to S3 to create buckets objects ,etc. and policy simulation checks show S3 access is not the issue. The problem appears the S3 bucket is never created (turns out there was not artifact defined in buildspec.yml).

Unable to access the artifact with Amazon S3 object key 'myslp-webserver-pipe/BuildArtif/******' located in the Amazon S3 artifact bucket 'codepipeline-us-east-1-******'. The provided role does not have sufficient permissions.

The build stage did not create the bucket noted above.

Deploy Stage Attempt 2

Action name: Deploy
Action provider: Amazon ECS
Region: us-east-1
Input Artifacts: BuildArtifact (from prior stage)
Cluster: myslp-cluster-dev
Service: myslp-webserer-service-dev
Image Definitions File: imagedefinitions.json

Here we are giving Deploy a hint that there will be an imagedefinitions.json file coming out of the Build stage. We also need to update our buildspec.yml to create that artifact in real-time.

version: 0.2

phases:
  pre_build:
    commands:
      - ACCOUNT_URI=$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com
      - REPOSITORY_URI=$ACCOUNT_URI/$IMAGE_REPO_NAME
      - echo Logging in to Amazon ECR...
      - aws --version
      - aws ecr get-login-password --region $AWS_DEFAULT_REGION | docker login --username AWS --password-stdin $ACCOUNT_URI
  build:
    commands:
      - echo Docker image build started on `date`
      - cd Docker/Images
      - docker build --platform=linux/arm64 --file Dockerfile-WordPress -t $IMAGE_REPO_NAME:$IMAGE_TAG .
      - docker tag $IMAGE_REPO_NAME:$IMAGE_TAG $REPOSITORY_URI:$IMAGE_TAG
      - cd ../..
  post_build:
    commands:
      - echo Docker image build and tagging completed on `date`
      - echo Pushing the Docker image...
      - docker push $REPOSITORY_URI:$IMAGE_TAG
      - echo Writing image definitions file...
      - printf '[{"name":"myslp2024-aarch64-container","imageUri":"%s"}]' $REPOSITORY_URI:$IMAGE_TAG > imagedefinitions.json
artifacts:
  files: imagedefinitions.json

About buildspec.yml

This file is used by the CodeBuild service on AWS to build the image as well as to generate an artifact that is stored in S3. One element of the artifact that is output by the build process and used by CodePipeline’s deploy stage is the imagedefinition.json file. The post_build stage of the CodeBuild generates that file with a printf statement. The format is a JSON string where the name value “myslp2024-aarch64-container” needs to match the container name you created when setting up your task definition for the Elastic Container Service (ECS).

The variables that are fed into this buildspec.yml file come from the CodeBuild configuration. You need to setup CodeBuild to pass in these variables; a better option than hard-coding them into the process. Here is an example config from CodeBuild:

Configuring The ECS Services

One issue that kept coming up when trying to deploy the new images whenever our image definition repo changed was the fact that our ECS cluser continually dropped the service definition. When that happened the AWS CloudFormation remained, but the underlying Application Load Balancer (ALB), Target Groups, Scaling Policy, and other elements were destroyed. This happened even after turning off the failure protection and rollback features of the service.

Turns out using the “Capicity provider strategy” was not the best option for this stage of our DevOps work. We will revisit this topic and learn more in the future. For now, the trick to keeping the service online was to use a launch type and set it to EC2.

With these piece in place we can now update our image definition on our local dev box and when we push this to the develop branch of our “ECS Kit” docker image builder repository , CodePipeline detects the change, pulls down the dev kit, fires up a temp server where it builds our docker image and publishes is to the ECR host and tags the images accordingly, then updates the ECS Cluster. The end result is our AWS cloud-hosted service is automatically updated to the latest server image with the latest code and is ready for testing within 10 minutes of the change.

Now on to automating our test suite in the deployment process and getting this to update our staging, and eventually production, SaaS service when those tests pass.

Post Image by Claudia Eichenseher from Pixabay