Creating Docker Images With Code Submodules

As we move to an updated deployment stack using AWS containers for our Store Locator Plus® SaaS service, there are several changes needed in our IT and development toolkit to automate this process. The prior Grunt + Bitbucket CI/CD pipeline tool works well, but it makes it very difficult to test and update the underlying operating system and nearby services such as PHP or the web service. Rather than continue with the manual EC2 image building and testing cycles we have made the choice to move toward a container based service for not only our development environment but also the staging and production services in our DSP environment.

As such we are researching and testing various develop, build, and deploy technologies centered around the AWS stack. This article serves as my notebook on what we are testing, what works and what does not, and how to move forward from here.

Current State of Devops

For this project we are focusing on using Elastic Container Service (ECS) as a method for launching both production and staging services on AWS. Our development remains on a local Docker installation, but we are moving toward using related images in the ECS deployments for consistency. In the past the Docker development environment was a “close proxy” to staging and production but technically did not share any primary components such as the OS, web services, and PHP stack.

Our current setup includes a test case ECS configuration that includes a staging environment. This consists of a Cluster definition that launches a Staging Service. The Service references and underlying task definition to determine the container environment and, between the two of them, references a Docker image stored on the Elastic Container Registry (ECR).

A local development environment includes a repository , myslp_aws_ecs_kit, that is the “Docker builder and launcher” kit used to make the images that are eventually stored in ECR for staging and production.

Current Goals

Our goal is to pull in the custom MySLP code from a designated branch (develop , staging, production) and create a corresponding Docker image that will be stored on ECR.

While this process may be manual , driven via a command line entry or IDE script, eventually we want this to be connected to a CI/CD process. The ultimate goal is to generate and publish a new ECR dev/stage/prod image whenever the dev/stage/prod branch of any included code repository is updated.

The idea is to use a “docker image manager” repo to build ECR images and quickly deploy different configurations. We only need to tweak the Dockerfile to do things like build a SaaS server running that latest version of PHP or try out our custom SaaS code against a newer version of WordPress. By using different branches of code we can have several types of environments going, each attached to a different image builder process on CodeBuild. This allows for parallel testing and development of various Devops stacks without compromising production services. When we pick a new stack or codebase we can easily roll it into our production branch and image builder to fire off an ECS stack update. If all goes well new production (or staging) servers can come online automatically.

Review of AWS Developer Tools

Instead of creating a local set of scripts with Grunt or other tools, let’s look at the AWS services and see if there is something we can use to build and deploy our Docker image for us. This will mean pulling in code with git subtree.

CodeCommit – private git repo hosting

The AWS ecosystem is far easier to use when all things are in AWS. While AWS services have a strong link to GitHub for repository hosting, we are trying out AWS CodeCommit for hosting our code and IT stack repositories.

Pricing : first 5 users with up to 5,000 repositories per account , 50 GB/month of storage, and 10,000 git requests/month is free.

CodePipeline – a CI/CD service for code repos

A CI/CD service that monitors code for changes and activates a series of actions when the code changes. This can be used to trigger a build, testing, and possibly deployment when code changes.

Pricing: $1/month per active pipeline

CodeBuild – build and test code including Docker images

CodeBuild works with multiple services including ECR, CodeCommit, and CodePipeline to build code, install it in images and create artifacts needed to deploy the staging and production stacks.

Pricing: pay for underyling AWS resources as needed

• Source can be from CodeCommit
  • Supports git submodules (does not appear to support subtrees)
• Can run locally see “Build locally” at AWS Codebuild user Guide

This is essentially a “Docker In Docker” configuration where a Docker image, in this case running on a temporary EC2 instance, builds and publishes another Docker image.

Our Docker Image Builder Process

We started with a baseline configuration using Docker Desktop on a M1 Mac. With this in mind we are using ARM64 as our base platform for both the host and guest images.

The Docker Image Builder Repo

We created a repo hosted on Code Commit that defines our Docker build environment. The ultimate goal is to build a container that hosts a WordPress multisite installation with our custom code for the Store Locator Plus® SaaS platform. It will be used for development, staging, and production servers deployed to different locations depending on the role. This is stored in a myslp_aws_ecs_kit directory.

Inside the directory we have supporting image files in ./Docker/Images that are used by our Dockerfile (Dockerfile-WordPress) to configured the image. There is a buildspec.yml to help AWS Code Build run the commands to build and publish the resulting docker image to ECR, the standard git stuff (.gitignore and .gitmodules for our submodules) and a README.md file.

Our CodeBuild Configuration

In our configuration we end up launching a temporary small EC2 server that pulls in our myslp_aws_ecs_kit Docker Image builder repo along with the defined submodules, runs a few commands, and spits out a Docker image hosted on ECR. Here are our configuration details that got it working with submodules that host various WordPress custom plugins, themes, and other code elements.

Project configuration

Most of this is non-critical. Come up with a project name (myslp_image_builder).

Source

We use our AWS CodeCommit repository. This is a set of files (in our case Dockerfile definitions, buildspecs, etc. needed to build a Docker image) that is loaded into the temp cloud server that Code Builder fires up.

Source provider: AWS CodeCommit
Repository: myslp_aws_ecs_kit (your repo with your Docker config stuff)
Git submodules: check off Use Git submodules (we have submodules attached to our repo to pull in related code from other repos, our standalone plugins, etc.)

Environment

We got this wrong on the first attempt, this is the details about the temporary cloud server that will be running the image builder commands. It is NOT related to the resulting Docker image. Using the CodeBuild defaults works best here.

Environment image: Managed image
Compute: EC2
Operating system: Amazon Linux (so we can use ARM architecture)
Runtime: Standard
Image: aws/codebuild/amazonlinux2-aarch64-standard:3.0
Image version: Always use the latest image for this runtime version
Service role: New service role
Role name: codebuild-myslp_ecs_dev_builder-service-role
in advanced…
Privileged: checked Enable this flag if you want to build Docker images or want your builds to get elevated privileges
Compute: 4GB, 2vCPU
Environment Variables: AWS_DEFAULT_REGION = us-east-1, AWS_ACCOUNT_ID=<our account id>, IMAGE_TAG=develop, IMAGE_REPO_NAME=slp-saas-test-arm-64

Environment Notes

The environment variables are important. Set the region to your preferred ECR storage region, the account ID to your AWS account ID.

The IMAGE_TAG and IMAGE_REPO_NAME are used by the buildspec.yml file to fill in the command line commands that tag and push to the repo. This must be named to match the existing ECR repo.

Role Notes

To build and push images to ECR you will need special privileges added to the newly-created role. After the CodeBuild specification is setup. In our case we are using CodeCommit submodules as well, so we needed to add some CodeCommit permissions for that, along with the standards CodeBuild permissions for CloudWatch, etc.

Unfortunately, despite checking the “Allow AWS CodeBuild to modify this service role so it can be used with this build project”, it does not even consider the submodule setting or Docker image deployments.

Our role settings editing in IAM under permissions for this role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Resource": [
                "arn:aws:logs:us-east-1:<account_id>:log-group:/aws/codebuild/myslp_ecs_dev_builder",
                "arn:aws:logs:us-east-1:<account_id>:log-group:/aws/codebuild/myslp_ecs_dev_builder:*"
            ],
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ]
        },
        {
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::codepipeline-us-east-1-*"
            ],
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:GetObjectVersion",
                "s3:GetBucketAcl",
                "s3:GetBucketLocation"
            ]
        },
        {
            "Effect": "Allow",
            "Resource": [
                "arn:aws:codecommit:us-east-1:<account_id>:myslp_aws_ecs_kit"
            ],
            "Action": [
                "codecommit:GitPull"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "codebuild:CreateReportGroup",
                "codebuild:CreateReport",
                "codebuild:UpdateReport",
                "codebuild:BatchPutTestCases",
                "codebuild:BatchPutCodeCoverages"
            ],
            "Resource": [
                "arn:aws:codebuild:us-east-1:<account_id>:report-group/myslp_dev_builder-*"
            ]
        }
    ]
}

Buildspec

Use a buildspec file is checked.

Artifacts

None set.

Logs

CloudWatch logs is checked — this makes it easier to debug CodeBuild issues.

What We Learned

For pulling custom code into a WordPress multisite installation, like that used in our SaaS product, you can use git submodules that will pull independent plugins into specific subdirectories. This works great for grabbing code from things like our custom plugins and dropping them in the right place in our Docker image that hosts our WordPress web app running our SaaS product.

Code Build can pull the image definition code, combine it with the submodule updates, and generate an image. It can include multiple custom build commands that can setup, tweak, and clean up the image before production. It can also work on various branches to drive different phases of a DSP Devops configuration.