AWS CodeBuild Testing Using Private ECRs

After another month of playing with my latest startup venture at Rally Creek USA, I am back to working on the Store Locator Plus® project. Since that project has been working with a skeleton crew as I attempt to update the project it has been lacking in quality assurance testing between releases. Before the next release of the WordPress plugin and SaaS platform update, it is critical to get some automated testing underway. While unit testing is OK, for a SaaS platform the best option is to perform full end-to-end testing. Part of that process involves AWS CodeBuild testing using private ECRs.

Sidebar: Private ECRs or “elastic container registries” is a place to store private Docker images that can be used by your organization. We are using it to create a WordPress image with debugging built in and enabled along with an SSL certificate enabled https:// access to running containers. This is critical as many Google Map API services do not function without a full SSL cert in place.

While I could write some basic tests and run them locally, given the new ventures and limited time to work on the Store Locator Plus® project it is clear that a test cycle that is fully integrated with the new CI/CD process built on AWS CodeBuild is necessary. Turns out setting up AWS CodeBuild for Playwright testing that runs agains a private ECR is not that difficult.

Or I should say “not that difficult” if you skip the 90% of online resources that provide old outdated or purely incorrect information. Turns out there is a TON of incorrect and horribly outdated information about how to access a Docker image stored on your private registry (private ECR) — even if you are running CodeBuild processes from the same primary account that owns the image.

Below are my “Cliff Notes” on how I got it working. This assumes a basic knowledge of AWS services including ECR, CodeBuild, and IAM. Later we will use this baseline to tie this into CodePipline and fire off not only the build, but the test and publication of the SLP WordPress plugin in a completely hands-off automated process.

CodeBuild Testing Using Private ECRs Outline

I assume you already have a Docker image that has your own “special sauce” baked in that is hiding in a private ECR.

There are a few basic steps to this process.

1. Prepare A User For Private ECR Access: Setup an IAM user specifically for this codebuild process “my-codebuilt-test-user” for example.
2. Configure The CodeBuild Environment
   • Choose the right image type for CodeBuild to run in.
   • Set some key environment variables
3. Create the Docker config.json to use ECR Login helper

Prepare A User For Private ECR Access

Setup an IAM user specifically for this codebuild process “my-codebuilt-test-user” for example.

Assign that user the permission policy “AmazonEC2ContainerRegistryPowerUser”

Note: you probably want to create a custom policy at some point without the push, delete, etc. ECR command access.

Create security credentials for that user and “write down” the access key and secure key… especially the secure key. You will need this later to set environment variables to feed to the CodeBuild process to access the private ECR. Even better — use the Secrets Manager and store the access key and secret key in there and retrieve it with AWS CLI via a CodeBuild command (I do not cover that in this article).

Configure The CodeBuild Environment

You will need to choose a CodeBuild environment image that supports Docker. The Lambda Amazon Linux variants do not support Docker. You will want to choose an image that supports Docker In Docker (DinD) . Remember the eventual purpose here is to launch a custom-crafted Docker container within a CodeBuild running server instance against which you will run Playwright test scripts (or at least that is our purpose).

I am using the following image is it is the latest Amazon Linux 2 iteration that runs on AArch64 which happens to be Apple silicon compliant (that allows me to run the same processes on my desktop even though Amazon broke their desktop CodeBuild tool and has no plans to fix it).

aws/codebuild/amazonlinux2-aarch64-standard:3.0

Besides choosing the right CodeBuild environment to enable testing on private ECR images, you will want to set some environment variables as part of the CodeBuild specification:

• AWS_ACCOUNT_ID set to your account ID
• AWS_DEFAULT_REGION set to the region where your private ECR image (and CodeBuild) are hosted
• AWS_ACCESS_KEY_ID with the access key for the newly created user lives – the name is important as it is the default lookup for the ecr login helper that allows Docker to connect to your private ECR repo
• AWS_SECRET_ACCESS_KEY with the secret key for that user. Same thing – name is critical for ecr login, and thus Docker login to work with default values.

While you can use AWS CLI commands and/or Docker login commands with custom environment variables, using these default names that are part of the built-in lookup list for the ECR login helper saves a lot of complexity and thus potential future debugging.

Configure the Docker config.json For ECR Login

This can be done as simply as adding a mkdir ~/.docker/ command followed by an echo “{ “credsStore”: “ecr-login” }” > ~/.docker/config.json command in the CodeBuild’s buildspec.yml file. Instead I opted to include a Docker/config.json file in my repository that CodeBuild is reading from to manage my test kit and use a mkdir command followed by copy command to set up that file.

As long as your CodeBuild instance that is running all of this ends up with a config.json that tells it to use the AWS “ecr-login” helper to access your repos when executing Docker commands (docker pull, or by specifying a private ECR repo URL in your Docker compose file) the above environment variables will work.

A Note About ECR Login

On my journey toward figuring this out I found a lot of very outdated articles including AWS documentation that references some arcane black magic to get this all working. It is all related to Docker login commands getting a token for user credentials that eventually allow access to a private image, like that hosted on ECR. Most of it is dead wrong. As is anything ChatGPT 4o or Google Gemini tries to “help with” which is obviously pulling some really old shit from the Internet archives and is usually dead wrong (don’t get me started on how it doesn’t even know how to build a proper buildspec YAML file).

Bottom line – Amazon Linux 2 already has the AWS CLI and Docker login helper “ecr login” built in. No need to “configure it on your path” or to use AWS CLI ecr commands (now defunct) and pipe them into Docker login. That is all old-school crap. As long as your ~/.docker/config.json on your running CodeBuild instance is in place and setup properly the ECR login will take over and read the previously-mentioned environment variables.

Some Things You Do NOT Need

If you search the “interwebs” like I did you will see a lot of these things you don’t need:

BuildSpec Commands no needed:

aws ecr get-login-password --region $AWS_DEFAULT_REGION | docker login --username AWS --password-stdin $ACCOUNT_URI # config.json does this for you

yum install amazon-ecr-credential-helper # already installed

amazon-linux-extras enable docker # already enabled

eval $(aws ecr get-login --no-include-email --region $AWS_DEFAULT_REGION) # AWS CLI ECR commands are defunct

My Configuration

Docker config.json

{ "credsStore": "ecr-login" }

buildspec.yml

version: 0.2

env:
  variables:
    DOCKER_BUILDKIT: 1
    TEST_BASE_URL: https://localwp.storelocatorplus.com/
    WORDPRESS_DB_HOST: slp-db
    WORDPRESS_DB_PASSWORD: wordpress_password
    WPADMIN_USER: admin
    WPADMIN_PASSWORD: password
phases:
  install:
    commands:
      - mkdir ~/.docker/
      - cp ./Docker/config.json ~/.docker/config.json
  pre_build:
    commands:
      - docker-compose -f ./Docker/Composers/docker-compose-codebuild-testing.yml up -d
      - sleep 30
  build:
    commands:
      - yarn
      - npx playwright test
  post_build:
    commands:
      - docker-compose -f docker-compose.yml down

docker-compose-codebuild-testing.yml

services:
  db:
    image: public.ecr.aws/docker/library/mysql:latest
    environment:
      MYSQL_ROOT_PASSWORD: somewordpress
      MYSQL_DATABASE: wordpress_db
      MYSQL_USER: wordpress_user
      MYSQL_PASSWORD: wordpress_password

  wp:
    image: <my-account-id>.dkr.ecr.us-east-1.amazonaws.com/<my-ecr-repo-name>:develop
    ports:
      - '443:443'
    extra_hosts:
      - 'localwp.storelocatorplus.com:127.0.0.1'
      - 'host.docker.internal:host-gateway'
    volumes:
      - ./Files/php/xdebug.ini:/usr/local/etc/php/conf.d/docker-php-ext-xdebug.ini
      - ./Files/Guest/debug.log:/var/www/html/wp-content/debug.log
    environment:
      WP_HOSTURL: 'localwp.storelocatorplus.com'
      WP_HOME: 'https://localwp.storelocatorplus.com/'
      WP_SITEURL: 'https://localwp.storelocatorplus.com/'
      WORDPRESS_DEBUG: 'true'
      WORDPRESS_CONFIG_EXTRA: |
        /* Debug */
        define( 'WP_DEBUG_LOG', '/var/www/html/debug.log');
        define( 'WP_DEBUG_DISPLAY', true);
        define( 'WP_DEBUG_SCRIPT', true);
        @ini_set('display_errors',1);

The CodeBuild Environment

Summary

After banging my head against the wall for a couple of days, even harder with the “help” of AI and outdated Stack Overflow articles, the solution was far simpler than expected. Maybe Amazon will update their CodeBuild and ECR articles soon… probably not. In the meantime, hopefully this saves you some headaches and if nothing else this will serve as a reminder of what I did in a week from now when everything breaks.

Image by aytalina from Pixabay