How To Parse JSON Objects With Bash

Working on the updates to the DevOps stack for the Store Locator Plus® SaaS, I found the need to parse a JSON object with bash. Many solutions you find suggest using jq, which is an add on package for most operating systems including MacOS. There are also ways to do this with python, ruby, and a dozen other inline processors. However, that is a lot of overhead for what should be a simple task.

AWS Secrets Manager JSON Objects

In my case I am trying to parse the JSON output from the AWS CLI when pulling a value from AWS Secrets Manager. The first step is to simplify the JSON object that is returned as well as to escape the JSON and store it in a text string representation of the JSON object.

Here is an example of using AWS CLI with Secrets Manager for a user profile lance.cleveland that was already logged in with aws sso login prior to the command execution:

aws secretsmanager get-secret-value --profile lance.cleveland --secret-id my_aws_secret --query SecretString --output text | grep -o '"username":"[^"]*' | grep -o '[^"]*$'

The –query parameter tells the system to only return the SecretString part of the object. Otherwise the entire Secrets object is returned along with the metadata. It looks similar to this:

–output text strips out the escape characters (\) from the SecretString making it easier to parse with grep.

Grep Helps Parse JSON Objects With Bash

Here is how to extract the username value from that key.

aws secretsmanager get-secret-value --profile lance.cleveland --secret-id my_aws_secret --query SecretString --output text | grep -o '"username":"[^"]*' | grep -o '[^"]*$'

Replace the “username” in the first grep command with “password” to get the password.

Add SED to Convert Special Characters in HTTPS git Requests

Since these secrets are being used as part of an HTTPS git request to clone repos, we cannot go passing special characters to git in the username/password URL. Any reserved character will need to be percent encoded. Here we encode the = and / common in most AWS IAM https passwords:

password=$(aws secretsmanager get-secret-value --profile lance.cleveland --secret-id my_aws_secret --query SecretString --output text | grep -o '"password":"[^"]*' | grep -o '[^"]*$' | sed 's/=/%3D/g' | sed 's/\//%2F/g')

A ChatGPT Summarization of How To Parse JSON Objects With Bash

After trying to get help with ChatGPT, which did not prove very helpful, I found this solution on Stack Overflow combined with some prior knowledge of AWS Secrets Manager and Bash script and command techniques. Since ChatGPT stumbled around the neighborhood like a drunken toddler after a football game, I decided to clue it in to this solution. I don’t think services like ChatGPT are yet taking in user inputs during conversations and feeding them back into the public knowledge, but I shared anyway. It does provide a good recap of what I found. And eventually the GPT engines will steal the content of this article for a future revision and share the knowledge her as its own. Progress, I guess.

Me: Regarding our prior conversation parsing aws secrets manager JSON responses. Many operating systems, including MacOS do not include the jq parser by default. Rather than installing them, you can add parameters the AWS CLI Secrets Manager command to return an escaped string of the JSON object stored in the secret. This can be passed to grep to extract a specific key. Here is an example of using AWS CLI with Secrets Manager for a user profile lance.cleveland that was already logged in with aws sso login prior to the command execution: aws secretsmanager get-secret-value –profile lance.cleveland –secret-id my_aws_secret –query SecretString –output text | grep -o ‘”username”:”[^”]*’ | grep -o ‘[^”]*$’

ChatGPT:

You’ve described an effective method for extracting specific values from JSON responses from AWS Secrets Manager without the need for additional tools like jq. Using the AWS CLI’s --query and --output parameters, you can directly get an escaped string of the JSON object. Then, by employing grep, you can extract the value associated with a specific key.

Your example command uses the AWS CLI with the lance.cleveland profile to retrieve a secret, outputting the text directly and parsing it with grep to extract the username. This method is particularly useful for environments where installing additional tools like jq is not preferable or possible. It leverages the powerful text processing capabilities of grep to extract the required information from the JSON string. This approach can be adapted to retrieve other values from the Secrets Manager response by modifying the key name in the grep command.

Image by Dean Moriarty from Pixabay