A Go-Chaos implementation for AWS infrastructure with Github Actions

Chaos engineering, although a relative new concept, is important for organizations to understand it. Many companies can benefit from chaos engineering, from small organizations that have simple applications to really big ones that have multi region applications and services. This post serves as a guide for chaos engineering with Go-chaos and Github Actions.

Modern approach to chaos engineering

There are several tools already in existence, however, I really think these tools are difficult to setup, to operate and to maintain. In response to that, Go-Chaos was developed. Go-Chaos is an app that let’s you create experiments as code, the format chosen for the experiments is Hashicorp Configuration Language (HCL), support several cloud providers out of the box and can limit the blast radius of the impacted resources, and the most important feature of all, is driven by simplicity.

Creating a directory and repository

Create the following structure in your local computer and push it to a GitHub repository

.
├── aws_ec2
│   ├── config.hcl
├── .github
│   └── workflows
│       └── main_aws.yaml
└── README.md

Creating a chaos experiment for AWS.

For this example, a single EC2 instance and a Lambda function will be affected. Modify the file called config.hcl inside the aws_ec2 directory:

app = "Login"
description = "Chaos Experiment for login app" 

job "aws" "ec2" {
    region = "us-east-1"
    config {
        tag = "Name:login-app-prod" 
        chaos = "terminate"         
        count = 1                   
    }
}

job "aws" "lambda" {
    region = "us-east-1"
    config {
        tag = "Name:resolution-scale-prod" 
        chaos = "stop"              
        count = 1                   
    }
}

Creating the workflow for GitHub Actions

Modify the main_aws.yaml file inside the directory .github/workflows/, this file will be the one to execute the chaos engineering for us.

name: Main Experiment on AWS. 
on: workflow_dispatch
env:
  go-chaos-version: "v0.1.1"
jobs: 
  github-go-chaos:
    runs-on: ubuntu-latest
    steps:
      - name: Check out repository code
        uses: actions/checkout@v3
      
      - name: Aws Credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-2

      - name: Install go-chaos
        run: | 
          echo "Installing go-chaos"
          wget https://github.com/gochaos-app/go-chaos/releases/download/${{ env.go-chaos-version }}/go-chaos-linux-amd64
          chmod +x go-chaos-linux-amd64 && mv go-chaos-linux-amd64 /usr/local/bin/go-chaos 
          go-chaos -h
      
      - name: go-chaos validate
        working-directory: aws_ec2
        run: go-chaos validate config.hcl
        shell: bash
        
      - name: go-chaos destroy
        working-directory: aws_ec2
        run: go-chaos destroy config.hcl

The file contains a single job called github-go-chaos and the sequence is actually really simple.

1. Clones the repository.
2. Set up AWS credentials.
3. Install Go-Chaos, version v0.1.1.
4. Validates the file.
5. Executes the instructions on the file.

A screenshot on the output is shown:

Please take a look at the repo and more examples at go-chaos-experiments