Compile in aws instances

In this post I cover how I create a cluster to compile the linux kernel. Recently I needed to compile the linux kernel for my personal PC and it was really slow with only 2 cores, 4 threads. it ocurred to me, I didn’t need a new computer, perhaps I could compile the kernel in aws using ec2 instances. However if I had only one EC2 instance it will be the same case, unless I chose a really big instance type, but I could create a cluster of cheaper machines and have them running to compile the linux kernel, then I could copy the generated files over ssh to my current laptop and destroy all the infrastructure.

The first step is to create an ubuntu image with packer, this image will be the one used for all the nodes, and it will have all the tools needed to compile a linux kernel.

main_image.json

{
    "variables" : {
        "provs"  : "provisioners"
    },
    "builders" : [
    {
        "type" : "amazon-ebs",
        "profile" : "your-profile",
        "region": "us-east-1",
        "source_ami_filter": {
            "filters" : {
                "virtualization-type":"hvm",
                "name":"ubuntu/images/*ubuntu-bionic-18.04-amd64-server-*",
                "root-device-type" : "ebs"
            },
            "owners": ["099720109477"],
            "most_recent": true
        },
        "instance_type": "t2.micro",
        "ami_name": "vault_image",
        "ssh_username": "ubuntu",
	"tags":{"Name":"compiler-image"}
    }],
    "provisioners" : [
    {
      "type" : "shell",
      "inline": [
        "sleep 30",
        "sudo apt update",
        "sudo apt install -y ansible",
        "sudo apt install -y build-essential libncurses-dev bison flex libssl-dev libelf-dev distcc"
       ]
    }]
}

The packages needed to perform this operation are build-essential and distcc, ansible is needed in order to run an ad-hoc command to start the distccd daemon

To actually create the infrastructure I used terraform, the following code is just the main file, the rest of the code can be seen at github.com/mental12345/aws_compilation_cluster

main.tf

provider "aws" {
    profile = "your-profile"
    region = var.region
}

module "vpc" {
  source = "terraform-aws-modules/vpc/aws"
  name = "vpc"
  cidr = "10.0.0.0/16"
  azs             = ["us-east-1a", "us-east-1b"]
  private_subnets = ["10.0.1.0/24", "10.0.2.0/24"]
  public_subnets  = ["10.0.101.0/24", "10.0.102.0/24"]
  enable_ipv6 = false
  enable_nat_gateway = true
  single_nat_gateway = true
  vpc_tags = {
    Name = "cluster-vpc"
  }
}

module "ec2_farm"{
  source = "./modules/comp_farm"
  vpc_id = module.vpc.vpc_id
  private_subnets = module.vpc.private_subnets
  ins_type = var.ins_type
  ami = var.ami
  env = var.env
  ec2number = 2
  ec2min = 1
  ec2max = 3
  ssh_key = "main.pub"
}

module "ec2_main" {
  source = "./modules/ec2_main"
  vpc_id = module.vpc.vpc_id
  public_subnet = module.vpc.public_subnets[0]
  ins_type = var.ins_type
  ami = var.ami
  env = var.env
  ssh_key = "main.pub"
}

There are 3 modules, the first is to create a vpc on which all of the operations are going to be performed, the ec2_farm module creates the instances in a private subnet with the ports 22 and 3632 open. And finally the module ec2_main creates only a ec2 instance with the same ports.

Once finished the creation of the infrastructure, ansible will help running the following commands in the node instances distccd –daemon –allow main_ip main_ip being the IP of the main ec2 where I’m performing all of the operations. Also on the main ec2 export the following variable, export DISTCC_HOSTS=‘localhost IP1 IP2 IP3 IPn’ where IP1 and IP2 are the private IPs of the ec2 isntances in the cluster.

In order to actually compile the linux kernel, make -j4 CC=“distcc gcc”

Testing

1 instance t2.micro time: 21m48.775s

3 instance t2.micro time: 10m22.580s