Azure DevOps Build Agent Automation Part1

Building a secure linux azure managed image with Packer

Our first step in building out a secure managed image to use in automating the deployment of Azure DevOps Build Agents is to get packer installed on your machine.

Packer is a virtual machine automation platform that allows us to define our virtual machine as code and create consistent virtual machines that are able to be created the same way every time reliably.

Install Packer on Windows

My preferred method for installing packer on windows is via Scoop

To install scoop make sure your powershell execution policy is set to RemoteSigned by running the following

  Set-ExecutionPolicy RemoteSigned -scope CurrentUser

Once you have that set you can run the following to install Scoop

  Invoke-Expression (New-Object System.Net.WebClient).DownloadString('https://get.scoop.sh')

Once scoop is installed you can run the following to install the latest version of packer

  scoop install packer

You can also install packer via chocolatey if you wish

  choco install packer

Install packer on linux

you can use the following bash script to install the latest version of packer

make sure you have jq and unzip installed first

  packerVersion=$(curl -s https://checkpoint-api.hashicorp.com/v1/check/packer | jq -r -M '.current_version')
  packerUrl=https://releases.hashicorp.com/packer/$packerVersion/packer_$(echo $packerVersion)_linux_amd64.zip
  wget -O packer.zip -q $packerUrl
  unzip packer.zip
  sudo mv packer /usr/local/bin/

Create the Packer configuration file

Packer uses the JSON language as its configuration so we are going to first create a folder to hold all of our configuration.

Create a folder somewhere to work out of and change your terminal over to that directory.

Next create a json file and call it build-agent.json. The name doesn’t really matter but that is what I will be calling mine throughout the rest of this post.

inside your build-agent.json file paste the following code

{
  "variables": {
    "client_id": "",
    "client_secret": "",
    "tenant_id": "",
    "subscription_id": "",
    "managed_image_resource_group_name": "",
    "managed_image_name": "",
    "location": "",
    "vm_size": "",
    "script_path": ""
  },
  "sensitive-variables": ["client_secret"],
  "builders": [
    {
      "type": "azure-arm",
      "client_id": "{{user `client_id`}}",
      "client_secret": "{{user `client_secret`}}",
      "tenant_id": "{{user `tenant_id`}}",
      "subscription_id": "{{user `subscription_id`}}",

      "managed_image_resource_group_name": "{{user `managed_image_resource_group_name`}}",
      "managed_image_name": "{{user `managed_image_name`}}",

      "os_disk_size_gb": 100,
      "os_type": "Linux",
      "image_publisher": "Canonical",
      "image_offer": "UbuntuServer",
      "image_sku": "18.04-LTS",

      "azure_tags": {
        "Tag1": "Tag value 1",
        "Tag2": "Tag Value 2"
      },

      "location": "{{user `location`}}",
      "vm_size": "{{user `vm_size`}}"
    }
  ],
  "provisioners": [
    {
      "type":"file",
      "source": "{{template_dir}}/daemon.json",
      "destination": "~/daemon.json"
    },
    {
      "type": "shell",
      "execute_command":"chmod +x {{ .Path }}; {{ .Vars }} sudo -E sh '{{ .Path }}'",
      "inline": [
        "mkdir /devopsagents",
        "chmod 777 /devopsagents"
      ]
    },
    {
      "type":"file",
      "source": "{{template_dir}}/setup-agents.sh",
      "destination": "/devopsagents/setup-agents.sh"
    },
    {
      "type": "shell",
      "execute_command": "chmod +x {{ .Path }}; {{ .Vars }} sudo -E sh '{{ .Path }}'",
      "script": "{{template_dir}}/setupAgent.sh",
      "timeout": "45m"
    }
  ]
}

Ok thats a lot of code so lets break that down into manageable chunks.

Here we are defining all of the variables we will reuse throughout the sections below. Its pretty straightforward json name value pairs.

"variables": {
    "client_id": "c403927a-8018-4d18-8890-9bff587d59a0",
    "client_secret": "e228c40c-1a66-497f-9ffa-8ab7b8ee3b36",
    "tenant_id": "b97afca1-ba09-454a-a202-fa05928dcf0c",
    "subscription_id": "0e9d960c-8cd8-44c8-8965-3da570b98323",
    "managed_image_resource_group_name": "rg-prod-inf",
    "managed_image_name": "delete-me",
    "location": "centralus",
    "vm_size": "Standard_B1s"
}

Next is the builder which defines the virtual machine that packer will spin up to build the managed image.

Inside this builder block we have all for the different settings starting with the type and credentials needed for packer to build an azure vm

"type": "azure-arm",
"client_id": "{{user `client_id`}}",
"client_secret": "{{user `client_secret`}}",
"tenant_id": "{{user `tenant_id`}}",
"subscription_id": "{{user `subscription_id`}}",

Followed by the settings for where the managed image will be created and what it will be called

"managed_image_resource_group_name": "{{user `managed_image_resource_group_name`}}",
"managed_image_name": "{{user `managed_image_name`}}",

Next is the vm properties so os type, size, base image, ect…

"os_disk_size_gb": 100,
"os_type": "Linux",
"image_publisher": "Canonical",
"image_offer": "UbuntuServer",
"image_sku": "18.04-LTS",

Then a list of optional tags.

"azure_tags": {
  "Tag1": "TagValue1",
  "Tag2": "TagValue2"
},

And lastly we have the location and vm size. Location is the azure region so centralus or eastus2. All of the valid values for location can be found Here and the azure vm size chart can be found Here

"location": "{{user `location`}}",
"vm_size": "{{user `vm_size`}}"

The last part of this code is the Provisioners. These are what connects to the vm after the builder has created it to run scripts and do other things. The three main types of provisioner are file, shell, and shell-local for linux but there are also ones for Ansible, Chef, Puppet, Salt, and others. you can see them all here.

The first provisioner is a file provisioner to upload the docker daemon.json file we will create later to the new vm.

{
  "type":"file",
  "source": "{{template_dir}}/daemon.json",
  "destination": "~/daemon.json"
}

The second provisioner is a shell provisioner that will ssh to the new vm and execute the inline script to make a folder for our new agents to run from.

{
  "type": "shell",
  "execute_command":"chmod +x {{ .Path }}; {{ .Vars }} sudo -E sh '{{ .Path }}'",
  "inline": [
    "mkdir /devopsagents",
    "chmod 777 /devopsagents"
  ]
}

Next we have another file provisioner to upload our setup-agents.sh script we will create here soon.

{
  "type":"file",
  "source": "{{template_dir}}/setup-agents.sh",
  "destination": "/devopsagents/setup-agents.sh"
}

And Lastly we have one more shell script to execute the script we uploaded in the last file provisioner.

{
  "type": "shell",
  "execute_command": "chmod +x {{ .Path }}; {{ .Vars }} sudo -E sh '{{ .Path }}'",
  "script": "{{template_dir}}/setupAgent.sh",
  "timeout": "45m"
}

This is all that will be needed to create the Packer config for our linux build agent master image

Creating the Setup Script to configure the new VM

Now we will create the bash script used to configure all of the installed software and to configure the initial agent directories.

Here is the script that will be explained in better detail below

Create a file called setup-agent.sh and save it next to your build-agent.json file.

#! /bin/bash

# Install initial updates
add-apt-repository universe
wget -q https://packages.microsoft.com/config/ubuntu/18.04/packages-microsoft-prod.deb
dpkg -i packages-microsoft-prod.deb
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"
apt-get update
apt-get upgrade -y

# Install Apt Get Software
apt-get install -y apt-transport-https ca-certificates curl software-properties-common unzip jq python3-pip dotnet-sdk-2.1 dotnet-sdk-2.2 dotnet-sdk-3.0 powershell nodejs npm docker-ce auditd audispd-plugins

# Install NPM Packages
npm install -g newman

# Install Azure CLI
curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash

#Install Kubectl and Helm
echo 'Downloading latest kubectl'
curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
chmod +x ./kubectl
echo 'Install latest kubectl'
mv ./kubectl /usr/local/bin/kubectl
curl -L https://git.io/get_helm.sh | bash

# Install Terraform
terraformVersion=$(curl -s https://checkpoint-api.hashicorp.com/v1/check/terraform | jq -r -M '.current_version')
terraformUrl=https://releases.hashicorp.com/terraform/$terraformVersion/terraform_$(echo $terraformVersion)_linux_amd64.zip
echo 'Downloading Latest Terraform'
wget -O terraform.zip -q $terraformUrl
unzip terraform.zip
mv terraform /usr/local/bin/

# Install Packer
packerVersion=$(curl -s https://checkpoint-api.hashicorp.com/v1/check/packer | jq -r -M '.current_version')
packerUrl=https://releases.hashicorp.com/packer/$packerVersion/packer_$(echo $packerVersion)_linux_amd64.zip
echo 'Downloading Latest Packer'
wget -O packer.zip -q $packerUrl
unzip packer.zip
mv packer /usr/local/bin/

# Download Devops Agent files for later use
mkdir /devopsagents/agent1
mkdir /devopsagents/agent2
mkdir /devopsagents/agent3
latest=$(curl -s https://api.github.com/repos/Microsoft/azure-pipelines-agent/releases/latest | jq -r '.tag_name' | cut -c2-)
echo 'Downloading Latest Devops Agent'
wget -q -O azureDevOpsAgent.tar.gz https://vstsagentpackage.azureedge.net/agent/$latest/vsts-agent-linux-x64-$latest.tar.gz
tar -C /devopsagents/agent1 -xzf azureDevOpsAgent.tar.gz
tar -C /devopsagents/agent2 -xzf azureDevOpsAgent.tar.gz
tar -C /devopsagents/agent3 -xzf azureDevOpsAgent.tar.gz

# Install AWS Cli
echo 'Installing AWS Cli'
pip3 install awscli --upgrade --user

# Move Docker daemon.json
echo 'Configuring Docker'
mkdir /etc/docker
mv ~/daemon.json /etc/docker/daemon.json

# Set file ownership to root for CIS-CE-3-17
chown root:root /etc/docker/daemon.json

# Set Permissions for uploaded agent setup script
chmod 755 /devopsagents/setup-agents.sh
chmod +x /devopsagents/setup-agents.sh

# One last check for updates
apt-get update
apt-get upgrade -y

echo 'Script Cleanup'
/usr/sbin/waagent -force -deprovision+user && export HISTSIZE=0 && sync

Ok this script is fairly straightforward but here is a break down. Feel free to remove any of this if you do not need this installed on your build agent.

First we are going to install some keys and repos to our ubuntu vm for things we will need later like docker and microsoft repos

add-apt-repository universe
wget -q https://packages.microsoft.com/config/ubuntu/18.04/packages-microsoft-prod.deb
dpkg -i packages-microsoft-prod.deb
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"
apt-get update
apt-get upgrade -y

Next we install all of the software we wish to install.

apt-get install -y apt-transport-https ca-certificates curl software-properties-common unzip jq python3-pip dotnet-sdk-2.1 dotnet-sdk-2.2 dotnet-sdk-3.0 powershell nodejs npm docker-ce auditd audispd-plugins
# Install NPM Packages
npm install -g newman

Next we install the azure cli using a nice bash script microsoft provides just for that task.

curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash

Next we install the latest versions of Kubectl, Helm, Terraform, and Packer.

#Install Kubectl and Helm
echo 'Downloading latest kubectl'
curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
chmod +x ./kubectl
echo 'Install latest kubectl'
mv ./kubectl /usr/local/bin/kubectl
curl -L https://git.io/get_helm.sh | bash

# Install Terraform
terraformVersion=$(curl -s https://checkpoint-api.hashicorp.com/v1/check/terraform | jq -r -M '.current_version')
terraformUrl=https://releases.hashicorp.com/terraform/$terraformVersion/terraform_$(echo $terraformVersion)_linux_amd64.zip
echo 'Downloading Latest Terraform'
wget -O terraform.zip -q $terraformUrl
unzip terraform.zip
mv terraform /usr/local/bin/

# Install Packer
packerVersion=$(curl -s https://checkpoint-api.hashicorp.com/v1/check/packer | jq -r -M '.current_version')
packerUrl=https://releases.hashicorp.com/packer/$packerVersion/packer_$(echo $packerVersion)_linux_amd64.zip
echo 'Downloading Latest Packer'
wget -O packer.zip -q $packerUrl
unzip packer.zip
mv packer /usr/local/bin/

Next we setup the agent files using the latest agent version and save those to be setup after a vm is created from our new image.

mkdir /devopsagents/agent1
mkdir /devopsagents/agent2
mkdir /devopsagents/agent3
latest=$(curl -s https://api.github.com/repos/Microsoft/azure-pipelines-agent/releases/latest | jq -r '.tag_name' | cut -c2-)
echo 'Downloading Latest Devops Agent'
wget -q -O azureDevOpsAgent.tar.gz https://vstsagentpackage.azureedge.net/agent/$latest/vsts-agent-linux-x64-$latest.tar.gz
tar -C /devopsagents/agent1 -xzf azureDevOpsAgent.tar.gz
tar -C /devopsagents/agent2 -xzf azureDevOpsAgent.tar.gz
tar -C /devopsagents/agent3 -xzf azureDevOpsAgent.tar.gz

Next we install the aws-cli using pip as the recommended way to install.

pip3 install awscli --upgrade --user

Next we move the daemon.json file we uploaded to its proper home and set its permissions

mv ~/daemon.json /etc/docker/daemon.json
chown root:root /etc/docker/daemon.json

Next we modify the permissions for the setup-agents.sh script we uploaded earlier that will be used by terraform to configure the new virtual machine once its built from the image we are building with packer

chmod 755 /devopsagents/setup-agents.sh
chmod +x /devopsagents/setup-agents.sh

Lastly we run a command to cleanup the packer user and prepare the vm to be imaged

/usr/sbin/waagent -force -deprovision+user && export HISTSIZE=0 && sync

Create the setup-agents.sh Script

Now we will create the script to be used by terraform to register the agent instances we downloaded in the previous script.

Create a new file called setup-agents.sh next to your build-agent.json file and paste the following script into that file.

#! /bin/bash

echo 'Configuring auditd'
echo "-w /usr/bin/docker -p wa" | sudo tee -a /etc/audit/rules.d/audit.rules
echo "-w /var/lib/docker -p wa" | sudo tee -a /etc/audit/rules.d/audit.rules
echo "-w /etc/docker -p wa" | sudo tee -a /etc/audit/rules.d/audit.rules
echo "-w /lib/systemd/system/docker.service -p wa" | sudo tee -a /etc/audit/rules.d/audit.rules
echo "-w /lib/systemd/system/docker.socket -p wa" | sudo tee -a /etc/audit/rules.d/audit.rules
echo "-w /etc/default/docker -p wa" | sudo tee -a /etc/audit/rules.d/audit.rules
echo "-w /etc/docker/daemon.json -p wa" | sudo tee -a /etc/audit/rules.d/audit.rules
echo "-w /usr/bin/docker-containerd -p wa" | sudo tee -a /etc/audit/rules.d/audit.rules
echo "-w /usr/bin/docker-runc -p wa" | sudo tee -a /etc/audit/rules.d/audit.rules
echo "Restarting autitd"
sudo systemctl restart auditd

# Configure Agents
agents="agent1 agent2 agent3"
echo "Changing /devopsagents permissions"
sudo chown -R buildadmin:buildadmin /devopsagents
for agent in $agents; do
  echo "Setting up Agent: $agent"
  cd /devopsagents/$agent
  ./config.sh --unattended --url https://dev.azure.com/xomelabs --auth pat --token $2 --pool $1 --agent $agent --replace --work _work --acceptTeeEula
  sudo bash ./svc.sh install
  sudo bash ./svc.sh start
done

Lets break that down a bit and see whats going on with this

First we are going to configure auditd to audit events around any docker activity. If you didn’t install auditd in your setup-agent.sh script then you can skip this part

echo 'Configuring auditd'
echo "-w /usr/bin/docker -p wa" | sudo tee -a /etc/audit/rules.d/audit.rules
echo "-w /var/lib/docker -p wa" | sudo tee -a /etc/audit/rules.d/audit.rules
echo "-w /etc/docker -p wa" | sudo tee -a /etc/audit/rules.d/audit.rules
echo "-w /lib/systemd/system/docker.service -p wa" | sudo tee -a /etc/audit/rules.d/audit.rules
echo "-w /lib/systemd/system/docker.socket -p wa" | sudo tee -a /etc/audit/rules.d/audit.rules
echo "-w /etc/default/docker -p wa" | sudo tee -a /etc/audit/rules.d/audit.rules
echo "-w /etc/docker/daemon.json -p wa" | sudo tee -a /etc/audit/rules.d/audit.rules
echo "-w /usr/bin/docker-containerd -p wa" | sudo tee -a /etc/audit/rules.d/audit.rules
echo "-w /usr/bin/docker-runc -p wa" | sudo tee -a /etc/audit/rules.d/audit.rules
echo "Restarting autitd"
sudo systemctl restart auditd

Next we configure the agents by looping through a list of agents to create and running the unattended configuration.

# Configure Agents
agents="agent1 agent2 agent3"
echo "Changing /devopsagents permissions"
sudo chown -R buildadmin:buildadmin /devopsagents
for agent in $agents; do
  echo "Setting up Agent: $agent"
  cd /devopsagents/$agent
  ./config.sh --unattended --url https://dev.azure.com/xomelabs --auth pat --token $2 --pool $1 --agent $agent --replace --work _work --acceptTeeEula
  sudo bash ./svc.sh install
  sudo bash ./svc.sh start
done

Building the managed image

Now that we have all the files built we should be ready to create the managed image we will be using to create the build agents from.

to do this first we make sure we have a terminal in the same folder that houses our build-agent.json file and run the following command to check if your syntax is correct.

Make sure to replace all of the variables listed here with correct values. See here if you need to create a service principle to use with Packer.

packer validate -var 'client_id=replaceme' -var 'client_secret=replaceme' -var 'tenant_id=replaceme' -var 'subscription_id=replaceme' -var 'managed_image_resource_group_name=replaceme' -var 'managed_image_name=replaceme' -var 'location=replaceme' -var 'vm_size=replaceme' build-agent.json

This will validate that the syntax is correct. Packer does require you to give the variables a value and later on in the last post I will go over how to set this all up in azure devops Pipelines.

Next we build the image by changing validate to build like so

packer build -var 'client_id=replaceme' -var 'client_secret=replaceme' -var 'tenant_id=replaceme' -var 'subscription_id=replaceme' -var 'managed_image_resource_group_name=replaceme' -var 'managed_image_name=replaceme' -var 'location=replaceme' -var 'vm_size=replaceme' build-agent.json

Once this is complete you will have a new automated Azure DevOps Linux Build Agent Managed Image.

Kass Eisenmenger
comments powered by Disqus