Launching AWS EC2 Slave Instances for Jenkins

Using AWS EC2 slave instances on Jenkins allows you to run your Jenkins jobs on a remote server, such as a separate AWS EC2 instance. This not only provides a performance boost by deploying additional instances whenever necessary, but it also enhances the security of your Jenkins jobs. By leveraging the power of AWS, the plugin enables you to efficiently manage your Jenkins builds and scale your infrastructure to meet your needs.

Outcome of this Article:

Ability to launch Jenkin jobs in AWS EC2 instances, and scale when in need.

 

Steps:

This article assumes that Jenkins is already installed.

1) Download AWS-EC2 Plugin

2) Generate IAM for Jenkins

- a. From AWS console, click `IAM > Users > Add Users`

 

- b. Fill in name and select `Attach Policies Directly > AmazonEC2FullAccess`.

- c. Select `Next` and complete IAM User Creation

- d. On IAM main page, select `'created IAM User' > Security Credentials > Create Access Key`

- e. On Create Access Key page, select `Other` option, and then click `Next`

- f. Finish creating access key by selecting `Create Access Key`

- g. Copy `Access Key` and `Secret Access Key` and select `Done`

- h. On Jenkins page, click `Manage Jenkins > Manage Credentials`

- i. On Manage Credentials page, click `(global) > Add Credentials`

- j. Using AWS IAM credentials copied from step g), fill as follows:

• Kind: AWS Credentials
• ID: aws.jenkins.iam.credential (or any other id of choice)
• Description: AWS Jenkins IAM Credential (or any matching description of choice)
• Access Key ID: `Access Key` from step g)
• Secret Access Key: `Secret Access Key` from step g)

- k. hit `Create`.

 

3) Generate SSH Key Pairs for Jenkins

 

 

4) Configure Amazon EC2 Plugin

- a. On Jenkins page, click `Manage Jenkins > Manage Nodes and Clouds`

- b. Click `Configure Clouds`

- c. Fill in as follows. The following is what author used to configure EC2 plugin[1]. Sensitive information is omitted.

Full Init Script code

# Install Java
sudo apt-get update
sudo apt install -y default-jdk
sudo apt install -y default-jre
# Install Docker
sudo apt-get install -y ca-certificates curl gnupg
sudo mkdir -m 0755 -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
# Manage docker as non-root user
# https://www.digitalocean.com/community/questions/how-to-fix-docker-got-permission-denied-while-trying-to-connect-to-the-docker-daemon-socket
sudo groupadd docker
sudo usermod -aG docker ${USER}
su -s ${USER}
sudo chown "$USER":"$USER" /home/"$USER"/.docker -R
sudo chmod g+rwx "$HOME/.docker" -R

5) Validate

 

FAQ:

Problem 1: Testing Connection Issues when Configuring Amazon EC2 Plugin

When testing for AWS EC2 connection under `configure cloud` after following this tutorial, it's returning the following error:

To solve this, make sure ssh private keys being inserted to credential starts with `-----BEGIN RSA PRIVATE KEY-----` and ends with `-----END RSA PRIVATE KEY----`. This is done by clicking `EC2 > Key Pairs > Create Key Pair`, and selecting the following before creation:

 

References:

[1] Cloud4DevOps. Jenkins Master to Launch AWS EC2 instances as Slaves using EC2 plugin. Youtube. https://www.youtube.com/watch?v=1XI9_4umWVk