How I Replaced Self-Managed GitLab Runners With Lambda MicroVMs

In my current project, most of our CI/CD pipelines run on shared GitLab runners managed by central platform team. We have our own runners for any pipeline related to database since centrally managed runners can’t connect to our private RDS. Now these self-managed runners remain idle for most of the times since we don’t run database operations frequently.

This is a perfect use case for Lambda MicroVMs and in this post we’ll see how we can replace GitLab Runners and use Lambda MicroVMs to perform database operations.

Step 1: Create an Aurora Postgres 15 cluster with no public access.

Step 2: Fork https://gitlab.com/parent-child-pipeline-test/lambda_microvm and clone it. Now run

export VPC_SUBNET_IDS="subnet-xxx,subnet-yyy" # Subnet used by Aurora Postgres
export SECURITY_GROUP_IDS="sg-xxx" # Security group which allows access to Aurora Postgres
chmod +x scripts/setup_aws.sh
./scripts/setup_aws.sh

This script will create our MicroVM image

and a network connector which enables our MicroVM to connect to our private RDS instance.

Step 3: In your GitLab project, setup all these environment variables

and run the pipeline which will execute a sample sql script given in the repo.

You can run your own script by changing the value for SCRIPT_PATH variable

SCRIPT_PATH: "scripts/sql/sample_setup.sql"

Once pipeline job executes successfully, you can verify the records by connecting to your database. For this you can launch an EC2 instance in your VPC’s public subnet and install postgresql15 package using

dnf install -y postgresql15

That’s all. Now we don’t need a self-managed gitlab runner to run database jobs as MicroVM can do this job perfectly.