Scale back GitHub runner prices by leveraging EC2 spot situations

We realized the exhausting manner, that GitHub Actions is getting costly when utilizing GitHub-hosted runners. Again in 2023, we determined to construct an answer for self-hosted runners on AWS to cut back prices. Just a few months later, we launched HyperEnv to the general public. Over time, we improved our answer step-by-step. With the launch of model 2.9.0 we achieved one other necessary milestone: leverage EC2 spot situations to make the most of unused EC2 capability within the AWS cloud. Within the following, I’ll share our learnings alongside the way in which.

EC2 Spot vs. On-Demand

There are three totally different pricing fashions for digital machines on AWS.

• On-Demand is the default, relying on the occasion kind, you pay an hourly charge, which is normally charged by the second. For instance, an m5.giant occasion (2 vCPUs and eight GiB of reminiscence) prices you $0.0960 per hour in area us-east-1.
• Spot grants reductions on unused capability in AWS’s knowledge facilities. The value modifications relying on the utilization of the info facilities. Whereas penning this, an m5.giant occasion is about $0.0348 in us-east-1 which is a 63.78% saving in contrast with on-demand. Right here is the catch: AWS reserves the best to terminate a spot occasion after 2 minutes discover, which AWS calls a spot interruption.
• Financial savings Plans are a easy deal: you decide to a particular use of EC2 situations and get a reduction from AWS. The method works finest for static workloads that may be deliberate for 1 and even 3 years upfront.

So, working GitHub runners on spot as a substitute of on-demand situations cuts down infrastructure prices by about 60%. Nevertheless, there are a number of caveats to contemplate.

Consideration: Ephemeral Runners

There are three approaches for internet hosting GitHub Runners on AWS:

• Lengthy-running: Launch an EC2 occasion, set up and begin GitHub runner. Preserve the occasion working 24/7.
• Auto-scaled: Launch and terminate EC2 situations relying on the variety of ready jobs.
• Ephemeral: Launch an EC2 occasion for each job. Terminate the machine after finishing the job.

A typical construct job takes 5 to fifteen minutes to finish. Due to this fact, ephemeral runners require a spot occasion for a short while solely, which reduces the danger of getting interrupted. That’s as a result of AWS takes the runtime of a spot occasion into consideration, when deciding on a spot occasion to interrupt to unlock capability.

So, ephemeral runners are an ideal match for EC2 spot situations.

Consideration: Fallback to On-Demand

Relying on the utilization of the info middle, the provision of spot situations could be restricted. It’s not unlikely, that AWS rejects a request to launch a spot occasion as a consequence of no capability.

When using spot situations, it’s important to bear in mind that the provision of those situations can fluctuate primarily based on demand. In some circumstances, AWS may not have sufficient capability accessible, leading to your request to launch a spot occasion being rejected as a consequence of no accessible capability.
To mitigate this threat, think about implementing a fallback technique the place you mechanically swap to launching on-demand situations when spot situations should not accessible.

One strategy to obtain that is by utilizing a mix of AWS Auto Scaling and CloudWatch metrics to observe the provision of spot situations. Based mostly on these metrics, your Auto Scaling group can scale up or down accordingly, making certain that your GitHub runners have entry to the mandatory sources.
By implementing such a method, you may reduce the affect of spot occasion unavailability and preserve a secure and dependable infrastructure on your GitHub runners.

This method ensures excessive availability and lets you make the most of the associated fee financial savings supplied by spot situations whereas minimizing the danger related to their availability.

Here’s what to do in such a state of affairs:

1. Attempt to launch the spot occasion in one other availability zone, which suggests in one other knowledge middle which may present spot capability.
2. Fallback to launching an on-demand occasion.

Consideration: Does the GitHub workflow stand up to an interruption?

Whereas likelihood is low that an ephemeral runner that runs on an EC2 spot occasion for a couple of minutes will get interrupted, it’s not zero. For a lot of GitHub workflows, it’s not a difficulty when a job will get caught and cancelled as a consequence of a spot interruption. A job like working unit exams, linting code, or constructing artifacts can sometimes be restarted with none unintended effects. Nevertheless, there are jobs the place an interruption may trigger in corrupt state or undesirable unintended effects. For instance, interrupting the run of terraform apply may trigger the next job to fail as a result of the Terraform state continues to be locked.

Due to this fact, with the ability to configure whether or not a spot occasion needs to be used on the job stage is important.

Structure for GitHub Actions working on EC2 spot situations

So what would an AWS structure for launching ephemeral GitHub runners on EC2 spot situations with a fallback to on-demand situations appear to be? The next diagram exhibits the answer that we ended up with for HyperEnv.

1. API Gateway receives an HTTP request from GitHub.
2. API Gateway invokes the Lambda perform named webhook.
3. The Lambda perform webhook verifies the incoming webhook occasion.
4. The Lambda perform webhook begins an execution of the Step Operate runner-orchestrator.
5. The Step Operate invokes the Lambda perform shopper which tries to launch a spot occasion.
6. In case a spot occasion is just not accessible within the chosen availability zone, the Step Operate retries launching a spot occasion in one other availability zone by calling the Lambda perform shopper a second time.
7. In case it’s not doable to launch a spot occasion, the Step Operate continues and calls the Lambda perform shopper once more to launch an on-demand occasion.

Give HyperEnv a attempt!

Do you favor a production-ready and well-maintained answer as a substitute of constructing this by yourself? Try our answer for self-hosted GitHub Actions runners for AWS: HyperEnv.