Fallback to on-demand EC2 situations if spot capability is unavailable

In current months, I used to be once more reminded that EC2 spot capability just isn’t at all times obtainable. For years, I used to be in search of a security internet for my spot-based Auto Scaling Teams (ASGs). If spot capability is unavailable, launch on-demand EC2 situations and substitute them with spot as quickly as spot capability is again. After many proofs of idea, I need to share my strategy to the issue.

I assume your present ASG is configured to unfold the load throughout as many availability zones and occasion sorts as attainable. Apart from that, I encourage you to allow Capability Rebalancing to deal with spot interruptions. Apart from that, add the next sources to implement the on-demand security internet:

• Fallback ASG to launch on-demand EC2 situations
• Two step scaling insurance policies to scale up/down the fallback ASG
• Two CloudWatch alarms to set off the scaling insurance policies

Configure present ASG

Allow your present ASG to emit the CloudWatch metrics GroupInServiceInstances and GroupDesiredCapacity.

In CloudFormation:

SpotAutoScalingGroup:
  Sort: 'AWS::AutoScaling::AutoScalingGroup'
  Properties:
    
    CapacityRebalance: true
    MaxSize: 10
    MinSize: 2
    MixedInstancesPolicy:
      
      InstancesDistribution:
          OnDemandAllocationStrategy: prioritized
          OnDemandBaseCapacity: 0
          OnDemandPercentageAboveBaseCapacity: 0
          SpotAllocationStrategy: 'capacity-optimized-prioritized'
    MetricsCollection:
    - Granularity: 1Minute
      Metrics:
      - GroupInServiceInstances
      - GroupDesiredCapacity

Configure extra fallback ASG

Add a brand new ASG to spin up on-demand capability. Use the identical launch template/configuration as your spot ASG.

FallbackAutoScalingGroup:
  Sort: 'AWS::AutoScaling::AutoScalingGroup'
  Properties:
    
    MetricsCollection:
    - Granularity: 1Minute
      Metrics:
      - GroupInServiceInstances
      - GroupDesiredCapacity
    MaxSize: 10 
    MinSize: 0

Create CloudWatch alarms to set off auto-scaling

The trick is to make use of the next system to calculate the variety of situations that have to be added/faraway from the fallback ASG:

desired spot-working spot-desired fallback

The next desk lets you perceive the system with some examples:

The next logic is required to work with the results of the system:

If outcome > 0: enhance the specified capability of the fallback ASG by outcome.
Else if outcome : lower the specified capability of the fallback ASG by outcome.
Else: do nothing.

The logic could be carried out with CloudWatch alarms and step scaling insurance policies.

CloudWatch alarms set off the step scaling insurance policies to scale up/down the fallback ASG. To cut back noise brought on by auto-scaling actions within the spot ASG, I configured the alarms solely to fireplace if the system is detrimental/optimistic thrice in a row. The next two CloudWatch alarms are largely similar, aside from the ComparisonOperator.

FallbackScaleUpAlarm:
  Sort: 'AWS::CloudWatch::Alarm'
  Properties:
    AlarmActions:
    - !Ref FallbackScaleUp
    ComparisonOperator: GreaterThanThreshold
    EvaluationPeriods: 3 
    Threshold: 0         
    TreatMissingData: notBreaching
    Metrics:
    - Id: working 
      Label: working
      MetricStat:
        Metric:
          Namespace: 'AWS/AutoScaling'
          MetricName: GroupInServiceInstances
          Dimensions:
          - Title: AutoScalingGroupName
            Worth: !Ref SpotAutoScalingGroup
        Interval: 60
        Stat: Most
      ReturnData: false
    - Id: desired  
      Label: desired
      MetricStat:
        Metric:
          Namespace: 'AWS/AutoScaling'
          MetricName: GroupDesiredCapacity
          Dimensions:
          - Title: AutoScalingGroupName
            Worth: !Ref SpotAutoScalingGroup
        Interval: 60
        Stat: Most
      ReturnData: false
    - Id: desiredfallback 
      Label: desiredfallback
      MetricStat:
        Metric:
          Namespace: 'AWS/AutoScaling'
          MetricName: GroupDesiredCapacity
          Dimensions:
          - Title: AutoScalingGroupName
            Worth: !Ref FallbackAutoScalingGroup
        Interval: 60
        Stat: Most
      ReturnData: false
    - Expression: 'desired-running-desiredfallback' 
      Id: e1
      Label: 'fallback'
      ReturnData: true
FallbackScaleDownAlarm:
  Sort: 'AWS::CloudWatch::Alarm'
  Properties:
    AlarmActions:
    - !Ref FallbackScaleDown
    ComparisonOperator: LessThanThreshold
    EvaluationPeriods: 3 
    Threshold: 0         
    TreatMissingData: notBreaching
    Metrics:
      

In a great world, we might use the results of the system to alter the specified capability instantly. Bear in mind, the system calculates the situations that have to be added (optimistic values)/eliminated (detrimental values) from the fallback ASG. Sadly, we should take a slight detour by way of a step scaling coverage.

1. The CloudWatch alarm triggers the step scaling coverage with the system outcome.
2. The step scaling coverage interprets the acquired worth right into a change in capability (adjustment)…
3. …and updates the specified depend of the ASG.

You’ll be able to configure how the step scaling coverage transforms the worth from CloudWatch right into a change in capability by defining step changes. A step is outlined by a decrease and higher sure and a change in capability.

I exploit the next steps to translate from the system outcome to a change in desired capability:

You’ll be able to outline as much as 20 changes per step scaling coverage.

FallbackScaleUp:
  Sort: 'AWS::AutoScaling::ScalingPolicy'
  Properties:
    AdjustmentType: ChangeInCapacity
    AutoScalingGroupName: !Ref FallbackAutoScalingGroup
    EstimatedInstanceWarmup: 300
    MetricAggregationType: Common
    PolicyType: StepScaling
    StepAdjustments: 
    - MetricIntervalLowerBound: 0
      MetricIntervalUpperBound: 2
      ScalingAdjustment: 1
    - MetricIntervalLowerBound: 2
      MetricIntervalUpperBound: 3
      ScalingAdjustment: 2
    - MetricIntervalLowerBound: 3
      MetricIntervalUpperBound: 4
      ScalingAdjustment: 3
    - MetricIntervalLowerBound: 4
      MetricIntervalUpperBound: 5
      ScalingAdjustment: 4
    - MetricIntervalLowerBound: 5
      MetricIntervalUpperBound: 10
      ScalingAdjustment: 5
    - MetricIntervalLowerBound: 10
      MetricIntervalUpperBound: 25
      ScalingAdjustment: 10
    - MetricIntervalLowerBound: 25
      ScalingAdjustment: 25
FallbackScaleDown:
  Sort: 'AWS::AutoScaling::ScalingPolicy'
  Properties:
    AdjustmentType: ChangeInCapacity
    AutoScalingGroupName: !Ref FallbackAutoScalingGroup
    EstimatedInstanceWarmup: 300
    MetricAggregationType: Common
    PolicyType: StepScaling
    StepAdjustments: 
    - MetricIntervalUpperBound: 0
      MetricIntervalLowerBound: -2
      ScalingAdjustment: -1
    - MetricIntervalUpperBound: -2
      MetricIntervalLowerBound: -3
      ScalingAdjustment: -2
    - MetricIntervalUpperBound: -3
      MetricIntervalLowerBound: -4
      ScalingAdjustment: -3
    - MetricIntervalUpperBound: -4
      MetricIntervalLowerBound: -5
      ScalingAdjustment: -4
    - MetricIntervalUpperBound: -5
      ScalingAdjustment: -5

Abstract

The next graph exhibits the fallback in motion:

The crimson line exhibits the specified spot, the orange line exhibits the working spot, and the inexperienced line exhibits the working fallback.

• 9:25 two spot situations are desired and working (desired spot = 4; working spot = 2).
• 9:27 one extra spot occasion is requested (desired spot = 3).
• 9:32 spot capability not obtainable; one fallback occasion is working (desired spot = 3; working spot = 2; working fallback = 1)
• 9:35 one extra spot occasion is requested (desired spot = 4)
• 9:40 spot capability not obtainable; two fallback situations are working (desired spot = 4; working spot = 2; working fallback = 2)

As you possibly can see, it takes round 5 minutes for on-demand capability to exchange the lacking spot capability. That is brought on by the three x 1-minute delay added by the CloudWatch alarm configuration and the delay launched by beginning an EC2 occasion earlier than it influences the GroupInServiceInstances metric. You might take away as much as 2 minutes of delay by adjusting the CloudWatch alarms to solely await one or two threshold violations earlier than triggering the scaling motion.