Scaling Policy Checks

A scaling policy can include several checks all of which produce a scaling suggestion. Each check can specify its own source of metrics data and apply different strategies based on the desired outcome.

policy {
  # ...
  check "cpu_allocated_percentage" {
    source = "prometheus"
    query  = "..."

    strategy "target-value" {
      target = 70
    }
  }

  check "high-memory-usage" {
    source = "prometheus"
    query  = "..."
    group  = "memory-usage"

    strategy "threshold" {
      upper_bound = 100
      lower_bound = 70
      delta       = 1
    }
  }

  check "low-memory-usage" {
    source = "prometheus"
    query  = "..."
    group  = "memory-usage"

    strategy "threshold" {
      upper_bound = 30
      lower_bound = 0
      delta       = -1
    }
  }
}

Resolving Conflicts

The checks are all executed at the same time during a policy evaluation and they can generate conflicting scaling actions. In a scenario like this, the Autoscaler iterates over the results and chooses the safest option, which is defined as the action that results in retaining the most capacity of the resource.

In a scenario where two checks return different desired scaling directions, the following logic is applied.

In situations where the same actions are suggested, but with different counts the following logic is applied, where the count is the final desired value.

Check Grouping

The above logic for resolving conflicts only works when the checks are independent from each other. If you use the same query in multiple check blocks, or if the underlying data being queried is somehow correlated, only one check will result in a scaling action.

In the example above, the high-memory-usage and low-memory-usage checks use the same query to retrieve memory usage information. We expect that memory usage is either low or high (or neither), but never both at the same time.

Without grouping the target is never be able to reduce its count, since the possible resulting actions and the final scaling outcome can only be one of the following:

To fix this problem, the correlated checks need to be set to the same group. The Nomad Autoscaler then computes a single scaling action for the entire group by applying a slightly different logic:

ScaleNone results are ignored unless all checks in the group return it and so a group is able to ScaleIn a target even when all other checks results in no action.