Preparing classic and VPC clusters for autoscaling

Understanding scale-up and scale-down

The cluster autoscaler periodically scans the cluster to adjust the number of worker nodes within the worker pools that it manages in response to your workload resource requests and any custom settings that you configure, such as scanning intervals. Every minute, the cluster autoscaler checks for the following situations.

• Pending pods to scale up: A pod is considered pending when insufficient compute resources exist to schedule the pod on a worker node. When the cluster autoscaler detects pending pods, the autoscaler scales up your worker nodes evenly across zones to meet the workload resource requests.
• Underutilized worker nodes to scale down: By default, worker nodes that run with less than 50% of the total compute resources that are requested for 10 minutes or more and that can reschedule their workloads onto other worker nodes are considered underutilized. If the cluster autoscaler detects underutilized worker nodes, it scales down your worker nodes one at a time so that you have only the compute resources that you need. If you want, you can customize the default scale-down utilization threshold of 50% for 10 minutes.

Scanning and scaling up and down happens at regular intervals over time, and depending on the number of worker nodes might take a longer period of time to complete, such as 30 minutes.

The cluster autoscaler adjusts the number of worker nodes by considering the resource requests that you define for your deployments, not actual worker node usage. If your pods and deployments don't request appropriate amounts of resources, you must adjust their configuration files. The cluster autoscaler can't adjust them for you. Also, keep in mind that worker nodes use some compute resources for basic cluster functionality, default and custom add-ons, and resource reserves.

What does scaling up and down look like?

In general, the cluster autoscaler calculates the number of worker nodes that your cluster needs to run its workload. Scaling the cluster up or down depends on many factors, including the following.

• The minimum and maximum worker node size per zone that you set.
• Your pending pod resource requests and certain metadata that you associate with the workload, such as anti-affinity, labels to place pods only on certain flavors, or pod disruption budgets.
• The worker pools that the cluster autoscaler manages, potentially across zones in a multizone cluster.

For more information, see the Kubernetes Cluster Autoscaler FAQs for How does scale-up work? and How does scale-down work?.

Can I change how scale-up and scale-down work?

You can customize settings or use other Kubernetes resources to affect how scaling up and down work.

Scale-up

Customize the cluster autoscaler ConfigMap values such as scanInterval, expander, skipNodes, or maxNodeProvisionTime. Review ways to overprovision worker nodes so that you can scale up worker nodes before a worker pool runs out of resources. You can also set up Kubernetes pod budget disruptions and pod priority cutoffs to affect how scaling up works.

Scale-down

Customize the cluster autoscaler ConfigMap values such as scaleDownUnneededTime, scaleDownDelayAfterAdd, scaleDownDelayAfterDelete, or scaleDownUtilizationThreshold.

Can I increase the minimum size per zone to trigger a scale up my cluster to that size?

No, setting a minSize does not automatically trigger a scale-up. The minSize is a threshold so that the cluster autoscaler does not scale to fewer than a certain number of worker nodes per zone. If your cluster does not yet have that number per zone, the cluster autoscaler does not scale up until you have workload resource requests that require more resources. For example, if you have a worker pool with one worker node per three zones (three total worker nodes) and set the minSize to 4 per zone, the cluster autoscaler does not immediately provision an additional three worker nodes per zone (12 worker nodes total). Instead, the scale-up is triggered by resource requests. If you create a workload that requests the resources of 15 worker nodes, the cluster autoscaler scales up the worker pool to meet this request. Now, the minSize means that the cluster autoscaler does not scale down to fewer than four worker nodes per zone even if you remove the workload that requests the amount.

How is this behavior different from worker pools that are not managed by the cluster autoscaler?

When you create a worker pool, you specify how many worker nodes per zone it has. The worker pool maintains that number of worker nodes until you resize or rebalance it. The worker pool does not add or remove worker nodes for you. If you have more pods than can be scheduled, the pods remain in pending state until you resize the worker pool. When you enable the cluster autoscaler for a worker pool, worker nodes are scaled up or down in response to your pod spec settings and resource requests. You don't need to resize or rebalance the worker pool manually.

Following scalable deployment practices

Make the most out of the cluster autoscaler by using the following strategies for your worker node and workload deployment strategies. For more information, see the Kubernetes Cluster Autoscaler FAQs.

Try out the cluster autoscaler with a few test workloads to get a good feel for how scale-up and scale-down work, you might want to configure, and any other aspects that you might want, like overprovisioning worker nodes or limiting apps. Then, clean up your test environment and plan to include these custom values and additional settings with a fresh installation of the cluster autoscaler.

Rebalancing or resizing autoscaled worker pools

Before you can rebalance or resize your worker pool, you must remove the worker pool from the autoscaler configmap to disable autoscaling.

1. Edit iks-ca-configmap and disable the worker pool that you want to resize or rebalance by removing it from the workerPoolsConfig.json section.

   kubectl edit cm -n kube-system iks-ca-configmap
   

   Example output

   apiVersion: v1
   data:
     workerPoolsConfig.json: |
       [
        {"name": "","minSize": 1,"maxSize": 2,"enabled":false}
       ]
   kind: ConfigMap
   

2. Save the iks-ca-configmap.

3. Resize or rebalance your worker pool.

4. Optional Update your VPC worker nodes.

5. Add the worker pool to the iks-ca-configmap.

   kubectl edit cm -n kube-system iks-ca-configmap
   

   Example

   apiVersion: v1
   data:
     workerPoolsConfig.json: |
       [
        {"name": "<worker_pool>","minSize": 1,"maxSize": 2,"enabled":false}
       ]
   kind: ConfigMap
   

Preparing classic or VPC Gen 2 clusters for autoscaling

Before you install the IBM Cloud cluster autoscaler add-on, you can set up your cluster to prepare the cluster for autoscaling.

The cluster autoscaler add-on is not supported for baremetal worker nodes.

1. Before you begin, Install the required CLI and plug-ins.

   • IBM Cloud CLI (ibmcloud)
   • IBM Cloud Kubernetes Service plug-in (ibmcloud ks)
   • IBM Cloud Container Registry plug-in (ibmcloud cr)
   • Kubernetes (kubectl)

2. Create a standard cluster.

3. Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.

4. Confirm that your IBM Cloud Identity and Access Management credentials are stored in the cluster. The cluster autoscaler uses this secret to authenticate credentials. If the secret is missing, create it by resetting credentials.

   kubectl get secrets -n kube-system | grep storage-secret-store
   

5. Plan to autoscale a worker pool other than the default worker pool, because the default worker pool has system components that can prevent automatically scaling down. Include a label for the worker pool so that you can set node affinity for the workloads that you want to deploy to the worker pool that has autoscaling enabled. For example, your label might be app: nginx. Choose from the following options:

   • Create a VPC or classic worker pool other than the default worker pool with the label that you want to use with the workloads to run on the autoscaled worker pool.
   • Add the label to an existing worker pool other than the default worker pool.

6. Confirm that your worker pool has the necessary labels for autoscaling. In the output, you see the required ibm-cloud.kubernetes.io/worker-pool-id label and the label that you previously created for node affinity. If you don't see these labels, add a worker pool, then add your label for node affinity.

   ibmcloud ks worker-pool get --cluster <cluster_name_or_ID> --worker-pool <worker_pool_name_or_ID> | grep Labels
   

   Example output of a worker pool with the label.

   Labels:             ibm-cloud.kubernetes.io/worker-pool-id=a1aa111111b22b22cc3c3cc444444d44-4d555e5
   

7. Taint the worker pools that you want to autoscale so that the worker pool does not accept workloads except the ones that you want to run on the autoscaled worker pool. You can learn more about taints and tolerations in the community Kubernetes documentation. As an example, you might set a taint of use=autoscale:NoExecute. In this example, the NoExecute taint evicts pods that don't have the toleration corresponding to this taint.

8. Install the cluster autoscaler add-on.

After install the add-on, update the ConfigMap values to enable autoscaling in your cluster.