Service limitations
IBM Cloud® Kubernetes Service and the Kubernetes open source project come with default service settings and limitations to ensure security, convenience, and basic functionality. Some limitations you might be able to change where noted.
If you anticipate reaching any of the following IBM Cloud Kubernetes Service limitations, contact IBM Support and provide the cluster ID, the new quota limit, the region, and infrastructure provider in your support ticket.
Service and quota limitations
IBM Cloud Kubernetes Service comes with the following service limitations and quotas that apply to all clusters, independent of what infrastructure provider you plan to use. Keep in mind that the classic and VPC cluster limitations also apply.
To view quota limits on cluster-related resources in your IBM Cloud account, use the ibmcloud ks quota ls command.
| Category | Description |
|---|---|
| API rate limits | 200 requests per 10 seconds to the IBM Cloud Kubernetes Service API from each unique source IP address. |
| App deployment | The apps that you deploy to and services that you integrate with your cluster must be able to run on the operating system of the worker nodes. |
| Calico network plug-in | Changing the Calico plug-in, components, or default Calico settings is not supported. For example, don't deploy a new Calico plug-in version, or modify the daemon sets or deployments for the Calico components, default IPPool resources, or Calico nodes. Instead, you can follow the documentation to create a Calico NetworkPolicy or GlobalNetworkPolicy, to change the Calico MTU,
or to disable the port map plug-in for the Calico CNI. |
| Cluster quota | You can't exceed 100 clusters per region and per infrastructure provider. However, as of 01 January 2024, quotas are increased incrementally before
reaching 100. If you need more of the resource, contact IBM Support. In the support case, include the new quota limit for the region and infrastructure provider that you want.. To
list quotas, run ibmcloud quota ls. |
| Kubernetes | Make sure to review the Kubernetes project limitations. |
| KMS provider | Customizing the IP addresses that are allowed to connect to your IBM® Key Protect for IBM Cloud® instance is not supported. |
| Kubernetes pod logs | To check the logs for individual app pods, you can use the command line to run kubectl logs <pod name>. Do not use the Kubernetes dashboard to stream logs for your pods, which might cause a disruption in your access
to the Kubernetes dashboard. |
| Load balancers | Although the Kubernetes SCTP protocol is generally available in the Kubernetes community release, creating load balancers that use this protocol is not supported in IBM Cloud Kubernetes Service clusters. |
| Operating system | Worker nodes must run one of the supported operating systems. You can't create a cluster with worker nodes that run different types of operating systems. For more information, see the Kubernetes version information. |
| Pod instances | You can run 110 pods per worker node. If you have worker nodes with 11 CPU cores or more, you can support 10 pods per core, up to a limit of 250 pods per worker node. The number of pods includes kube-system and ibm-system pods that run on the worker node. For improved performance, consider limiting the number of pods that you run per compute core so that you don't overuse the worker node. For example, on a worker node with a b3c.4x16 flavor,
you might run 10 pods per core that use no more than 75% of the worker node total capacity. |
| Worker node quota | A maximum 500 worker nodes for any accounts created before 01 January 2024. For accounts created on or after that date, the maximum quota is 200 after a period of lower quotas. Quotas apply per cluster infrastructure provider.
If you need more of the resource, contact IBM Support. In the support case, include the new quota limit for the region and infrastructure provider that you want.. To list quotas
run, ibmcloud ks quota ls. |
| Worker pool size | You must always have a minimum of 1 node in your cluster. Because of the worker node quota, you are limited in the number of worker pools per cluster and number of worker nodes per worker pool. For example, with the default worker node quota of 500 per region, you might have up to 500 worker pools of 1 worker node each in a region with only 1 cluster. Or, you might have 1 worker pool with up to 500 worker nodes in a region with only 1 cluster. |
| Red Hat Enterprise Linux CoreOS worker nodes | The maximum amount of zones added to a cluster is 15. For example, 4 RHCOS worker pools with 3 zones each will account for 12/15 of the quota for that cluster. |
| Cluster naming | To ensure that the Ingress subdomain and certificate are correctly registered, the first 24 characters of the clusters' names must be different. If you create and delete clusters with the same name or names that have the same first 24 characters 5 times or more within 7 days, such as for automation or testing purposes, you might reach the Let's Encrypt Duplicate Certificate rate limit. |
| Resource groups | A cluster can be created in only one resource group that you can't change afterward. If you create a cluster in the wrong resource group, you must delete the cluster and re-create it in the correct resource group. Furthermore, if you need
to use the ibmcloud ks cluster service bind command to integrate with an IBM Cloud service, that service must be in the same resource group as
the cluster. Services that don't use resource groups like IBM Cloud Container Registry or that don't need service binding like IBM Log Analysis work even if the cluster is in a different resource group. |
Classic cluster limitations
Classic infrastructure clusters in IBM Cloud Kubernetes Service are released with the following limitations.
Compute
Keep in mind that the service limitations also apply.
| Category | Description |
|---|---|
| Reserved instances | Reserved capacity and reserved instances are not supported. |
| Worker node flavors | Worker nodes are available in select flavors of compute resources. |
| Worker node host access | For security, you can't SSH into the worker node compute host. |
Networking
Keep in mind that the service limitations also apply.
| Category | Description |
|---|---|
| Ingress ALBs |
|
| Istio managed add-on | See Istio add-on limitations. |
| Network load balancers (NLB) |
|
| strongSwan VPN service | See strongSwan VPN service considerations. |
| Service IP addresses | You can have 65,000 IP addresses per cluster in the 172.21.0.0/16 range that you can assign to Kubernetes services within the cluster. |
| Subnets per VLAN | Each VLAN has a limit of 40 subnets. |
Storage
Keep in mind that the service limitations also apply.
| Category | Description |
|---|---|
| Volume instances | You can have a total of 250 IBM Cloud infrastructure file and block storage volumes per account. If you mount more than this amount, you might see an out of capacity message when you provision persistent volumes. For more
FAQs, see the file and block storage docs. If you want to mount more volumes,
contact IBM Support. In your support ticket, include your account ID and the new file or block storage volume quota that you want. |
| Portworx | Review the Portworx limitations. |
VPC cluster limitations
VPC clusters in IBM Cloud Kubernetes Service are released with the following limitations. Additionally, all the underlying VPC quotas, VPC limits, VPC service limitations, and regular service limitations apply.
Compute
Keep in mind that the service limitations also apply.
| Category | Description |
|---|---|
| Encryption | The secondary disks of your worker nodes are encrypted at rest by default by the underlying VPC infrastructure provider. However, you can't bring your own encryption to the underlying virtual server instances. |
| Location | VPC clusters are available only in select multizone regions. |
| Virtual Private Cloud | See Limitations and Quotas. |
| Worker node flavors | Only certain flavors are available for worker node virtual machines. Bare metal machines are not supported. |
| Worker node host access | For security, you can't SSH into the worker node compute host. |
| Worker node updates | You can't update or reload VPC worker nodes. Instead, you can delete the worker node and rebalance the worker pool with the ibmcloud ks worker replace command. If you replace multiple worker nodes at the same time, they
are deleted and replaced concurrently, not one by one. Make sure that you have enough capacity in your cluster to reschedule your workloads before you replace worker nodes. |
Networking
Keep in mind that the service limitations also apply.
| Category | Description |
|---|---|
| App URL length | DNS resolution is managed by the cluster's virtual private endpoint (VPE), which can resolve URLs up to 130 characters. If you expose apps in your cluster with URLs, such as the Ingress subdomain, ensure that the URLs are 130 characters or fewer. |
| Istio managed add-on | See Istio add-on limitations. |
| Network speeds | VPC profile network speeds refer to the speeds of the worker node interfaces. The maximum speed available to your worker nodes is 25Gbps. Because IP in IP encapsulation is required
for traffic between pods that are on different subnets, data transfer speeds between pods on different subnets might be slower, about half the compute profile network speed. Overall network speeds for apps that you deploy to your cluster
depend on the worker node size and application's architecture. |
| NodePort | You can access an app through a NodePort only if you are connected to your private VPC network, such as through a VPN connection. To access an app from the internet, you must use a VPC load balancer or Ingress service instead. |
| Pod network | VPC access control lists (ACLs) filter incoming and outgoing traffic for your cluster at the subnet level, and security groups filter incoming and outgoing traffic for your cluster at the worker nodes level. To control traffic within the cluster at the pod-to-pod level, you can't use VPC security groups or ACLs. Instead, use Calico and Kubernetes network policies, which can control the pod-level network traffic that uses IP in IP encapsulation. |
| strongSwan VPN service | The strongSwan service is not supported. To connect your cluster to resources in an on-premises network or another VPC, see Using VPN with your VPC. |
| Subnets |
|
| VPC load balancer | See VPC load balancer limitations. |
Storage
Keep in mind that the service limitations also apply.
| Category | Description |
|---|---|
| Block Storage for VPC cluster add-on | The Block Storage for VPC cluster add-on is enabled by default on VPC clusters. However, the add-on is not currently supported for clusters with UBUNTU_18_S390X worker nodes. When you create a VPC cluster with UBUNTU_18_S390X worker nodes, the add-on pods will remain in a Pending state. You can disable the add-on by running the ibmcloud ks cluster addon disable command. |
| Storage class for profile sizes | For more information, see available volume profiles. |
| Supported types |
You can set up Block Storage for VPC, IBM Cloud Object Storage and Cloud Databases only.
|
| Volume attachments | See Volume attachment limits. |
| Portworx | Review the Portworx limitations. |
| Block Storage for VPC | The default storage class in VPC clusters can not be changed. However, you can create your own storage class. |