Configuring classic subnets and IP addresses

VLANs

When you create a cluster, the cluster's worker nodes are connected automatically to a VLAN. A VLAN configures a group of worker nodes and pods as if they were attached to the same physical wire and provides a channel for connectivity among the workers and pods.

VLANs for standard clusters

In standard clusters, the first time that you create a cluster in a zone, a public VLAN and a private VLAN in that zone are automatically provisioned for you in your IBM Cloud infrastructure account. For every subsequent cluster that you create in that zone, you must specify the VLAN pair that you want to use in that zone. You can reuse the same public and private VLANs that were created for you because multiple clusters can share VLANs. You can either connect your worker nodes to both a public VLAN and the private VLAN, or to the private VLAN only. If you want to connect your worker nodes to a private VLAN only, you can use the ID of an existing private VLAN or create a private VLAN and use the ID during cluster creation.

To see the VLANs that are provisioned in each zone for your account, run ibmcloud ks vlan ls --zone <zone>. To see the VLANs that one cluster is provisioned on, run ibmcloud ks cluster get --cluster <cluster_name_or_ID> --show-resources and look for the Subnet VLANs section.

IBM Cloud infrastructure manages the VLANs that are automatically provisioned when you create your first cluster in a zone. If you let a VLAN become unused, such as by removing all worker nodes from a VLAN, IBM Cloud infrastructure reclaims the VLAN. After, if you need a new VLAN, contact IBM Cloud support.

Can I change my VLAN decision later?

You can change your VLAN setup by modifying the worker pools in your cluster. For more information, see Changing your worker node VLAN connections.

Subnets and IP addresses

In addition to worker nodes and pods, subnets are also automatically provisioned onto VLANs. Subnets provide network connectivity to your cluster components by assigning IP addresses to them.

The following subnets are automatically provisioned on the default public and private VLANs.

Finding subnets provisioned in your account

To see all the subnets provisioned in all resource groups of your account, run ibmcloud ks subnets --provider classic. To see the portable public and portable private subnets that are bound to one cluster, you can run ibmcloud ks cluster get --cluster <cluster_name_or_ID> --show-resources and look for the Subnet VLANs section.

In IBM Cloud Kubernetes Service, VLANs have a limit of 40 subnets. If you reach this limit, first check to see whether you can reuse subnets in the VLAN to create new clusters. If you need a new VLAN, order one by contacting IBM Cloud support. Then, create a cluster that uses this new VLAN.

Do the IP addresses for my worker nodes change?

Your worker node is assigned an IP address on the public or private VLANs that your cluster uses. After the worker node is provisioned, the worker node IP address persists across reboot and update operations, but the worker node IP address changes after a replace operation. Additionally, the private IP address of the worker node is used for the worker node identity in most kubectl commands. If you change the VLANs that the worker pool uses, new worker nodes that are provisioned in that pool use the new VLANs for their IP addresses. Existing worker node IP addresses don't change, but you can choose to remove the worker nodes that use the old VLANs.

Can I specify subnets for pods and services in my cluster?

If you plan to connect your cluster to on-premises networks through IBM Cloud Direct Link or a VPN service, you can avoid subnet conflicts by specifying a custom subnet CIDR that provides the private IP addresses for your pods, and a custom subnet CIDR to provide the private IP addresses for services.

To specify custom pod and service subnets during cluster creation, use the --pod-subnet and --service-subnet options in the ibmcloud ks cluster create CLI command.

Network segmentation

Network segmentation describes the approach to divide a network into multiple sub-networks. Apps that run in one sub-network can't see or access apps in another sub-network. For more information about network segmentation options and how they relate to VLANs, see this cluster security topic.

However, in several situations, components in your cluster must be permitted to communicate across multiple private VLANs. For example, if you want to create a multizone cluster, if you have multiple VLANs for a cluster, or if you have multiple subnets on the same VLAN, the worker nodes on different subnets in the same VLAN or in different VLANs can't automatically communicate with each other. You must enable either a Virtual Router Function (VRF) or VLAN spanning for your IBM Cloud infrastructure account.

Virtual Routing and Forwarding (VRF)

VRF enables all the VLANs and subnets in your infrastructure account to communicate with each other. Additionally, a VRF is required to allow your workers and master to communicate over the private cloud service endpoint. To enable VRF, see Enabling VRF. To check whether a VRF is already enabled, use the ibmcloud account show command. Note that VRF eliminates the VLAN spanning option for your account, because all VLANs are able to communicate unless you configure a gateway appliance to manage traffic.

VLAN spanning

If you can't or don't want to enable VRF, enable VLAN spanning. To perform this action, you need the Network > Manage Network VLAN Spanning infrastructure permission, or you can request the account owner to enable it. To check if VLAN spanning is already enabled, use the ibmcloud ks vlan spanning get --region <region> command. Note that you can't enable the private cloud service endpoint if you choose to enable VLAN spanning instead of a VRF.

How does VRF or VLAN spanning affect network segmentation?

When VRF or VLAN spanning is enabled, any system that is connected to any of the private VLANs in the same IBM Cloud account can communicate with workers. You can isolate your cluster from other systems on the private network by applying Calico private network policies. IBM Cloud Kubernetes Service is also compatible with all IBM Cloud infrastructure firewall offerings. You can set up a firewall, such as a Virtual Router Appliance, with custom network policies to provide dedicated network security for your standard cluster and to detect and remediate network intrusion.

Viewing available portable public IP addresses

To list all the portable IP addresses in your cluster, both used and available, you can run the following command.

kubectl get cm ibm-cloud-provider-vlan-ip-config -n kube-system -o yaml

To list only portable public IP addresses that are available to create public NLBs or more public ALBs, you can use the following steps:

Before you begin

• Ensure that you have the Writer or Manager IBM Cloud IAM service access role for the default namespace.
• Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.

To list available portable public IP addresses,

1. Create a Kubernetes service configuration file that is named myservice.yaml and define a service of type LoadBalancer with a dummy NLB IP address. The following example uses the IP address 1.1.1.1 as the NLB IP address. Replace <zone> with the zone where you want to check for available IPs.

   apiVersion: v1
   kind: Service
   metadata:
     labels:
       run: myservice
     name: myservice
     namespace: default
     annotations:
       service.kubernetes.io/ibm-load-balancer-cloud-provider-zone: "<zone>"
   spec:
     ports:
     - port: 80
       protocol: TCP
       targetPort: 80
     selector:
       run: myservice
     sessionAffinity: None
     type: LoadBalancer
     loadBalancerIP: 1.1.1.1
   

2. Create the service in your cluster.

   kubectl apply -f myservice.yaml
   

3. Inspect the service.

   kubectl describe service myservice
   

   The creation of this service fails because the Kubernetes master can't find the specified NLB IP address in the Kubernetes ConfigMap. When you run this command, you can see the error message and a list of available public IP addresses for the cluster.

   Error on cloud load balancer a8bfa26552e8511e7bee4324285f6a4a for service default/myservice with UID 8bfa2655-2e85-11e7-bee4-324285f6a4af: Requested cloud provider IP 1.1.1.1 is not available. The following cloud provider IP addresses are available: <list_of_IP_addresses>
   

Freeing up used IP addresses

You can free up a used portable IP address by deleting the network load balancer (NLB) service or disabling the Ingress application load balancer (ALB) that is using the portable IP address.

Before you begin

• Ensure that you have the Writer or Manager IBM Cloud IAM service access role for the default namespace.
• Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.

To delete an NLB or disable an ALB,

1. List available services in your cluster.

   kubectl get services | grep LoadBalancer
   

2. Remove the load balancer service or disable the ALB that uses a public or private IP address.
   • Delete an NLB:

     kubectl delete service <service_name>
     

   • Disable an ALB:

     ibmcloud ks ingress alb disable --alb <ALB_ID> -c <cluster_name_or_ID>
     

Adding portable IPs by ordering more subnets

You can get more portable IPs for NLB services by creating a new subnet in an IBM Cloud infrastructure account and making it available to your specified cluster.

Portable public IP addresses are charged monthly. If you remove portable public IP addresses after your subnet is provisioned, you still must pay the monthly charge, even if you used them only for a short amount of time.

Before you begin

• Ensure that you have the Operator or Administrator IAM platform access role for the cluster.
• Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.

To order a subnet,

1. Provision a new subnet.

   ibmcloud ks cluster subnet create --cluster <cluster_name_or_id> --size <subnet_size> --vlan <VLAN_ID>
   

   Parameters for a subnet

   Parameter	Description
   <cluster_name_or_id>	Replace <cluster_name_or_id> with the name or ID of the cluster.
   <subnet_size>	Replace <subnet_size> with the number of IP addresses that you want to create in the portable subnet. Accepted values are 8, 16, 32, or 64. Note that when you add portable IP addresses for your subnet, three IP addresses are used to establish cluster-internal networking. You can't use these three IP addresses for your Ingress application load balancers (ALBs) or to create network load balancer (NLB) services. For example, if you request eight portable public IP addresses, you can use five of them to expose your apps to the public.
   <VLAN_ID>	Replace <VLAN_ID> with the ID of the public or private VLAN on which you want to allocate the portable public or private IP addresses. You must select a public or private VLAN that an existing worker node is connected to. To review the public or private VLANs that your worker nodes are connected to, run ibmcloud ks cluster get --cluster <cluster> --show-resources and look for the Subnet VLANs section in the output. The subnet is provisioned in the same zone that the VLAN is in.

2. Verify that the subnet was successfully created and added to your cluster. The subnet CIDR is listed in the Subnet VLANs section.

   ibmcloud ks cluster get --cluster <cluster_name_or_ID> --show-resources
   

   In this example output, a second subnet was added to the 2234945 public VLAN.

   Subnet VLANs
   VLAN ID   Subnet CIDR          Public   User-managed
   2234947   10.xxx.xx.xxx/29     false    false
   2234945   169.xx.xxx.xxx/29    true     false
   2234945   169.xx.xxx.xxx/29    true     false
   

3. Important: To enable communication between workers that are on different subnets on the same VLAN, you must enable routing between subnets on the same VLAN.

Adding portable IPs by adding existing subnets to your cluster

You can get more portable IPs for NLB services by making an existing subnet in an IBM Cloud infrastructure account available to your cluster.

Before you begin

• Ensure that you have the Operator or Administrator IAM platform access role for the cluster.
• Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.

To make a subnet available to your cluster,

1. Review the IDs of the public or private VLANs on which you want to allocate the portable public or private IP addresses. You must select a public or private VLAN that an existing worker node is connected to.

   ibmcloud ks cluster get --cluster <cluster_name_or_id> --show-resources
   

   In the output, look for VLAN IDs in the Subnet VLANs section.

   Subnet VLANs
   VLAN ID   Subnet CIDR          Public   User-managed
   2234947   10.xxx.xx.xxx/29     false    false
   2234945   169.xx.xxx.xxx/29    true     false
   

2. Get the ID of the subnet to use. Ensure that the subnet is on one of the VLAN IDs that you found in the previous step, and that the subnet is not already bound to another cluster.

   ibmcloud ks subnets --provider classic
   

   In this example output, the subnet ID is 1602829, which is on the VLAN ID 2234945.

   GETting subnet list...
   OK
   ID        Network             Gateway          VLAN ID   Type      Bound Cluster
   1550165   10.xxx.xx.xxx/26    10.xxx.xx.xxx    2234947   private
   1602829   169.xx.xxx.xxx/28   169.xx.xxx.xxx   2234945   public
   

3. Make the subnet available to your cluster.

   ibmcloud ks cluster subnet add --cluster <cluster_name_or_id> --subnet-id <subnet_ID>
   

4. Verify that the subnet was successfully created and added to your cluster. The subnet CIDR is listed in the Subnet VLANs section.

   ibmcloud ks cluster get --cluster <cluster_name> --show-resources
   

   In this example output, a second subnet was added to the 2234945 public VLAN.

   Subnet VLANs
   VLAN ID   Subnet CIDR          Public   User-managed
   2234947   10.xxx.xx.xxx/29     false    false
   2234945   169.xx.xxx.xxx/29    true     false
   2234945   169.xx.xxx.xxx/29    true     false
   

5. Important: To enable communication between workers that are on different subnets on the same VLAN, you must enable routing between subnets on the same VLAN.

Enabling routing between primary subnets on the same VLAN

When you create a cluster, primary public and private subnets are provisioned on the public and private VLANs. The primary public subnet ends in /28 and provides 14 public IPs for worker nodes. The primary private subnet ends in /26 and provides private IPs for up to 62 worker nodes.

You might exceed the initial 14 public and 62 private IPs for worker nodes by having a large cluster or several smaller clusters in the same location on the same VLAN. When a public or private subnet reaches the limit of worker nodes, another primary subnet in the same VLAN is ordered.

To ensure that workers in these primary subnets on the same VLAN can communicate, you must turn on VLAN spanning. For instructions, see Enable or disable VLAN spanning.

To check if VLAN spanning is already enabled, use the ibmcloud ks vlan spanning get --region <region> command.

Managing subnet routing for gateway appliances

When you create a cluster, a portable public and a portable private subnet are ordered on the VLANs that the cluster is connected to. These subnets provide IP addresses for Ingress application load balancer (ALB) and network load balancer (NLB) services.

However, if you have an existing router appliance, such as a Virtual Router Appliance (VRA), the newly added portable subnets from those VLANs that the cluster is connected to are not configured on the router. To use NLBs or Ingress ALBs, you must ensure that network devices can route between different subnets on the same VLAN by enabling a Virtual Router Function (VRF) for your IBM Cloud infrastructure account. To enable VRF, see Enabling VRF. If you can't or don't want to enable VRF, enable VLAN spanning.