Understanding secure by default Cluster VPC Networking

Overview

With Secure by Default Networking, when you provision a new IBM Cloud Kubernetes Service VPC cluster at version 1.30 or later, only the traffic that is necessary for the cluster to function is allowed and all other access is blocked. To implement Secure by Default Networking, IBM Cloud Kubernetes Service uses various security groups and security group rules to protect cluster components. These security groups and rules are automatically created and attached to your worker nodes, load balancers, and cluster-related VPE gateways.

Virtual Private Cloud security groups filter traffic at the hypervisor level. Security group rules are not applied in a particular order. However, requests to your worker nodes are only permitted if the request matches one of the rules that you specify. When you allow traffic in one direction by creating an inbound or outbound rule, responses are also permitted in the opposite direction. Security groups are additive, meaning that if your worker nodes are attached to more than one security group, all rules included in the security groups are applied to the worker nodes. Newer cluster versions might have more rules in the kube-<clusterID> security group than older cluster versions. Security group rules are added to improve the security of the service and do not break functionality.

Virtual private endpoint (VPE) gateways

When the first VPC cluster at IBM Cloud Kubernetes Service 1.28+ is created in a given VPC, or a cluster in that VPC has its master updated to 1.28+, then several shared VPE Gateways are created for various IBM Cloud services. Only one of each of these types of shared VPE Gateways is created per VPC. All the clusters in the VPC share the same VPE Gateway for these services. These shared VPE Gateways are assigned a single Reserved IP from each zone that the cluster workers are in.

The following VPE gateways are created automatically when you create a VPC cluster.

† All supported VPC clusters have a VPE Gateway for the cluster master that gets created at in the your account when the cluster gets created. This VPE Gateway is used by the cluster workers, and can be used by other things in the VPC, to connect to the cluster's master API server. This VPE Gateway is assigned a single Reserved IP from each zone that the cluster workers are in, and this IP is created in one of the VPC subnets in that zone that has cluster workers. For example, if the cluster has workers in only a single zone region (us-east-1) and single VPC subnet, then a single IP is created in that subnet and assigned to the VPE Gateway. If a cluster has workers in all three zones like us-east-1, us-east-2, and us-east-3 and these workers are spread out among 4 VPC subnets in each zone, then 12 VPC subnets altogether, three IPs are created, one in each zone, in one of the four VPC subnets in that zone. Note that the subnet is chosen at random.

Managed security groups

IBM Cloud Kubernetes Service automatically creates and updates the following security groups and rules for VPC clusters.

• Worker security group (kube-<clusterID>).
• Master VPE gateway security group (kube-vpegw-<clusterID>).
• Shared VPE gateway security group (kube-vpegw-<vpcID>).
• Load balancer services security group (kube-lbaas-<clusterID>).

Limitations

Worker node security groups

Because the worker nodes in your VPC cluster exist in a service account and aren't listed in the VPC infrastructure dashboard, you can't create a security group and apply it to your worker node instances. You can only modify the existing kube-<clusterID> security group.

Logging and monitoring

When setting up logging and monitoring on a 1.30 or later cluster, you must use the private service endpoint when installing the logging agent in your cluster. Log data is not be saved if the public endpoint is used.

Monitoring clusters with RHCOS worker nodes

The monitoring agent relies on kernel headers in the operating system, however RHCOS doesn't have kernel headers. In this scenario, the agent reaches back to sysdig.com to use the pre-compiled agent. In clusters with no public network access this process fails. To allow monitoring on RHCOS clusters, you must either allow outbound traffic or see the Sysdig documentation for installing the agent on air-gapped environments.

VPC cluster quotas

There is a maximum of 15 rules that can target other security groups as their source or destination. By default, IBM Cloud Kubernetes Service applies 1 rule that targets the kube-<clusterID> security group for each cluster in the VPC. Because of this quota, only 15 clusters can be created in a given VPC. For more information, see VPC quotas.

Backup communication over the public network

VPC cluster workers use the private network to communicate with the cluster master. Previously, for VPC clusters that had the public service endpoint enabled, if the private network was blocked or unavailable, then the cluster workers could fall back to using the public network to communicate with the cluster master. In Secure by Default clusters, falling back to the public network is not an option because public outbound traffic from the cluster workers is blocked. You might want to disable outbound traffic protection to allow this public network backup option, however, there is a better alternative. Instead, if there a temporary issue with the worker-to-master connection over the private network, then, at that time, you can add a temporary security group rule to the kube-clusterID security group to allow outbound traffic to the cluster master apiserver port. Later, when the problem is resolved, you can remove the temporary rule.

OpenShift Data Foundation and Portworx encryption

If you plan to use OpenShift Data Foundation or Portworx in a cluster with no public network access, and you want to use Hyper Protect Crypto Services or Key Protect for encryption, you must create a virtual private endpoint gateway (VPE) that allows access to your KMS instance. Make sure to bind at least 1 IP address from each subnet in your VPC to the VPE.