Why are the ibm-cloud-provider-ip
pods for the Istio ingress gateway stuck in pending
?
Virtual Private Cloud Classic infrastructure
Multizone clusters only
When you run kubectl get pod -n ibm-system
, the ibm-cloud-provider-ip
pod that provides the external IP address for your Istio ingress gateway is stuck in the pending
status.
Additionally, when you run kubectl describe pod <pod_name> -n ibm-system
for the ibm-cloud-provider-ip
pod, you notice a scheduling conflict error in the pod's events section of the output.
To identify the ibm-cloud-provider-ip
pod for your gateway, you can run kubectl get service -n istio-system
to find your gateway's load balancer service, note its EXTERNAL-IP, and look for the IP address
in the ibm-cloud-provider-ip
pod name.
When an istio-ingressgateway
load balancer service is created, an ibm-cloud-provider-ip
pod is created to provide an external IP address to the load balancer.
These ibm-cloud-provider-ip
pods have a node affinity rule so that they are created in the same zone as the subnet from which the IP address is derived. However, when the ExternalTrafficPolicy
setting for the istio-ingressgateway
load balancer service is set to Local
, then the ibm-cloud-provider-ip
pod also has a pod affinity rule to be deployed to the same zone as the gateway load balancer pod. If the IP address and gateway load balancer exist
in different zones in your cluster, the ibm-cloud-provider-ip
pod can't properly deploy.
To verify that the ibm-cloud-provider-ip
and istio-ingressgateway
pods don't exist in the same zone:
-
Identify the NODE that your
istio-ingressgateway
pod is deployed to.kubectl get pod -n istio-system -o wide
-
Get the EXTERNAL-IP address for the gateway's load balancer service.
kubectl get service -n istio-system
-
Identify the NODE that the
ibm-cloud-provider-ip
pod for the load balancer is deployed to. Replace<IP-with-hyphens>
with the IP address that you found in the previous step. In the IP address, use hyphens (-) instead of periods (.). For example,169.12.345.67
becomes169-12-345-67
.kubectl get pod -n ibm-system -o wide -l ibm-cloud-provider-ip=<IP-with-hyphens>
-
List the zones that the worker nodes are in. Compare the worker nodes that you found in step 1 and 3 to determine whether the pods are deployed to worker nodes in different zones.
kubectl get node --no-headers -L ibm-cloud.kubernetes.io/zone
Example output
10.176.48.106 Ready <none> 529d v1.31+IKS dal10 10.176.48.107 Ready <none> 196d v1.31+IKS dal10 10.184.58.23 Ready <none> 2y38d v1.31+IKS dal12 10.184.58.42 Ready <none> 529d v1.31+IKS dal12
To resolve this issue, you can either move the istio-ingressgateway
pod to the zone that the ibm-cloud-provider-ip
pod exists in, or vice versa.
- Moving the
istio-ingressgateway
pod: The gateway's load balancer retains the same external IP address after it moves. However, in version 1.9 and earlier of the add-on, the changes that you make might be overwritten when the zone labels for gateways are automatically populated during Istio patch updates. - Moving the
ibm-cloud-provider-ip
pod: The gateway does not retain the same external IP address after it moves, and is assigned a new IP address. However, no changes are overwritten during add-on updates.
Moving the istio-ingressgateway
pod
Move the istio-ingressgateway
pod to the same zone as the ibm-cloud-provider-ip
pod.
- Edit the
managed-istio-custom
ConfigMap resource.kubectl edit cm managed-istio-custom -n ibm-operators
- For the gateway that you want to move, change the
istio-ingressgateway-zone-1|2|3
setting to the zone where theibm-cloud-provider-ip
pod exists. - Save and close the configuration file. The
istio-ingressgateway
pod is moved to a worker node in the same zone as theibm-cloud-provider-ip
pod, and theibm-cloud-provider-ip
pod properly deploys. - To verify that the pods now exist in the same zone, follow the steps in the Why it’s happening section.
Moving the ibm-cloud-provider-ip
pod
Move the ibm-cloud-provider-ip
pod to the same zone as the istio-ingressgateway
pod.
-
Edit the
managed-istio-custom
ConfigMap resource.kubectl edit cm managed-istio-custom -n ibm-operators
-
Note the name of the
istio-ingressgateway-zone-<1|2|3>
setting for the gateway. In this example ConfigMap, to move theibm-cloud-provider-ip
pod for the gateway that exists indal10
, note the key calledistio-ingressgateway-zone-1
.apiVersion: v1 data: istio-ingressgateway-zone-1: dal10 istio-ingressgateway-zone-2: dal12 istio-ingressgateway-public-1-enabled: "true" istio-ingressgateway-public-2-enabled: "true" istio-monitoring: "true" kind: ConfigMap ...
-
Temporarily disable the gateway by changing the corresponding
istio-ingressgateway-public-<1|2|3>-enabled
setting to"false"
. In this example ConfigMap, to move theibm-cloud-provider-ip
pod for the gateway that exists indal10
, changeistio-ingressgateway-public-1-enabled
to"false"
.apiVersion: v1 data: istio-ingressgateway-zone-1: dal10 istio-ingressgateway-zone-2: dal12 istio-ingressgateway-public-1-enabled: "false" istio-ingressgateway-public-2-enabled: "true" istio-monitoring: "true" kind: ConfigMap ...
-
Save and close the configuration file.
-
Verify that the gateway's load balancer pod is deleted. Note that it might take up to 30 minutes for the changes to complete.
kubectl get service -n istio-system -o wide
-
Open the
managed-istio-custom
ConfigMap resource.kubectl edit cm managed-istio-custom -n ibm-operators
-
Re-enable the gateway by changing the
istio-ingressgateway-public-<1|2|3>-enabled
setting back to"true"
. -
Save and close the configuration file. A new load balancer service for the gateway (the
istio-ingressgateway
pod) is created, and a newibm-cloud-provider-ip
pod for that load balancer is deployed to a worker node in the same zone as theistio-ingressgateway
pod. -
To verify that the pods now exist in the same zone, follow the steps in the Why it’s happening section.