Configuring the cluster DNS provider
Each service in your IBM Cloud® Kubernetes Service cluster is assigned a Domain Name System (DNS) name that the cluster DNS provider registers to resolve DNS requests. For more information about DNS for services and pods, see the Kubernetes documentation.
The cluster DNS provider is CoreDNS, which is a general-purpose, authoritative DNS server that provides a backwards-compatible, but extensible, integration with Kubernetes. Because
CoreDNS is a single executable and single process, it has fewer dependencies and moving parts that could experience issues than other cluster DNS providers. The project is also written in the same language as the Kubernetes project, Go
,
which helps protect memory. Finally, CoreDNS supports more flexible use cases than other cluster DNS providers because you can create custom DNS entries such as the common setups in the CoreDNS docs.
Autoscaling the cluster DNS provider
By default, CoreDNS includes a deployment to autoscale the CoreDNS pods in response to the number of worker nodes and cores within the cluster. You can fine-tune the CoreDNS autoscaler parameters by editing the CoreDNS autoscaling ConfigMap. For example, if your apps heavily use the cluster DNS provider, you might need to increase the minimum number of CoreDNS pods to support the app. For more information, see the Kubernetes documentation.
Before you begin: Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.
-
Verify that the CoreDNS autoscaler deployment is available. In your CLI output, verify that one deployment is AVAILABLE.
kubectl get deployment -n kube-system coredns-autoscaler
Example output
NAME READY UP-TO-DATE AVAILABLE AGE coredns-autoscaler 1/1 1 1 69d
-
Edit the default settings for the CoreDNS autoscaler. Look for the
data.linear
field, which defaults to one CoreDNS pod per 16 worker nodes or 256 cores, with a minimum of two CoreDNS pods regardless of cluster size (preventSinglePointFailure: true
). For more information, see the Kubernetes documentation.kubectl edit configmap -n kube-system coredns-autoscaler
Example output
apiVersion: v1 data: linear: '{"coresPerReplica":256,"nodesPerReplica":16,"preventSinglePointFailure":true}' kind: ConfigMap metadata: ...
Customizing the cluster DNS provider
You can customize CoreDNS by editing the CoreDNS ConfigMap. For example, you might want to configure stub domains and upstream DNS servers to resolve services that point to external hosts. Additionally, you can configure multiple Corefiles within the CoreDNS ConfigMap. For more information, see the Kubernetes documentation.
NodeLocal
DNS caching relies on CoreDNS to maintain the cache of DNS resolutions. Keep applicable NodeLocal
DNS cache and CoreDNS configurations such as stub domains the same to maintain DNS resolution consistency.
Before you begin: Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.
-
Verify that the CoreDNS deployment is available. In your CLI output, verify that one deployment is AVAILABLE.
kubectl get deployment -n kube-system coredns
Example output
NAME READY UP-TO-DATE AVAILABLE AGE coredns 3/3 3 3 69d
-
Edit the default settings for the CoreDNS ConfigMap. Use a Corefile in the
data
section of the ConfigMap to customize stub domains and upstream DNS servers. For more information, see the Kubernetes documentation.The CoreDNS
proxy
plug-in is deprecated and replaced with theforward
plug-in. If you update the CoreDNS ConfigMap, make sure to replace allproxy
instances withforward
.kubectl edit configmap -n kube-system coredns
CoreDNS example output
apiVersion: v1 kind: ConfigMap metadata: name: coredns namespace: kube-system data: Corefile: | import <MyCorefile> .:53 { errors health { lameduck 10s } ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa ttl 30 } prometheus :9153 forward . /etc/resolv.conf cache 30 loop reload loadbalance } <MyCorefile>: | abc.com:53 { errors cache 30 loop forward . 1.2.3.4 }
-
Optional: Add custom Corefiles to the CoreDNS ConfigMap. In the following example, include the
import <MyCoreFile>
in thedata.Corefile
section, and fill out thedata.<MyCorefile>
section with your custom Corefile information. For more information, see the Corefile import documentation.The CoreDNS
proxy
plug-in is deprecated and replaced with theforward
plug-in. If you update the CoreDNS ConfigMap, make sure to replace allproxy
instances withforward
.kubectl edit configmap -n kube-system coredns
Custom Corefile example output
apiVersion: v1 kind: ConfigMap metadata: name: coredns namespace: kube-system data: Corefile: | import <MyCorefile> .:53 { errors health kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure upstream 172.16.0.1 fallthrough in-addr.arpa ip6.arpa } prometheus :9153 forward . /etc/resolv.conf cache 30 loop reload loadbalance } <MyCorefile>: | abc.com:53 { errors cache 30 loop forward . 1.2.3.4 }
-
After a few minutes, the CoreDNS pods pick up the ConfigMap changes.
Setting up NodeLocal DNS cache
Set up the NodeLocal
DNS caching agent on select worker nodes for improved cluster DNS performance and availability in your IBM Cloud Kubernetes Service cluster. For more information, see the Kubernetes docs.
By default, cluster DNS requests for pods that use a ClusterFirst
DNS policy are sent to the
cluster DNS service. If you enable NodeLocal
DNS caching on a worker node, the cluster DNS requests for these pods that are on the worker node are sent instead to the local DNS cache, which listens on link-local IP address 169.254.20.10.
The DNS cache also listens on the cluster IP of the kube-dns
service in the kube-system
namespace.
Do not add the DNS cache label when you already use zone-aware DNS in your cluster. Also, NodeLocal
DNS caching relies on CoreDNS to maintain the cache of DNS resolutions. Keep applicable NodeLocal
DNS cache and CoreDNS configurations such as stub domains the same to maintain DNS resolution consistency.
NodeLocal
DNS cache is generally available in clusters that run Kubernetes 1.18 or later, but still disabled by default.
Enable NodeLocal DNS cache
Enable NodeLocal
DNS cache for one or more worker nodes in your Kubernetes cluster.
The following steps update DNS pods that run on particular worker nodes. You can also label the worker pool so that future nodes inherit the label.
Before you begin, update any DNS egress network policies that are impacted by this feature, such as policies that rely on pod or namespace selectors for DNS egress.
kubectl get networkpolicy --all-namespaces -o yaml
-
If you customized stub domains and upstream DNS servers for CoreDNS, you must also customize the
NodeLocal
DNS cache with these stub domains and upstream DNS servers. -
List the nodes in your cluster. The
NodeLocal
DNS caching agent pods are part of a daemon set that run on each node.kubectl get nodes
-
Add the
ibm-cloud.kubernetes.io/node-local-dns-enabled=true
label to the worker node. The label starts the DNS caching agent pod on the worker node.-
Add the label to one or more worker nodes.
-
To label all worker nodes in the cluster: Add the label to all existing worker pools.
-
To label an individual worker node:
kubectl label node <node_name> --overwrite "ibm-cloud.kubernetes.io/node-local-dns-enabled=true"
-
-
Verify that the node has the label by checking that the
NODE-LOCAL-DNS-ENABLED
field is set totrue
.kubectl get nodes -L "ibm-cloud.kubernetes.io/node-local-dns-enabled"
Example output
NAME STATUS ROLES AGE VERSION NODE-LOCAL-DNS-ENABLED 10.xxx.xx.xxx Ready,SchedulingDisabled <none> 28h v1.31+IKS true
-
Verify that the DNS caching agent pod is running on the worker node.
kubectl get pods -n kube-system -l k8s-app=node-local-dns -o wide
Example output
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES node-local-dns-pvnjn 1/1 Running 0 1m 10.xxx.xx.xxx 10.xxx.xx.xxx <none> <none>
-
-
Repeat the previous steps for each worker node to enable DNS caching.
Disable NodeLocal DNS cache
You can disable the NodeLocal
DNS cache for one or more worker nodes.
-
Remove the
ibm-cloud.kubernetes.io/node-local-dns-enabled
label from the worker node. This action terminates the DNS caching agent pod on the worker node.Run the following command to remove the label from all worker nodes in the cluster.
kubectl label node --all --overwrite "ibm-cloud.kubernetes.io/node-local-dns-enabled-"
Run the following command to remove the label from an individual worker node.
kubectl label node <node_name> "ibm-cloud.kubernetes.io/node-local-dns-enabled-"
-
Verify that the label is removed by checking that the
NODE-LOCAL-DNS-ENABLED
field is empty.kubectl get nodes -L "ibm-cloud.kubernetes.io/node-local-dns-enabled"
Example output
NAME STATUS ROLES AGE VERSION NODE-LOCAL-DNS-ENABLED 10.xxx.xx.xxx Ready,SchedulingDisabled <none> 28h v1.31+IKS
-
Verify that the pod is no longer running on the node where DNS cache is disabled. The output shows no pods.
kubectl get pods -n kube-system -l k8s-app=node-local-dns -o wide
Example output
No resources found.
-
-
Repeat the previous steps for each worker node to disable DNS caching.
Customizing NodeLocal DNS cache
You can customize the NodeLocal
DNS cache by editing either of the two configmaps.
node-local-dns
ConfigMap: Customize theNodeLocal
DNS cache configuration.node-local-dns-config
ConfigMap: Extend theNodeLocal
DNS cache configuration by customizing stub domains or upstream DNS servers to resolve services that point to external hosts.
NodeLocal
DNS caching relies on CoreDNS to maintain the cache of DNS resolutions. Keep applicable NodeLocal
DNS cache and CoreDNS configurations such as stub domains the same to maintain DNS resolution consistency.
Editing the node-local-dns
ConfigMap for general configuration updates
Edit the node-local-dns
ConfigMap to customize the NodeLocal
DNS cache configuration.
Before you begin: Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.
-
Verify that the
NodeLocal
DNS cache daemon set is available.kubectl get ds -n kube-system node-local-dns
Example output
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE node-local-dns 4 4 4 4 4 ibm-cloud.kubernetes.io/node-local-dns-enabled=true 82d
-
Edit the default settings or add custom Corefiles to the
NodeLocal
DNS cache ConfigMap. Each Corefile that you import must use thecoredns
path. For more information, see the Kubernetes documentation.Only a limited set of plug-ins is supported for the
NodeLocal
DNS cache.kubectl edit configmap -n kube-system node-local-dns
Example output
apiVersion: v1 kind: ConfigMap metadata: name: node-local-dns namespace: kube-system data: Corefile: | # Add your NodeLocal DNS customizations as import files under ./coredns directory. # Refer to /docs/containers?topic=containers-cluster_dns for details. import ./coredns/<MyCorefile> cluster.local:53 abc.com:53 { errors cache { success 9984 30 denial 9984 5 } reload loop bind 169.254.20.10 172.21.0.10 forward . __PILLAR__CLUSTER__DNS__ { force_tcp } prometheus :9253 health 169.254.20.10:8080 } in-addr.arpa:53 { errors cache 30 reload loop bind 169.254.20.10 172.21.0.10 forward . __PILLAR__CLUSTER__DNS__ { force_tcp } prometheus :9253 } ip6.arpa:53 { errors cache 30 reload loop bind 169.254.20.10 172.21.0.10 forward . __PILLAR__CLUSTER__DNS__ { force_tcp } prometheus :9253 } .:53 { errors cache 30 reload loop bind 169.254.20.10 172.21.0.10 forward . __PILLAR__UPSTREAM__SERVERS__ { force_tcp } prometheus :9253 } <MyCorefile>: | # Add custom corefile content ...
-
After a few minutes, the
NodeLocal
DNS cache pods pick up the ConfigMap changes.
Editing the node-local-dns-config
ConfigMap to extend with stub domains or upstream servers
Edit the node-local-dns-config
ConfigMap to extend the NodeLocal
DNS cache configuration such as by customizing stub domains or upstream DNS servers. For more information, see the Kubernetes documentation.
Before you begin: Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.
-
Verify that the
NodeLocal
DNS cache daemon set is available.kubectl get ds -n kube-system node-local-dns
Example output
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE node-local-dns 4 4 4 4 4 ibm-cloud.kubernetes.io/node-local-dns-enabled=true 82d
-
Confirm that the
NodeLocal
DNS cache has a ConfigMap.-
Determine if the
NodeLocal
DNS cache ConfigMap exists.kubectl get cm -n kube-system node-local-dns-config
Example output if no ConfigMap exists
Error from server (NotFound): ConfigMaps "node-local-dns-config" not found
-
If the ConfigMap does not exist, create a
NodeLocal
DNS cache ConfigMap.kubectl create cm -n kube-system node-local-dns-config
Example output
configmap/node-local-dns-config created
-
-
Edit the
NodeLocal
DNS cache ConfigMap. The ConfigMap uses the KubeDNS syntax to customize stub domains and upstream DNS servers. For more information, see the Kubernetes documentation.kubectl edit cm -n kube-system node-local-dns-config
Example output
apiVersion: v1 kind: ConfigMap metadata: name: node-local-dns-config namespace: kube-system data: stubDomains: | {"abc.com" : ["1.2.3.4"]}
-
After a few minutes, the
NodeLocal
DNS cache pods pick up the ConfigMap changes.
Setting up zone-aware DNS
Set up zone-aware DNS for improved cluster DNS performance and availability in your multizone IBM Cloud Kubernetes Service cluster. This setup extends NodeLocal
DNS cache to prefer cluster DNS traffic within the same zone.
By default, your cluster is set up with cluster-wide DNS resources, not zone-aware DNS resources. Even after you set up zone-aware DNS, the cluster-wide DNS resources remain running as a backup DNS. Your zone-aware DNS resources are separate from the cluster-wide DNS, and changing zone-aware DNS does not impact the cluster-wide DNS.
Do not use the DNS cache label when you use zone-aware DNS in your cluster.
Setting up zone-aware DNS
Complete the following step to set up zone-aware DNS in your multizone cluster.
-
Update any DNS egress network policies that are impacted by zone-aware DNS, such as policies that rely on pod or namespace selectors for DNS egress. Run the following command to get a list of your network policies.
kubectl get networkpolicy --all-namespaces -o yaml
-
Label the
coredns
ConfigMap in thekube-system
namespace withibm-cloud.kubernetes.io/deploy-zone-aware-dns=true
.kubectl label cm -n kube-system coredns --overwrite "ibm-cloud.kubernetes.io/deploy-zone-aware-dns=true"
-
Refresh the cluster master to deploy the zone-aware DNS resources.
ibmcloud ks cluster master refresh -c <cluster_name_or_ID>
-
Watch for the refresh operation to complete by reviewing the Master Health in the cluster details.
ibmcloud ks cluster get -c <cluster_name_or_ID>
-
If you customized stub domains and upstream DNS servers for CoreDNS, you must also customize the
NodeLocal
DNS cache with these stub domains and upstream DNS servers. -
Set an environment variable for the zones of the cluster.
ZONES=$(kubectl get nodes --no-headers --ignore-not-found=true -o jsonpath='{range .items[*]}{.metadata.labels.topology\.kubernetes\.io/zone}{"\n"}{end}' | uniq)
-
Start the CoreDNS and CoreDNS autoscaler pods in all zones.
for ZONE in ${ZONES}; do kubectl scale deployment -n kube-system "coredns-autoscaler-${ZONE}" --replicas=1 done
-
Verify that the CoreDNS and CoreDNS autoscaler pods are running in all zones.
for ZONE in ${ZONES}; do kubectl get pods -n kube-system -l "k8s-app=coredns-autoscaler-${ZONE}" -o wide kubectl get pods -n kube-system -l "k8s-app=coredns-${ZONE}" -o wide done
-
Start the
NodeLocal
DNS cache pods on all workers nodes.kubectl label nodes --all --overwrite "ibm-cloud.kubernetes.io/zone-aware-dns-enabled=true"
-
Verify that the
NodeLocal
DNS cache pods are running on all workers nodes.for ZONE in ${ZONES}; do kubectl get pods -n kube-system -l "k8s-app=node-local-dns-${ZONE}" -o wide done
-
Label your worker pools so that future worker nodes inherit the
ibm-cloud.kubernetes.io/zone-aware-dns-enabled=true
label.
Disabling and deleting zone-aware DNS
To remove zone-aware DNS, you must first disable zone-aware DNS in each zone of your multizone cluster. Then, delete the zone-aware DNS resources.
-
Remove the
ibm-cloud.kubernetes.io/zone-aware-dns-enabled=true
label from your worker pools. -
Set an environment variable for the zones in the cluster.
ZONES=$(kubectl get nodes --no-headers --ignore-not-found=true -o jsonpath='{range .items[*]}{.metadata.labels.topology\.kubernetes\.io/zone}{"\n"}{end}' | uniq)
-
Stop the
NodeLocal
DNS cache pods on all worker nodes.kubectl label nodes --all --overwrite "ibm-cloud.kubernetes.io/zone-aware-dns-enabled-"
-
Stop the CoreDNS autoscaler pods in all zones.
for ZONE in ${ZONES}; do kubectl scale deployment -n kube-system "coredns-autoscaler-${ZONE}" --replicas=0 done
-
Verify that the CoreDNS autoscaler pods are no longer running in all zones.
for ZONE in ${ZONES}; do kubectl get pods -n kube-system -l "k8s-app=coredns-autoscaler-${ZONE}" done
-
Stop the CoreDNS pods in all zones.
for ZONE in ${ZONES}; do kubectl scale deployment -n kube-system "coredns-${ZONE}" --replicas=0 done
-
Remove the
ibm-cloud.kubernetes.io/deploy-zone-aware-dns=true
label from thecoredns
ConfigMap in thekube-system
namespace.kubectl label cm -n kube-system coredns --overwrite "ibm-cloud.kubernetes.io/deploy-zone-aware-dns-"
-
Refresh the cluster master to the delete zone-aware DNS resources.
ibmcloud ks cluster master refresh --cluster <cluster-name-or-id>
-
Watch for the refresh operation to complete by reviewing the Master Health in the cluster details.
ibmcloud ks cluster get -c <cluster_name_or_ID>