Reviewing service, API server, and worker node logs

Forward audit logs for IBM Cloud Kubernetes Service, the Kubernetes API server, and the worker nodes to a logging instance such as IBM Cloud® Activity Tracker. With audit logs, you're able to understand better what operations are initiated by users in your cluster, which can help you troubleshoot issues or report compliance to industry and internal standards.

Kubernetes API server audit logs

To monitor user-initiated, Kubernetes administrative activity made within your cluster, you can collect and forward audit events that are passed through your Kubernetes API server to IBM Log Analysis or an external server.

Considerations and prerequisites

Before you set up a Kubernetes API audit configuration, review the following information.

Audit logs use the following policies in the IBM-Cloud/kube-samples GitHub repo. Starting with version 1.30, the policies were updated to closely follow those used by Red Hat for OpenShift. Both sets of policies can be found below.

You can't modify the default policy or apply your own custom policy.

For Kubernetes audit logs and verbosity, see the Kubernetes documentation.
Only one audit webhook can be created in a cluster.
You must have the Administrator IBM Cloud IAM platform access role for the IBM Cloud Kubernetes Service cluster.

To get started, follow the instructions to send Kubernetes API audit logs to a resource in the IBM Cloud private network, or to an external server.

Forwarding Kubernetes API audit logs to Log Analysis

To forward audit logs to IBM Log Analysis, you can create a Kubernetes audit system by using the provided image and deployment.

The following example uses the icr.io/ibm/ibmcloud-kube-audit-to-logdna image to forward logs to Log Analysis. This image is for demonstration purposes only. For a production solution, configure and maintain your own log forwarding image.

The Kubernetes audit system in your cluster consists of an audit webhook, a log collection service and web server app, and a logging agent. The webhook collects the Kubernetes API server events from your cluster master. The log collection service is a Kubernetes ClusterIP service that is created from an image from the public IBM Cloud registry. This service exposes a simple node.js HTTP web server app that is exposed only on the private network. The web server app parses the log data from the audit webhook and creates each log as a unique JSON line. Finally, the logging agent forwards the logs from the web server app to IBM Log Analysis, where you can view the logs.

Before you begin: Ensure that you reviewed the considerations and prerequisites and that you have the Administrator IBM Cloud IAM platform access role for IBM Log Analysis.

Target the global container registry for public IBM Cloud images.
```
ibmcloud cr region-set global
```
Optional: For more information about the kube-audit image, inspect icr.io/ibm/ibmcloud-kube-audit-to-logdna.
```
ibmcloud cr image-inspect icr.io/ibm/ibmcloud-kube-audit-to-logdna
```

Create a configuration file named ibmcloud-kube-audit.yaml. This configuration file creates a log collection service and a deployment that pulls the icr.io/ibm/ibmcloud-kube-audit-to-logdna image to create a log collection container.

apiVersion: v1
kind: List
metadata:
  name: ibmcloud-kube-audit
items:
  - apiVersion: v1
    kind: Namespace
    metadata:
      name: ibm-kube-audit
      labels:
        pod-security.kubernetes.io/enforce: restricted
        pod-security.kubernetes.io/enforce-version: latest
        pod-security.kubernetes.io/audit: restricted
        pod-security.kubernetes.io/audit-version: latest
        pod-security.kubernetes.io/warn: restricted
        pod-security.kubernetes.io/warn-version: latest
  - apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: ibmcloud-kube-audit
      namespace: ibm-kube-audit
      labels:
        app: ibmcloud-kube-audit
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: ibmcloud-kube-audit
      template:
        metadata:
          labels:
            app: ibmcloud-kube-audit
        spec:
          containers:
            - name: ibmcloud-kube-audit
              image: 'icr.io/ibm/ibmcloud-kube-audit-to-logdna:latest'
              imagePullPolicy: Always
              ports:
                - containerPort: 3000
              securityContext:
                allowPrivilegeEscalation: false
                runAsNonRoot: true
                capabilities:
                  drop:
                  - ALL
                seccompProfile:
                  type: RuntimeDefault
  - apiVersion: v1
    kind: Service
    metadata:
      name: ibmcloud-kube-audit-service
      namespace: ibm-kube-audit
      labels:
        app: ibmcloud-kube-audit
    spec:
      selector:
        app: ibmcloud-kube-audit
      ports:
        - protocol: TCP
          port: 80
          targetPort: 3000
      type: ClusterIP
  - kind: NetworkPolicy
    apiVersion: networking.k8s.io/v1
    metadata:
      name: ibmcloud-kube-audit
      namespace: ibm-kube-audit
    spec:
      podSelector:
        matchLabels:
          app: ibmcloud-kube-audit
      policyTypes:
      - Ingress
      ingress:
      - ports:
        - protocol: TCP
          port: 3000
        from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: konnectivity-agent

Create the deployment in the ibm-kube-audit namespace of your cluster.
```
kubectl create -f ibmcloud-kube-audit.yaml
```

Verify that the ibmcloud-kube-audit-service pod has a STATUS of Running.

kubectl get pods -n ibm-kube-audit -l app=ibmcloud-kube-audit

Example output

NAME                                             READY   STATUS             RESTARTS   AGE
ibmcloud-kube-audit-c75cb84c5-qtzqd              1/1     Running   0          21s

Verify that the ibmcloud-kube-audit-service service is deployed in your cluster.

kubectl get svc -n ibm-kube-audit -l app=ibmcloud-kube-audit

Example output

NAME                          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
ibmcloud-kube-audit-service   ClusterIP   172.21.xxx.xxx   <none>        80/TCP           1m

Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster. Make sure to specify the --admin option to download the client-certificate and the client-key files to your local machine. These files are used later to configure the audit webhook.
```
ibmcloud ks cluster config --cluster <cluster> --admin
```

Query the certificate-authority of the cluster and save it into a file.

ibmcloud ks cluster ca get -c <cluster> --output json | jq -r .caCert | base64 -D > <certificate-authority>

View your current config by running the kubectl config view command and review the output for the client-certificate and client-key.

kubectl config view --minify

Example output

clusters:
- cluster:
    ...
    ...
    client-certificate: /Users/user/.bluemix/plugins/container-service/clusters/cluster-name-a111a11a11aa1aa11a11-admin/admin.pem
    client-key: /Users/user/.bluemix/plugins/container-service/clusters/cluster-name-a111a11a11aa1aa11a11-admin/admin-key.pem

Configure the audit webhook and specify the certificate-authority, client-certificate, and client-key. The certificate-authority was retrieved in step 8 and theclient-certificate and client-key were retrieved in the previous step.

ibmcloud ks cluster master audit-webhook set --cluster CLUSTER --remote-server https://127.0.0.1:2040/api/v1/namespaces/ibm-kube-audit/services/ibmcloud-kube-audit-service/proxy/post --ca-cert CERTIFICATE-AUTHORITY --client-cert CLIENT-CERT --client-key CLIENT-KEY [--policy default|verbose]

Verify that the audit webhook is created in your cluster.

ibmcloud ks cluster master audit-webhook get --cluster <cluster_name_or_ID>

Example output

Server:   https://127.0.0.1:2040/api/v1/namespaces/ibm-kube-audit/services/ibmcloud-kube-audit-service/proxy/post   
Policy:   default

Apply the webhook to your Kubernetes API server by refreshing the cluster master. It might take several minutes for the master to refresh.
```
ibmcloud ks cluster master refresh --cluster <cluster_name_or_ID>
```
While the master refreshes, provision an instance of IBM Log Analysis and deploy a logging agent to every worker node in your cluster. The logging agent is required to forward logs from inside your cluster to the IBM Log Analysis service. If you already set up logging agents in your cluster, you can skip this step.
After the master refresh completes and the logging agents are running on your worker nodes, you can view your Kubernetes API audit logs in IBM Log Analysis.

After you set up the audit webhook in your cluster, you can monitor version updates to the kube-audit-to-logdna image by running ibmcloud cr image-list --include-ibm | grep ibmcloud-kube-audit. To see the version of the image that currently runs in your cluster, run kubectl get pods | grep ibmcloud-kube-audit to find the audit pod name, and run kubectl describe pod <pod_name> to see the image version.

Forwarding Kubernetes API audit logs to a resource in the IBM Cloud private network

Forward audit logs to a resource other than Log Analysis that is outside of your cluster and accessible in the IBM Cloud private network.

The following example uses the haproxytech/haproxy-alpine:2.6 image to forward logs. This image is for demonstration purposes only and should not be used in production environments. For a production solution, configure and maintain your own log forwarding image.

Before you begin, ensure that you reviewed the considerations and prerequisites.

Create a new directory kube-audit-forwarder and create a file haproxy.cfg in it with the following contents. Do not forget to replace <REMOTE-IP>:<REMOTE-PORT> in the file to the IP address and port of your remote log consumer.
```
global
  log stdout format raw local0 info
defaults
  mode http
  timeout client 10s
  timeout connect 5s
  timeout server 10s
  timeout http-request 10s
  log global
frontend myfrontend
  bind :3000
  default_backend remotelogstash
# Use remote log consumer IP and port here
backend remotelogstash
  server s1 <REMOTE-IP>:<REMOTE-PORT> check
```
If your log consumer server is enforcing secure connection (TLS), you can add your certificate files to this directory and change the backend section in haproxy.cfg to use these files. For more information, see the HAProxy documentation.

Create a configmap from the contents of kube-audit-forwarder directory.

kubectl create namespace ibm-kube-audit; kubectl create configmap -n ibm-kube-audit kube-audit-forwarder-cm --from-file=kube-audit-forwarder

Create a configuration file that is named kube-audit-forwarder-remote-private-ip.yaml. This configuration file creates a deployment and a service that forwards audit logs from the cluster to the IP address of the remote resource through the IBM Cloud private network.

kind: Deployment
apiVersion: apps/v1
metadata:
  labels:
    app: kube-audit-forwarder
  name: kube-audit-forwarder
  namespace: ibm-kube-audit
spec:
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: kube-audit-forwarder
  strategy:
    rollingUpdate:
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: kube-audit-forwarder
    spec:
      containers:
      - image: haproxytech/haproxy-alpine:2.6
        imagePullPolicy: IfNotPresent
        name: haproxy
        volumeMounts:
        - name: config-volume
          mountPath: /usr/local/etc/haproxy/haproxy.cfg
          subPath: haproxy.cfg
      volumes:
      - name: config-volume
        configMap:
          name: kube-audit-forwarder-cm
---
apiVersion: v1
kind: Service
metadata:
  name: kube-audit-forwarder
  namespace: ibm-kube-audit
spec:
  selector:
    app: kube-audit-forwarder
  ports:
    - protocol: TCP
      port: 80
      targetPort: 3000
---
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: kube-audit-forwarder
  namespace: ibm-kube-audit
spec:
  podSelector:
    matchLabels:
      app: kube-audit-forwarder
  policyTypes:
  - Ingress
  ingress:
  - ports:
    - protocol: TCP
      port: 3000
    from:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
      podSelector:
        matchLabels:
          k8s-app: konnectivity-agent
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
      podSelector:
        matchLabels:
          app: konnectivity-agent
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
      podSelector:
        matchLabels:
          app: vpn

If you added certificate files to the kube-audit-forwarder in the previous step, do not forget to list those files in volumeMounts section as a subPath.

Create the deployment and service.

kubectl create -f kube-audit-forwarder-remote-private-ip.yaml

Verify that the kube-audit-forwarder deployment and service is deployed in your cluster.

kubectl get svc -n ibm-kube-audit

Example output

NAME                          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
...
kube-audit-forwarder  ClusterIP   10.xxx.xx.xxx   <none>        80/TCP           1m

kubectl get deployment -n ibm-kube-audit

Example output

NAME                   READY   UP-TO-DATE   AVAILABLE   AGE
...
kube-audit-forwarder   1/1     1            1           6m27s

Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster. Make sure to specify the --admin option to download the client-certificate and the client-key files to your local machine. These files are used later to configure the audit webhook.
```
ibmcloud ks cluster config --cluster <cluster> --admin
```

Query the certificate-authority of the cluster and save it into a file.

 ibmcloud ks cluster ca get -c <cluster> --output json | jq -r .caCert | base64 -D > <certificate-authority>

View your current config by running the kubectl config view command and review the output for the client-certificate and client-key.

kubectl config view --minify

Example output

clusters:
- cluster:
    ...
    ...
    client-certificate: /Users/user/.bluemix/plugins/container-service/clusters/cluster-name-a111a11a11aa1aa11a11-admin/admin.pem
    client-key: /Users/user/.bluemix/plugins/container-service/clusters/cluster-name-a111a11a11aa1aa11a11-admin/admin-key.pem

Configure the audit webhook and specify the certificate-authority, client-certificate, and client-key that you retrieved in the steps 5-7.

ibmcloud ks cluster master audit-webhook set --cluster <cluster> --remote-server https://127.0.0.1:2040/api/v1/namespaces/ibm-kube-audit/services/kube-audit-forwarder/proxy/post --ca-cert <certificate-authority> --client-cert <client-certificate> --client-key <client-key> [--policy default|verbose]

Verify that the audit webhook is created in your cluster.

ibmcloud ks cluster master audit-webhook get --cluster <cluster_name_or_ID>

Example output

OK
Server:            https://127.0.0.1:2040/api/v1/namespaces/ibm-kube-audit/services/kube-audit-forwarder/proxy/post
Policy:            default

Apply the webhook to your Kubernetes API server by refreshing the cluster master. The master might take several minutes to refresh.
```
ibmcloud ks cluster master refresh --cluster <cluster_name_or_ID>
```

After the master refresh completes, your logs are sent to the private IP address of your logging resource.

Forwarding Kubernetes API audit logs to an external server on the public Internet

To audit any events that are passed through your Kubernetes API server, you can create a configuration that uses Fluentd to forward events to an external server.

Before you begin, ensure that you reviewed the considerations and prerequisites. Note that log filters are not supported.

Set up the webhook. If you don't provide any information in the options, a default configuration is used.

ibmcloud ks cluster master audit-webhook set --cluster <cluster_name_or_ID> --remote-server <server_URL_or_IP> --ca-cert <CA_cert_path> --client-cert <client_cert_path> --client-key <client_key_path> [--policy default|verbose]

Understanding this command's components
Option	Description
`<cluster_name_or_ID>`	The name or ID of the cluster.
`<server_URL>`	A publicly accessible URL or IP address for the remote logging service that you want to send logs to. Certificates are ignored if you provide an unsecure server URL.
`<CA_cert_path>`	The file path for the CA certificate that is used to verify the remote logging service.
`<client_cert_path>`	The file path for the client certificate that is used to authenticate against the remote logging service.
`<client_key_path>`	The file path for the corresponding client key that is used to connect to the remote logging service.

Verify that log forwarding was enabled by viewing the URL for the remote logging service.

ibmcloud ks cluster master audit-webhook get --cluster <cluster_name_or_ID>

Example output

OK
Server:            https://8.8.8.8

Apply the configuration update by restarting the Kubernetes master.

ibmcloud ks cluster master refresh --cluster <cluster_name_or_ID>

Optional: If you want to stop forwarding audit logs, you can disable your configuration.
1. For the cluster that you want to stop collecting API server audit logs from: Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.
2. Disable the webhook back-end configuration for the cluster's API server.
```
ibmcloud ks cluster master audit-webhook unset --cluster <cluster_name_or_ID>
```
3. Apply the configuration update by restarting the Kubernetes master.
```
ibmcloud ks cluster master refresh --cluster <cluster_name_or_ID>
```

Managing API server log forwarding

See Verifying, updating, and deleting log forwarding.

If you are noticing errors when retrieving audit logs when they've been previously working, this might be due to the certificate authority used for audit logs expiring or rotated. Repeat the previous steps to setup a webhook and renew the certificate. Also, you can look for the Kubernetes metric called apiserver_audit_error_total{plugin="webhook"} which indicates whether your webhook certificate has expired.

Service audit logs

By default, IBM Cloud Kubernetes Service generates and sends events to IBM Cloud Activity Tracker. To see these events, you must create an IBM Cloud Activity Tracker instance. For more information, see IBM Cloud Activity Tracker events.