Learning about Code Engine architecture and workload isolation
IBM Cloud® Code Engine is the IBM Cloud platform that unifies container images, 12-factor-apps, functions, and batch jobs as a one-stop-shop. It's a multi-tenant system that consists of three major building blocks: A control plane, a (set of) shard (or shards), and a routing layer. The control plane and the shards are realized as separate multi-zone Kubernetes clusters. The following diagram gives a graphical overview of the architecture.
Code Engine is based on IBM Cloud Kubernetes Service clusters and depends on the components and workload isolation of the IBM Cloud Kubernetes Service. For more information, see IBM Cloud Kubernetes Service VPC cluster architecture.
All components are managed and owned by IBM and run in the IBM Cloud account. Each cluster is running in its own VPC and separated from other clusters.
The Code Engine control plane runs the components that are shared among all Code Engine users and make the Kubernetes cluster a true multi-tenant system. The control-plane consists of four microservices that are deployed on it.
| Component | Purpose |
|---|---|
| Resource broker | Creates and deletes Code Engine project resources in the IBM Cloud resource controller and requests the placement of the project on a shard. |
| Project placement controller | Selects a shard and requests the creation, deletion, and isolation of the project on the shard. |
| API server | Provides the target information (KUBECONFIG file) for the selected project. It also performs IAM access policy checks and writes audit records. |
Kube API proxy |
Proxies each API request to the proper shard cluster, perform IAM policy checks, and writes audit records. |
The shards are running the customer workload, such as builds, batch jobs, or apps. Therefore, the shard cluster runs the following microservices to control the customer workloads.
| Component | Purpose |
|---|---|
| Project isolation controller | Manages and isolates the Kubernetes namespace corresponding to the Code Engine project resource. It monitors and ensures the isolation aspects like role-based-access-control (RBAC), pod security policies, resource quota, and network policies are enforced. |
| Project domain and cert controller | Manages the domain and certificates for the route endpoint of the project. The endpoint consists of a DNS entry and a wildcard certificate. |
| Knative and Istio | Manage the lifecycle of applications. Knative is responsible for scaling the application. Istio is responsible for routing the traffic to the proper revision and container of the application. |
| Batch controller | Manages the lifecycle and containers for jobs and job runs. |
| Build controller | Manages the lifecycle and containers for builds and build runs. |
| Service binding and IBM Cloud operator | Manage the lifecycle of secrets that are associated to bindings of IBM Cloud services to applications and jobs. |
| IBM Cloud Object Storage event source controller | Manage the lifecycle of event subscriptions from the IBM Cloud Object Storage service. |
| Cluster node autoscaler | Scales the shard by adding and removing worker nodes based on capacity demand. |
| IBM Cloud Monitoring | Sends service metrics to IBM Cloud Monitoring. For more information about these metrics, see Monitoring for Code Engine. |
| IBM® Cloud Logs | Forward platform logs and metrics to IBM Cloud Logs. For more information, see Auditing events for Code Engine. |
Code Engine workload isolation
Code Engine is a multi-tenant, regional service where tenants share the same network and compute infrastructure. In particular, the network and compute infrastructure are shared resources and some management components are common to all tenants. Code Engine implements the following controls to separate tenants:
- Code Engine project resources are separated within a secured Kubernetes environment that is running in an IBM Cloud multi-zone region.
- Code Engine projects and its containing resources, such as application, builds, and jobs that run on shared clusters that use shared management components.
- To separate the access to project resources, Code Engine performs several levels of authentication and authorization checks within the
apiserverandkube-api-proxycomponents (see previous table),- IAM authentication and access policies checks are performed on a project level.
- To manage multi-tenant access to the underlying Kubernetes API, direct access to the API server is not allowed. Instead, use the Code Engine custom
Kube-api-proxyAPI for access. - Role-based access control checks are performed on a resource level to allow only authorized users to perform certain operations on project resources.
- The authorization is controlled by the customer by assigning
manager,reader, orwriterroles to users for a Code Engine project resource within IAM. - To restrict customer workload, Code Engine enforces the following concepts,
- Container isolation through various Linux isolation techniques. These techniques ensure multiple layers of security to prevent the privilege escalation of containers and to restrict containers to use a limited set of system privileges.
- Resource quota and
LimitRangeto prevent excessive resource consumption. - Network policies to control the network flows within the environment.
- Shared multi-tenant components are secured, for example, by disabling reverse lookup in
KubeDNS. - To limit the blast radius, each shard cluster is running in its own VPC, which is isolated from other shard VPCs.
- DDoS protection is enabled automatically for all endpoints, including customer applications, through Cloud Internet Services (CIS) with no additional cost to you. DDoS protection covers System Interconnection (OSI) Layer 3 and Layer 4 (TCP/IP) protocol attacks, but not Layer 7 (HTTP) attacks. See DDoS protection.