Physical infrastructure design

Public network

IBM Cloud data centers and network points of presence (PoPs) have multiple 1 Gbps or 10-Gbps connections to the top-tier transit and peering network carriers. Network traffic from anywhere in the world connects to the closest network PoP, and travels directly across the network to its data center, minimizing the number of network hops and handoffs between providers.

Inside the data center, IBM Cloud provides 1 Gbps, 10 Gbps, or 25 Gbps of network bandwidth to individual servers through a pair of separate, peer-aggregated front-end customer switches (FCS). These aggregated switches are attached to a pair of separate routers, FCR, for L3 networking.

This multitier design allows the network to scale across racks, rows, and pods within an IBM Cloud data center.

Private network

All IBM Cloud data centers and PoPs are connected by the private network backbone. This private network is separate from the public network, and it enables connectivity to services in IBM Cloud data centers around the world. Moving data between IBM Cloud data centers is done through multiple 10 Gbps or 40 Gbps connections to the private network.

Similar to the public network, the private network is multitiered in that servers and other infrastructure components are connected to aggregated back-end customer switches (BCS). These aggregated switches are attached to a pair of separate back-end customer routers (BCR) for L3 networking. The private network also supports the ability to use jumbo frames (MTU 9000) for physical host connections.

Host management

On the private network, each IBM Cloud server is connected for management by using the private primary network subnet. This connection allows Intelligent Platform Management Interface (IPMI) access to the server independently of its CPU, firmware, and operating system, for maintenance and administration purposes.

Primary and portable IP blocks

IBM Cloud allocates two types of IP addresses to be used within the IBM Cloud infrastructure:

• Primary IP addresses are assigned to devices, bare metal, and virtual servers that are provisioned by IBM Cloud. Do not assign any IP addresses in these blocks.
• Portable IP addresses are provided for you to assign and manage as needed. vCenter Server provisions several portable IP ranges for its use. Use only the portable IP address ranges that are assigned to specific NSX®-T components and specified for customer use. For example, Customer EDGE.

Primary or portable IP addresses can be made routable to any VLAN within your account when the account is configured as a Virtual Routing and Forwarding (VRF) account.

Virtual Routing and Forwarding

The IBM Cloud infrastructure customer portal account must be configured as a Virtual Routing and Forwarding (VRF) account to enable automatic global routing between subnet IP blocks. All accounts with Direct-Link connections must be converted to, or created as, a VRF account.

As various connectivity options along with network routing options require that the IBM Cloud account is in a VRF mode, it is recommended that the account is in VRF mode before provisioning vCenter Server.

Physical host connections

Each physical host in this design has two redundant pairs of 10 Gbps or 25 Gbps Ethernet connections into each IBM Cloud Top of Rack (ToR) switch (public and private). The adapters are set up as individual connections (unbonded) for a total of 4 x 10 Gbps or 4 x 25 Gbps connections. This configuration allows networking interface card (NIC) connections to work independently from each other.

Removing physical network connectivity to the public or private network for the IBM Cloud bare metal servers that are used within the Automated offering is not possible. Physical ports on the internal NIC of the bare metal can be disabled, but no support exists for unplugging the cables.

VLANs and underlay to overlay routing

The IBM Cloud for VMware Solutions offerings are designed with 3 VLANs, one public and two private, assigned upon deployment. As shown in the previous figure, the public VLAN is assigned to eth1 and eth3, and the private VLANs are assigned to eth0 and eth2.

The public and the first private VLAN created and assigned in this design are untagged by default within the IBM Cloud. Then, the additional private VLAN is trunked on the physical switch ports and tagged within the VMware port groups that are using these subnets.

The private network consists of two VLANs within this design. Four subnets are allocated to the first of these VLANs (here designated Private VLAN A):

• The first subnet is a primary private IP subnet range that IBM Cloud assigns to the physical hosts.
• The second subnet is used for management virtual machines (VMs), such as vCenter Server Appliance and NSX Controller.
• The third subnet is used for the encapsulated overlay network Tunnel Endpoints (VTEPs) assigned to each host and edge through the NSX Manager.
• The fourth subnet is used for egress from the encapsulated overlay network.

In addition to Private VLAN A, a second private VLAN (here designated Private VLAN B) exists to support VMware features such as vSAN™, vMotion, and NFS. As such, the VLAN is divided into two or more portable subnets:

• The first subnet is assigned to a kernel port group for vMotion traffic.
• The remaining subnet or subnets are used for storage traffic:
  • If you use vSAN, a subnet is assigned to kernel port groups that are used for vSAN traffic.
  • If you use NFS attached NAS, a subnet is assigned to a port group that is dedicated to NFS traffic.

All subnets that are configured as part of a VMware Cloud Foundation for Classic - Automated deployment use IBM Cloud-managed ranges, which ensure that any IP address can be routed to any data center within the IBM Cloud account when you need the connection.

Review the following table for a summary.

In this design, all VLAN-backed hosts and VMs are configured to point to the IBM Cloud back-end “private network” customer router (BCR) as the default route. While the Automated instances enable the use of Software-Defined Networking (SDN), network overlays created within a VMware instance that include routing to internal subnets are not known by the IBM Cloud managed routers.

If you want to route between the overlay and underlay, you must deploy an IBM firewall device for the particular default private VLAN when the Automated instance is deployed. This device allows the insertion of static routes and dynamic routing protocol peering with the overlay networking devices to allow routing between the underlay and the overlay.

The private network connections are configured to use a jumbo frame MTU size of 9000 to improve performance for large data transfers, such as storage and vMotion. This value is the maximum MTU that is allowed within VMware and by IBM Cloud. The public network connections use a standard Ethernet MTU of 1500. This value must be maintained as any changes might cause packet fragmentation over the internet.