Centralize communication through a VPC Transit Hub and Spoke architecture - Part one
This tutorial may incur costs. Use the Cost Estimator to generate a cost estimate based on your projected usage.
A Virtual Private Cloud (VPC) provides network isolation and security in the IBM Cloud. A VPC can be a building block that encapsulates a corporate division (marketing, development, accounting, ...) or a collection of microservices owned by a DevSecOps team. VPCs can be connected to an on-premises enterprise and each other. This may create the need to route traffic through centralized firewall-gateway appliances. This tutorial will walk through the implementation of a hub and spoke architecture depicted in this high-level view:
This is part one of a two part tutorial. This part will introduce the VPC transit hub as the conduit to the enterprise. Enterprise to spoke VPC connectivity between microservices will be discussed and implemented. This architecture will support a number of scenarios:
- The hub is a central point of traffic routing between enterprise and the cloud.
- Enterprise to cloud traffic is routed through the hub, and can be monitored, and logged through a Network Function Virtualization (NFV) appliance running inside the hub.
- The hub can monitor all or some of the traffic: spoke <-> spoke, spoke <-> transit, or spoke <-> enterprise.
- The hub can hold shared microservices used by spokes.
- The hub can hold shared cloud resources, like databases, accessed through virtual private endpoint gateways controlled with VPC security groups and subnet access control lists, shared by spokes.
- The hub can hold the VPN resources that are shared by the spokes.
(Part two) will extend this tutorial by routing all VPC to VPC traffic through the hub, implement a highly available firewall-router and route traffic to IBM Cloud service instances with DNS resolution.
There is a companion GitHub repository that provisions resources and configures routing in incremental layers. In the tutorial thin layers enable the introduction of bite size challenges and solutions.
During the journey the following are explored:
- VPC Network planning.
- VPC egress and ingress routing.
- Connectivity via IBM Cloud® Direct Link.
- Connectivity via IBM Cloud Transit Gateway.
- Virtual Network Functions.
A layered architecture will introduce resources and demonstrate connectivity. Each layer will add additional connectivity and resources. A layer may introduce small problems and demonstrate solutions in the context of a larger architecture. The layers are implemented using Infrastructure as Code in the form of Terraform configuration files. It will be possible to change parameters, like number of zones, by changing a Terraform variable.
Objectives
- Understand the concepts behind a VPC based hub and spoke model.
- Understand the implementation of a firewall-router and a transit VPC environment.
- Understand VPC ingress and egress routing.
- Identify and optionally resolve asymmetric routing issues.
- Connect VPCs via a Transit Gateway.
Before you begin
This tutorial requires:
terraform
to use Infrastructure as Code to provision resources,python
to optionally run the pytest commands,- Implementing a firewall-router will require that you enable IP spoofing checks,
- An SSH key to connect to the virtual servers. If you don't have an SSH key, follow the instructions for creating a key for VPC.
See the prerequisites for a few options including a Dockerfile to easily create the prerequisite environment.
In addition:
- Check for user permissions. Be sure that your user account has sufficient permissions to create and manage all the resources in this tutorial. See the list of:
IP Address and Subnet Layout
In this step you will provision the VPC network resources. Carefully plan by designing an addressing plan for a VPC and use non-overlapping CIDR blocks.
It is tempting to divide up the CIDR space first by VPC but this complicates routing. Instead think of an availability zone as a single CIDR block and each VPC as consuming a slice of it.
This diagram shows just zone 1 in more detail. The subnet sizes and layout are identical in the other zones:
Above the enterprise is on the left and the IBM Cloud on the right. In the IBM Cloud for simplictiy a single zone is depicted for the transit VPC and Spoke 0. Notice the CIDR blocks do not overlap and VPCs all consume a CIDR block in each zone:
- The on-premises CIDR is 192.168.0.0/16.
- The zones in this multi-zone region are 10.*.0.0/16. The second digit: 1, 2, 3 is the zone number (shown for Dallas/us-south):
- 10.1.0.0/16, zone 1, Dallas 1, us-south-1.
- 10.2.0.0/16, zone 2, Dallas 2, us-south-2.
- 10.3.0.0/16, zone 3, Dallas 3, us-south-3.
- The transit VPC consumes CIDRs 10.*.15.0/24:
- 10.1.15.0/24, zone 1.
- 10.2.15.0/24, zone 2.
- 10.3.15.0/24, zone 3.
- Spoke 0 consumes 10.*.0.0/24 or CIDRs:
- 10.1.0.0/24, zone 1.
- 10.2.0.0/24, zone 2.
- 10.3.0.0/24, zone 3.
- The subnet CIDRs further divide the /24 into /26.
The subnets in the transit and spoke are for the different resource types:
- worker - network accessible compute resources VPC instances, load balancers, Red Hat OpenShift, etc. VPC instances are demonstrated in this tutorial.
- dns - DNS Services location appliances used in part two.
- vpe - VPE for VPC used in part two.
- fw - firewall-router VPC instances (only in transit).
Provision VPC network resources
-
The companion GitHub Repository has the source files to implement the architecture. In a desktop shell clone the repository:
git clone https://github.com/IBM-Cloud/vpc-transit cd vpc-transit
-
The config_tf directory contains configuration variables that you are required to configure.
cp config_tf/template.terraform.tfvars config_tf/terraform.tfvars
-
Edit config_tf/terraform.tfvars and use the comments in that file as your guide.
-
Since it is important that each layer is installed in the correct order and some steps in this tutorial will install multiple layers a shell command ./apply.sh is provided. The following will display help:
./apply.sh
-
You could apply all of the layers configured by executing
./apply.sh : :
. The colons are shorthand for first (or config_tf) and last (power_tf). The -p prints the layers:./apply.sh -p : :
It will look something like:
directories: config_tf enterprise_tf transit_tf spokes_tf transit_spoke_tgw_tf test_instances_tf test_lbs_tf enterprise_link_tf firewall_tf transit_ingress_tf spokes_egress_tf all_firewall_tf all_firewall_asym_tf dns_tf vpe_transit_tf vpe_spokes_tf power_tf
-
If you don't already have one, obtain a Platform API key and export the API key for use by Terraform:
export IBMCLOUD_API_KEY=YourAPIKEy
-
In this first step apply in config_tf, enterprise_tf, transit_tf, spokes_tf and transit_spoke_tgw_tf:
./apply.sh : transit_spoke_tgw_tf
The VPCs and subnets have been created. The transit VPC and spoke VPCs have been connected through a provisioned Transit Gateway. Open the Virtual Private Clouds in the browser. Open the transit VPC and note the CIDR blocks for address prefixes and subnets. Examine the enterprise and spoke VPCs as well. Open Transit Gateway and click the transit gateway to see the connection between the transit and spoke VPCs.
Create test instances
VPC Virtual Server Instances, VSIs, are provisioned to test the network connectivity. A test instance will be added to each of the worker subnets (one per zone) in the enterprise, transit and each of the spokes. If the default configuration of 3 zones and 2 spokes is used then 12 instances will be provisioned.
-
Create the test instances
./apply.sh test_instances_tf
It can be enlightening to explore the resources created at each step in the IBM Cloud console. Optionally open the Virtual Private Clouds. On the left click on the Virtual server instances and notice the instances that were created.
Testing
This tutorial will add communication paths one layer at a time. A pytest test suite will be used to exhaustively tests communication paths. By the end of the tutorial all of the tests are expected to pass.
It is not required for the reader to use pytest to verify the results. Follow along in the tutorial, apply the layers, and trust the results described in the tutorial. The reader can still explore the VPC resources like VSIs, subnets and route tables after they are created.
Each pytest test will SSH to one of the instances and perform a type of connectivity test, like executing a curl
command to one of the other instances. The default SSH environment is used to log in to the instances.
If you see unexpected test results try the pytest troubleshooting section.
-
Run the zone 1
curl
tests in the suite by using the -m (markers) flag. Choose the tests marked with curl, lz1 (left zone 1) and rz1 (right zone 1).Your expected results are: Connectivity within a VPC, like enterprise <-> enterprise will be PASSED. Connectivity between transit and spokes will be Passed. Cross VPC from enterprise -> transit or spokes will be FAILED.
pytest -m "curl and lz1 and rz1"
Below is an example output:
root@ea28970e0897:/usr/src/app# pytest -m "curl and lz1 and rz1" ===================================================== test session starts ====================================================== platform linux -- Python 3.12.3, pytest-8.1.1, pluggy-1.4.0 -- /usr/local/bin/python cachedir: .pytest_cache rootdir: /usr/src/app configfile: pytest.ini testpaths: py plugins: xdist-3.5.0 collected 36 items / 20 deselected / 16 selected py/test_transit.py::test_curl[l-enterprise-z1-worker -> r-enterprise-z1-worker] PASSED [ 6%] py/test_transit.py::test_curl[l-enterprise-z1-worker -> r-transit-z1-worker] FAILED [ 12%] py/test_transit.py::test_curl[l-enterprise-z1-worker -> r-spoke0-z1-worker] FAILED [ 18%] py/test_transit.py::test_curl[l-enterprise-z1-worker -> r-spoke1-z1-worker] FAILED [ 25%] py/test_transit.py::test_curl[l-transit-z1-worker -> r-enterprise-z1-worker] FAILED [ 31%] py/test_transit.py::test_curl[l-transit-z1-worker -> r-transit-z1-worker] PASSED [ 37%] py/test_transit.py::test_curl[l-transit-z1-worker -> r-spoke0-z1-worker] PASSED [ 43%] py/test_transit.py::test_curl[l-transit-z1-worker -> r-spoke1-z1-worker] PASSED [ 50%] py/test_transit.py::test_curl[l-spoke0-z1-worker -> r-enterprise-z1-worker] FAILED [ 56%] py/test_transit.py::test_curl[l-spoke0-z1-worker -> r-transit-z1-worker] PASSED [ 62%] py/test_transit.py::test_curl[l-spoke0-z1-worker -> r-spoke0-z1-worker] PASSED [ 68%] py/test_transit.py::test_curl[l-spoke0-z1-worker -> r-spoke1-z1-worker] PASSED [ 75%] py/test_transit.py::test_curl[l-spoke1-z1-worker -> r-enterprise-z1-worker] FAILED [ 81%] py/test_transit.py::test_curl[l-spoke1-z1-worker -> r-transit-z1-worker] PASSED [ 87%] py/test_transit.py::test_curl[l-spoke1-z1-worker -> r-spoke0-z1-worker] PASSED [ 93%] py/test_transit.py::test_curl[l-spoke1-z1-worker -> r-spoke1-z1-worker] PASSED [100%] =================================================== short test summary info ==================================================== FAILED py/test_transit.py::test_curl[l-enterprise-z1-worker -> r-transit-z1-worker] - assert False FAILED py/test_transit.py::test_curl[l-enterprise-z1-worker -> r-spoke0-z1-worker] - assert False FAILED py/test_transit.py::test_curl[l-enterprise-z1-worker -> r-spoke1-z1-worker] - assert False FAILED py/test_transit.py::test_curl[l-transit-z1-worker -> r-enterprise-z1-worker] - assert False FAILED py/test_transit.py::test_curl[l-spoke0-z1-worker -> r-enterprise-z1-worker] - assert False FAILED py/test_transit.py::test_curl[l-spoke1-z1-worker -> r-enterprise-z1-worker] - assert False ========================================= 6 failed, 10 passed, 20 deselected in 38.76s =========================================
A change to the network configuration can take a couple of test runs for the underlying VPC network system to become consistent. If you do not see the expected results initially be prepared to run the test again a couple of times.
The r- and l- stand for right and left. The middle part of the name identifies enterprise, transit, spoke0, spoke1, ... The z1, z2, ... identify the zone. The test will SSH to the left instance. On the left instance the connectivity to the right instance is attempted. The test_curl performs a curl connectivity on the left instance to the right instance.
In summary the test test_curl[l-enterprise-z1 -> r-transit-z1]
will:
- SSH to a test instance in enterprise zone 1.
- Execute a curl to transit zone 1.
- Assert the return string contains the ID of transit zone 1 to mark pass or fail.
The README.md in the companion GitHub Repository has more details and the source code.
Connect Enterprise to Transit via Direct Link and Transit Gateway
Provision a IBM Cloud® Direct Link using Transit Gateway.
IBM Cloud® Direct Link is a high speed secure data path for connecting an enterprise to the IBM Cloud. In this tutorial Transit Gateway is used for distribution. The use of Transit Gateway is optional for an on-premises connection.
The enterprise in this tutorial is simulated with another VPC. Connecting this simulated enterprise (actually another VPC) via the Transit Gateway will ensure an experience very close to what you would experience with a Direct Link.
-
Apply the enterprise_link_tf layer:
./apply.sh enterprise_link_tf
-
Run the zone 1 curl tests in the suite my using the -m (markers) flag. Choose the tests marked with curl, lz1 (left zone 1) and rz1 (right zone 1).
Your expected results are: Connectivity within a VPC, transit <-> spoke(s), enterprise <-> transit, spoke(s) <-> spoke(s) pass but enterprise <-> spoke(s) fail.
pytest -m "curl and lz1 and rz1"
Connect Enterprise to Spoke(s) via Transit NFV Firewall-Router
The incentive for a transit VPC for enterprise <-> cloud traffic is typically to route, inspect, monitor and log network traffic. In this step a firewall-router appliance will be installed in each zone of the transit VPC.
NFV Router
Provision the firewall-router appliances. An ingress route table for Transit Gateway has been added to the transit VPC as indicated by the dotted lines. A subnet has been created in each of the zones of the transit VPC to hold the firewall-router.
Connectivity from the enterprise to a spoke is achieved through a Network Function Virtualization, NFV, firewall-router instance in the transit VPC. In production you can choose one from the catalog or bring your own. This demonstration will use an Ubuntu stock image with kernel iptables set up to forward all packets from the source to destination. In this tutorial, no firewall inspection is performed.
The Terraform configuration will configure the firewall-router instance with allow_ip_spoofing. You must enable IP spoofing checks before continuing.
-
Apply the firewall_tf layer:
./apply.sh firewall_tf
-
Run the test suite.
Your expected results are: Connectivity within a VPC, enterprise -> transit, enterprise <-> spoke same zone pass. But all transit -> spoke, all transit -> enterprise fail due to asymmetric routing issues.
pytest -m "curl and lz1 and (rz1 or rz2)"
Part two of this tutorial will route all VPC <-> different VPC traffic through the firewall-router and resolve these issues. But first it is important to learn what is happening.
Ingress Routing
Traffic reaches the firewall-router appliance through routing tables.
- Visit the VPCs in the IBM Cloud console.
- Select the transit VPC.
- Click on Manage routing tables.
- Click on the tgw-ingress routing table.
The zone is determined by the Transit Gateway which will examine the destination IP address of each packet and route it to the matching zone based on the routes learned. The Transit Gateway learns the routes advertised from the connections. Each VPC will advertise its address prefixes which allows VPCs to communicate with each other after connecting to a Transit Gateway. But how would the spokes learn the routes to the enterprise? How does the enterprise learn the routes to the spokes? The enterprise and spokes are not connected to the same Transit Gateway.
Both sets of routes are in the transit's ingress routing table (shown for Dallas/us-south). And the Advertise flag is set to ON to pass those routes to all Transit Gateways.
Zone | Destination | Next hop | Advertise |
---|---|---|---|
Dallas 1 | 10.1.0.0/16 | 10.1.15.197 | On |
Dallas 2 | 10.2.0.0/16 | 10.2.15.197 | On |
Dallas 3 | 10.3.0.0/16 | 10.3.15.197 | On |
Dallas 1 | 192.168.0.0/16 | 10.1.15.197 | On |
Dallas 2 | 192.168.0.0/16 | 10.2.15.197 | On |
Dallas 3 | 192.168.0.0/16 | 10.3.15.197 | On |
The next_hop identifies the firewall-router. In the table above 10.1.15.196 zone Dallas 1 and 10.2.15.196 zone Dallas 2, etc. You can observe this using the IBM Cloud console.
- Open Virtual server instances for VPC to find the fw instances and associated Reserved IP (click the Name column header to sort).
- Match them up with the table above to verify the next hop relationship.
Removing the firewall for Transit destination traffic
The IBM Cloud VPC uses the industry standard state-based routing for secure TCP connection tracking. It requires that the TCP connections use the same path on the way in as the way out. One exception is Direct Server Return used by routers like Network Load Balancers. It allows incoming connections from the enterprise to pass through the firewall to the transit test instance and then return directly to the originator.
This does not help with the traffic originating in the transit test instance passing through the Transit Gateway then back through ingress routing to the firewall-router. This connection gets stuck at the firewall-router (3) and will not get forwarded back to the worker as shown in red below. Traffic transit -> enterprise and transit -> spoke are failing.
One possible solution is to stop sending traffic destined to the transit VPC to the firewall-router. The wide ingress routes for the transit are currently routing traffic to the firewall-router. More specific routes can be added for the transit to Delegate to the default behavior - send directly to the intended destination instead of the firewall-router.
This diagram shows the traffic flow that is desired for this step. Only the enterprise <-> spoke is passing through the firewall:
- enterprise <-> transit
- spoke <-> transit
- spoke <-> spoke
- enterprise <--transit firewall-router--> spoke
This routing can be achieved by adding these routes to the transit ingress route table:
Zone | Destination | Next hop |
---|---|---|
Dallas 1 | 10.1.15.0/24 | Delegate |
Dallas 2 | 10.2.15.0/24 | Delegate |
Dallas 3 | 10.3.15.0/24 | Delegate |
-
To observe the current value of the ingress route table visit the Routing tables for VPC in the IBM Cloud console. Select the transit VPC from the drop down and then select the tgw-ingress routing table.
-
Make the changes to the routing table by applying the transit_ingress layer:
./apply.sh transit_ingress_tf
-
Refresh the browser display of the routing table to observe the new routes.
-
Run the test suite.
Your expected results are: All tests will result in PASSED.
pytest -m "curl and lz1 and (rz1 or rz2)"
It is interesting to note how cross zone traffic flows between the enterprise and spokes in the configuration. Enterprise sends traffic to the correct zone and through the firewall-router via ingress routing in the transit VPC. The Transit Gateway has learned that 192.168.0.0/16 is available in all zones and will route to the transit VPC using the advertised enterprise routes in the same zone as the spoke as shown in the diagram below:
Routing Summary
Basic routing is complete:
- enterprise <-> transit
- transit <-> spoke(s)
- enterprise <--(transit firewall-router)--> spoke
Production Notes and Conclusions
The VPC reference architecture for IBM Cloud for Financial Services has much more detail on securing workloads in the IBM Cloud.
Some obvious changes to make:
- CIDR blocks were chosen for clarity and ease of explanation. The Availability Zones in the Multi zone Region could be 10.0.0.0/10, 10.64.0.0/10, 10.128.0.0/10 to conserve address space. The address space for Worker nodes could be expanded at the expense of firewall, DNS and VPE space.
- Security Groups for each of the network interfaces for worker VSIs, Virtual Private Endpoint Gateways, DNS Locations and firewalls should all be carefully considered.
- Network Access Control Lists for each subnet should be carefully considered.
Floating IPs were attached to all test instances to support connectivity tests via SSH. This is not required or desirable in production.
Implement context-based restrictions rules to further control access to all resources.
In this tutorial you created a hub VPC and a set of spoke VPCs. You identified the required Availability Zones for the architecture and created a set of subnets in the VPCs. You created a transit VPC firewall-router in each zone to forwards traffic. Test instances were used to verify connectivity and identify potential problems. Routing table routes were used to identify the traffic paths required.
Remove resources
It is not required to remove the resources if you plan to continue with the second part of this tutorial.
Execute terraform destroy
in all directories in reverse order using the ./apply.sh
command:
./apply.sh -d : spokes_egress_tf
Expand the tutorial
You are encouraged to continue on to part two of this tutorial where all cross VPC traffic is routed through the firewall-router, VPE for VPC and DNS are examined.
Your architecture will likely be different than the one presented but will likely be constructed from the fundamental components discussed here. Ideas to expand this tutorial:
- Integrate incoming public Internet access using IBM Cloud® Internet Services.
- Add flow log capture in the transit.
- Put each of the spokes in a separate account in an enterprise.
- Force some of the spoke to spoke traffic through the firewall and some not through the firewall.
- Replace the worker VSIs with Red Hat OpenShift on IBM Cloud and VPC load balancer.
- Force all out bound traffic through the firewall in the transit VPC and through Public gateways .