Red Hat OpenShift 4.7 user provider infrastructure installation

Creating the Red Hat OpenShift Installer yaml file

Use the following table to document the parameters you need for your deployment. Examples are shown that match the deployment described in this document.

• The SSH key can be copied after it is displayed by using the command: cat /root/.ssh/id_rsa.pub.
• The pull secret collected from Red Hat®. For more information, see Red Hat OpenShift infrastructure providers (requires logging in to your Red Hat account).

Using the following install-config.yaml file shown in the figure, update it using your deployment details from the previous table:

1. Update the base domain name.
2. Update the metadata name with the Red Hat OpenShift cluster name.
3. Provide the following vCenter Server information:
   • vCenter IP address
   • vCenter username and password
   • Datacenter name
   • Datastore name
4. Paste the Pull Secret.
5. Paste the SSH key.
6. The following figure shows the install-config.yaml file. Copy the file to the clipboard.

apiVersion: v1
baseDomain: acmeskb.net
compute:
- hyperthreading: Enabled
  name: worker
  replicas: 0
controlPlane:
  hyperthreading: Enabled
  name: master
  replicas: 3
metadata:
  name: ocp
platform:
  vsphere:
    vcenter: 10.208.17.2
    username: administrator@vsphere.local
    password: 's3cretPassw0rd'
    datacenter: datacenter1
    defaultDatastore: vsanDatastore
pullSecret: 'COPY PULL SECRET HERE'
sshKey: 'COPY PUBLIC SSH KEY HERE'

In the SSH session to the bastion node with root privileges, use the following commands to create the install-config.yaml file:

cd /opt/ocpinstall
vi install-config.yaml

Type i to enter insert mode, paste the file contents. Press Esc, then type :wq to save the file and exit the vi editor.

The Red Hat OpenShift Installer deletes this file so to keep a copy use the following command:

cp install-config.yaml install-config.bak

Running the Red Hat OpenShift Ignition command

Now that the install-config.yaml is created and populated run the Red Hat OpenShift Installer to create the ignition files.

cd /opt/ocpinstall/
openshift-install create ignition-configs --dir=/opt/ocpinstall/

The Ignition files are valid for 24 hours and your Red Hat OpenShift deployment must be completed within this time. Otherwise, you must regenerate the Ignition files. For more information, see Troubleshooting Red Hat OpenShift problems.

The following files are produced by the Red Hat OpenShift Installer:

.
├── auth
│   ├── kubeadmin-password
│   └── kubeconfig
├── bootstrap.ign
├── master.ign
├── metadata.json
└── worker.ign

Copying the bootstrap Ignition file to the web server

The bootstrap.ign file needs to be copied to the document root of NGINX. The bootstrap ignition file must be hosted on the web server so the bootstrap node can reach it on the first start. The Ignition file is too large to fit in a vApp property.

Copy the bootstrap.ign file to the document root of NGINX.

cp bootstrap.ign /usr/share/nginx/html

Terraform files

You must update the following Terraform files.

• ignition.tf - This file needs to be updated so that the DNS entry matches the deployment.
• terraform.tfvars - This file needs to be created and holds the variables that are used by the main.tf file
• main.tf - This file needs to be updated to remove the DNS section.

locals {
  mask = "${element(split("/", var.machine_cidr), 1)}"
  gw   = "${cidrhost(var.machine_cidr,1)}"

  ignition_encoded = "data:text/plain;charset=utf-8;base64,${base64encode(var.ignition)}"
}

data "ignition_file" "hostname" {
  count = "${var.instance_count}"

  filesystem = "root"
  path       = "/etc/hostname"
  mode       = "420"

  content {
    content = "${var.name}-${count.index}"
  }
}

data "ignition_file" "static_ip" {
  count = "${var.instance_count}"

  filesystem = "root"
  path       = "/etc/sysconfig/network-scripts/ifcfg-ens192"
  mode       = "420"

  content {
    content = <<EOF
TYPE=Ethernet
BOOTPROTO=none
NAME=ens192
DEVICE=ens192
ONBOOT=yes
IPADDR=${local.ip_addresses[count.index]}
PREFIX=${local.mask}
GATEWAY=${local.gw}
DOMAIN=${var.cluster_domain}
DNS1=10.187.214.66
EOF
  }
}

data "ignition_systemd_unit" "restart" {
  count = "${var.instance_count}"

  name = "restart.service"

  content = <<EOF
[Unit]
ConditionFirstBoot=yes
[Service]
Type=idle
ExecStart=/sbin/reboot
[Install]
WantedBy=multi-user.target
EOF
}

data "ignition_config" "ign" {
  count = "${var.instance_count}"

  append {
    source = "${var.ignition_url != "" ? var.ignition_url : local.ignition_encoded}"
  }

  systemd = [
    "${data.ignition_systemd_unit.restart.*.id[count.index]}",
  ]

  files = [
    "${data.ignition_file.hostname.*.id[count.index]}",
    "${data.ignition_file.static_ip.*.id[count.index]}",
  ]
}

bootstrap_ip = "192.168.133.9"
control_plane_ips = ["192.168.133.10","192.168.133.11","192.168.133.12"]
compute_ips = ["192.168.133.13","192.168.133.14","192.168.133.15"]
machine_cidr = "192.168.133.0/24"
cluster_id = "ocp"
cluster_domain = "ocp.dallas.ibm.local"
base_domain = "dallas.ibm.local"
vsphere_server = "10.208.17.2"
vsphere_user = "administrator@vsphere.local"
vsphere_password = "s3cretPassw0rd"
vsphere_cluster = "cluster1"
vsphere_datacenter = "datacenter1"
vsphere_datastore = "vsanDatastore"
vm_template = "rhcos-latest"
vm_network = "vxw-dvs-22-virtualwire-24-sid-6011-OpenShift-LS"

bootstrap_ignition_url = "http://192.168.133.51/bootstrap.ign"

control_plane_ignition = <<END_OF_MASTER_IGNITION
COPY IN CONTENTS OF MASTER.IGN HERE
END_OF_MASTER_IGNITION

compute_ignition = <<END_OF_WORKER_IGNITION
COPY IN CONTENTS OF WORKER.IGN HERE
END_OF_WORKER_IGNITION

module "dns" {
  source = "./route53"

  base_domain         = "${var.base_domain}"
  cluster_domain      = "${var.cluster_domain}"
  bootstrap_count     = "${var.bootstrap_complete ? 0 : 1}"
  bootstrap_ips       = ["${module.bootstrap.ip_addresses}"]
  control_plane_count = "${var.control_plane_count}"
  control_plane_ips   = ["${module.control_plane.ip_addresses}"]
  compute_count       = "${var.compute_count}"
  compute_ips         = ["${module.compute.ip_addresses}"]
}

Run Terraform

Terraform is used to deploy the VMs for bootstrap, control-plane nodes, and compute nodes. Terraform operates in the following way:

• terraform init - initializes the Terraform plug-ins and modules.
• terraform plan - completes a dry run on the templates, ensuring it can connect to the vCenter.
• terraform apply - runs the templates. The auto-approve switch can be added so that confirmation of each deployment is not required.

After Terraform provisions the VMs, the Red Hat OpenShift cluster bootstraps itself.

1. The bootstrap node starts and begins hosting the remote resources that are required for the control-plane to start.
2. The control-plane nodes fetch the remote resources from the bootstrap node and finish starting.
3. The control-plane nodes use the bootstrap node to form an etcd cluster.
4. The bootstrap node starts a temporary Kubernetes control plane by using the new etcd cluster.
5. The temporary control plane schedules the production control plane to the control-plane nodes.
6. The temporary control plane shuts down and passes control to the production control plane.
7. The bootstrap node injects Red Hat OpenShift Container Platform components into the production control plane.
8. The control plane sets up the compute nodes.
9. The control plane installs more services in the form of a set of Operators.
10. The result of this bootstrapping process is a fully running Red Hat OpenShift cluster. The cluster then downloads and configures remaining components that are needed for the day-to-day operation.

In the SSH session to the bastion node, with root privileges, run the following commands, ensuring that each one completes with no errors before you enter the next command.

cd /opt/ocpinstall/installer/upi/vsphere/
terraform init
terraform plan
terraform apply -auto-approve

Post deployment

1. After the virtual machines start, run the following command to monitor the installation:

   cd /opt/ocpinstall
   openshift-install --dir=. wait-for bootstrap-complete --log-level debug
   

2. Wait until you see the following information:

   INFO It is now safe to remove the bootstrap resources
   

3. The Cluster Image Registry does not select a storage backend in UPI mode. Therefore, the cluster operator continually waits for an administrator to configure a storage backend. A workaround is to point the image-registry to an empty directory to allow the installation to complete by running the following commands:

   mkdir /root/.kube/
   cp /opt/ocpinstall/auth/kubeconfig ~/.kube/config
   oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":{"storage":{"emptyDir":{}}}}'
   

   After the installation is complete, change the image-registry to a suitable location.

4. Run the following command to monitor the installation:

   openshift-install --dir=. wait-for install-complete
   

5. Wait until you see the following message. The system provides the URL and credentials to log in to the Red Hat OpenShift Console after the installation is complete. Your URL and password is different:

   INFO Install complete!
   INFO To access the cluster as the system:admin user when using 'oc', run 'export KUBECONFIG=/root/go/src/github.com/openshift/installer/bin/auth/kubeconfig'
   INFO Access the OpenShift web-console here: https://console-openshift-console.opc.dallas.ibm.local
   INFO Login to the console with user: kubeadmin, password: my-kube-password
   

   The password for the user that was created during installation can also be found in the auth subdirectory in the install-dir. You can log in through oc login and have access to the web console. The URL for the console is https://console-openshift-console.<cluster>.<base_domain>.

6. Run the following command from the /opt/ocpinstall directory:

   watch -n5 oc get clusteroperators
   

7. Monitor for cluster completion. An output is provided in the following example. Replace 4.x.5 with the current Red Hat OpenShift version, for example, 4.7.5.

   Every 5.0s: oc get clusteroperators
   
   NAME                                 VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
   authentication                       4.x.5    True        False         False      20m
   cloud-credential                     4.x.5    True        False         False      38m
   cluster-autoscaler                   4.x.5    True        False         False      38m
   console                              4.x.5    True        False         False      27m
   dns                                  4.x.5    True        False         False      35m
   image-registry                       4.x.5    True        False         False      14m
   ingress                              4.x.5    True        False         False      30m
   kube-apiserver                       4.x.5    True        False         False      33m
   kube-controller-manager              4.x.5    True        False         False      33m
   kube-scheduler                       4.x.5    True        False         False      32m
   machine-api                          4.x.5    True        False         False      38m
   machine-config                       4.x.5    True        False         False      33m
   marketplace                          4.x.5    True        False         False      30m
   monitoring                           4.x.5    True        False         False      28m
   network                              4.x.5    True        False         False      37m
   node-tuning                          4.x.5    True        False         False      32m
   openshift-apiserver                  4.x.5    True        False         False      31m
   openshift-controller-manager         4.x.5    True        False         False      33m
   openshift-samples                    4.x.5    True        False         False      24m
   operator-lifecycle-manager           4.x.5    True        False         False      35m
   operator-lifecycle-manager-catalog   4.x.5    True        False         False      35m
   service-ca                           4.x.5    True        False         False      38m
   service-catalog-apiserver            4.x.5    True        False         False      32m
   service-catalog-controller-manager   4.x.5    True        False         False      32m
   storage                              4.x.5    True        False         False      30m
   

Related links

• VMware Solutions and Red Hat OpenShift overview