Upgrading to a new Terraform version

The open source IaC tools that are used by Schematics evolve, with new versions of Terraform and Helm, and the supporting Terraform providers. Over time it is necessary for long lived workspace environments to be upgraded to utilize the latest version of Terraform as older versions are deprecated and go out of support.

All Schematics users are encourage to regularly upgrade to the latest Terraform version to ensure continuity of operations and support. Schematics follows the Hashicorp support model for Terraform releases and deprecates versions to the in line with Hashicorp.

Terraform v1.0 was a major release for Terraform, marking the transition to a stable 1.x release. Hashicorp made compatibility promises for the 1.x releases, that for core features, no additional changes are required to upgrade through the 1.x releases.

In short, it aims to make upgrades between v1.x releases straightforward, requiring no changes to your configuration, no commands to run upgrade steps, and no changes to any automation you've set up around Terraform.

To upgrade to the 1.x releases requires no specific Schematics workspace actions. Review the Terraform upgrade guides for release specific changes that might require TF config/template updates.

Upgrading the Terraform template version 1.x and higher

Since Terraform 1.0, Schematics workspaces can be updated to more recent 1.x releases, through a simple change to the workspace version. To update from 0.x releases refer to section Upgrading the Terraform template version 0.x

Schematics supports Terraform_v1.x and plans to make releases available 45-60 days after general availability. It is recommended that Terraform templates use a version range constraint, such as, >, >=, or ~> for the required_version parameter in the versions.tf of Terraform template, that allows upgrade for minor and patch releases. This allows Schematics to automatically adopt the latest patch or minor release of Terraform version as set by the workspace version.

terraform {
required_version = "~> 1.1"
}

Updating the workspace Terraform 1.x version

The in use version of Terraform for a workspace can be updated through the Schematics workspace Update API.

The workspace terraform version parameter is of the form terraform_v1.4 or terraform_v1.5

Select the workspace to be updated and verify that it is in Normal state and that a Plan operation does not generate any proposed resource changes. Save the workspace_id and note the region the workspace is hosted in.
Update the workspace terraform version using the IBM Cloud CLI and API. These workspace operations are region specific. Note the workspace region from the UI as this is required for the following commands:
- Login to the IBM Cloud CLI with ibmcloud login
- Set the CLI target region with ibmcloud target -r <region> to be the same as the workspace you are updating.
- Generate an IAM oauth token to use with the Schematics API, with the command ibmcloud iam oauth-tokens.
- Copy the token data and insert in to the following command text, replacing the string <token-data>, set <terraform_version> to the required Terraform version and the <workspace_id>:
- The workspace is updated by executing a cURL command to call the Schematics Update API to update the Terraform version. This operation is region specific and must specify the desired Schematics API region endpoint for the workspace target region. Replace the text <schematics-region-endpoint> in the command with the endpoint for the target workspace region.
```
    curl -X PUT https://<schematics-region-endpoint>.cloud.ibm.com/v1/workspaces/<w_id> \
    -H 'Authorization: Bearer <token>' \
    -H 'refresh_token: <token>' \
    -d '{
    "type": [
        "<terraform_version>"
    ],
    "template_data": [
        {
            "folder": ".",
            "type": "<terraform_version>"
        }
    ]
    }'
```
Verify in the workspace settings page the TF version is now set to the desired version.
Run a Generate Plan operation against the workspace. Validate that the command runs successfully without error and no unexpected messages are logged. The Plan should result in no proposed changes to the resources.
Run a Apply Plan operation against the workspace. Validate that the command runs successfully without error and no unexpected messages are logged.
You have now successfully upgraded.

Upgrading the Terraform template version 0.x to 1.x

At 0.x releases, Terraform version upgrade is a stepwise process, upgrading through each release. Upgrading does not support upgrading across multiple releases and must be performed, release by release. Some updates requires execution of Terraform upgrade commands to modify the config files also changes to the Terraform state file. These steps cannot be performed within Schematics. Workspace Terraform templates must be upgraded using a local copy of Terraform. Follow the steps for upgrading the 0.x releases.

Terraform versions list
Version	Recommendation
`v0.12`	Review the v0.13 upgrade guide and follow the instructions Upgrading a Terraform v0.12 workspace to v0.13. Schematics has deprecated `Terraform v0.12`.
`v0.13`	For the `Terraform v0.14` upgrade, you must run `terraform apply` with `Terraform v0.14` to complete its state format upgrades. If you get any errors, see the v0.14 upgrade guide. Follow the instructions upgrade-13-to10
`v0.14`	You can upgrade directly to the `Terraform v1.0` version. Review the v0.15 upgrade guide.
`v0.15`	You can upgrade directly to the `Terraform v1.0` version. Review the v1.0 upgrade guide.

Upgrading a Terraform v0.12 workspace to v0.13

Upgrading a v0.12 workspace to use the v0.13 version of Terraform is a multi-step task. You must carefully review the Terraform upgrade guide for the related version upgrade.

Use the following steps to upgrade to the current Terraform version in the Schematics workspace.

Upgrade the Terraform configuration files to use the newer syntax and semantics.
Migrate the Terraform state file to be compatible with the newer version. Schematics does not support built in modification the Terraform state file. Therefore, you need to follow these steps.
1. Prepare the upgraded version of Terraform configuration files and Terraform state file, in your local machine.
2. Create the new Schematics workspace with the new Terraform configuration files and Terraform state file.
3. Delete the older workspace without destroying the resources.

The following are the detailed steps to upgrade from 0.12 to 0.13:

Check whether your Schematics workspaces at v0.12 has resources, the last apply was successful and the workspace is in normal state. Verify the Terraform configuration files and Terraform state file, are in a consistent state for Terraform v0.12.
Download or clone the Git repository that is used by your Terraform v0.12 Schematics workspace to your local machine.
Install Terraform 0.13 on your local machine.
Change directory to your cloned repository and upgrade your config files to Terraform v0.13 by running the Terraform v0.13upgrade command. For more information, see Upgrading to Terraform v0.13 documentation. The upgrade command generates a versions.tf file with a terraform configuration block.

Edit the versions.tf file to set the source parameter to source = "IBM-Cloud/ibm" as shown in the code block.

versions.tf file

terraform {
    required_providers {
    ibm = {
      # TF-UPGRADE-TODO
      #
      # No source detected for this provider. You must add a source address
      # in the following format:
      #
      source = "IBM-Cloud/ibm"
      #
      # For more information, see the provider source documentation:
      #
    }
    }
    required_version = ">= 0.13"
}

Download the Terraform state file from the existing Schematics workspace, by using Schematics state pull command.

When, the workspace is created with the tfstate Schematics considers it as a secure file. Also, you cannot pull the created tfstate file through UI. You need to use the command line to pull the state file and create workspace.

Copy the downloaded state file as terraform.tfstate into the Terraform execution folder.
Run the state replace provider command in on the command-line to update the IBM Cloud provider version in the state file.
```
terraform state replace-provider registry.terraform.io/-/ibm registry.terraform.io/ibm-cloud/ibm.
```
Verify that the updates are made to the terraform.tfstate file with Terraform version update from 1.3 to >= 1.4 and the provider that is updated as registry.terraform.io/ibm-cloud/ibm.
Push the upgraded TF config files and version.tf back to your Git repository.
Copy the content of modified terraform.tfstate file to state.json file.

Create or update a workspace.json file as shown in the code block.

{
    "name": "gb",
    "type": [
        "terraform_v1.4"
    ],
    "description": "migration workspace",
    "template_repo": {
        "url": "Provide your Git repository link"
    },
    "workspace_status" : {
        "frozen": false
    },
    "template_data": [{
        "folder": ".",
        "type": "terraform_v1.4"
    }]
}

Run these commands through command-line to create a new Terraform v0.13 workspace:
- ibmcloud schematics workspace new --file workspace.json --state state.json.
- ibmcloud schematics workspace get --id <workspace-id>. If your workspace status is not inactive, wait for few seconds and retry the command.
- ibmcloud schematics plan id <workspace id>.
- ibmcloud schematics job get --id <job-id form plan>. If your workspace plan status is not success, wait for few seconds and retry the command.
- ibmcloud schematics apply --id <workspace id>.
- ibmcloud schematics job get --id <job-id from apply>.
[Optional] you can delete the Schematics workspace that uses Terraform v0.12.

Do not destroy the resources that are used by your old workspace.

Upgrade Terraform template from `v0.13` and higher to `v1.0`

Versions 0.13 through 0.15 require a stepwise upgrade, 0.13 to 0.14, 0.14 to 0.15, 0.15 to 1.0.

The process is the same for each version step. It is mandatory that a Terraform Apply is run after each version change. This updates the Terraform state file with schema changes related to that version and that version only. After successfully upgrading a single version, the next version update can be performed.