Migrating to Flexible Deployment Options

Flexible Deployment Options supports migrations between like versions of the application. We do not support combining an application upgrade and migrating from Replicated to Flexible Deployment Options. Upgrade your current installation of TFE to the latest version before proceeding with your Flexible Deployment Options migration.

We support the following migration paths from Replicated releases of Terraform Enterprise to Flexible Deployment Options.

Overview

At a high level, migrating from a Replicated release of Terraform Enterprise to Flexible Deployment Options (FDO); either Docker or managed Kubernetes, follows the same pattern:

Back up your current Terraform Enterprise data tier (such as Postgres and Object Storage).
Upgrade your current Terraform Enterprise (Replicated) installation to a FDO-compatible Terraform Enterprise version - v202309-1 or later. Migrations from Replicated to Flexible Deployment Options are only supported between like versions.
Prepare your new installation of Terraform Enterprise (Docker or cloud-managed Kubernetes).
Migrate your data tier to your new installation.

Prerequisites

Back up data tier

We always recommend backing up your data tier before conducting maintenance or upgrade operations. The backup methodology will depend on your existing installation method.
1. For mounted disk installations, read the recommended pattern for backing up your mounted disk.
2. For external services or active active installations, read the recommended pattern for backing up your object store, as well as the recommended pattern for database back up.

In both cases, we recommend backing up the existing replicated configuration. These configuration values will serve as reference for configuration of the Flexible Deployment Options.

$ replicatedctl app-config export > replicated-app-config.backup.json

Upgrade existing TFE installation to a Flexible Deployment Options compatible version

Upgrade the existing Terraform Enterprise (Replicated) installation to a version compatible for migration to Flexible Deployment Options. The minimum version is v202309-1.
Validate that Terraform Enterprise (Replicated) upgraded successfully by doing the following:
1. Validate the release version.
```
$ replicatedctl app inspect
```
2. Check that Replicated has started.
```
$ replicatedctl app status
```
3. Check that the docker containers are up.
```
$ docker ps
```
4. Do a health check.
```
$ tfe-admin health-check
```
5. Do a terraform plan and a terraform apply.

Mounted disk to Docker

Before proceeding with this migration guide, make sure you meet all the prerequisites, and that a Flexible Deployment Options license file has been provided by your Hashicorp business partner. We do not encourage moving forward with this guide if any of the prerequisites is not fulfilled.

If at any point you need to revert your settings, see the rollback steps.

Migration steps

Step 1: Backup your data tier

We always recommend backing up your data tier before conducting any maintenance or migration. See the backup data guide under the prerequisites section for a detailed guide on how to move forward with the backup process.

Step 2: Upgrade TFE to Flexible Deployment Options compatible section

The existing Terraform Enterprise installation (Replicated) should run a Flexible Deployment Options compatible version. See the upgrade guide under the prerequisites section for a step-by-step guide on how to upgrade the Terraform Enterprise for the migration.

Step 3: Verify Docker Engine verson

Docker should already be installed for you because Replicated is installed on this host. You may still need to install Docker Compose if you have not already. See the Docker Engine Requirements for more information.

If Docker Engine cannot be upgraded or installed in a supported docker configuration then Flexible Deployment Options should be installed and migrated to on a different host.

Step 4: Prepare the host and install your new Terraform Enterprise

To make migration smoother and faster, we recommend using the same host as your current Replicated instance.

Note: If you want to use a separate host for your new docker-based Terraform Enterprise, we can provide alternative steps. Reach out to support for assistance.

Create a directory at /etc/terraform-enterprise on the host.
Create a /etc/terraform-enterprise/docker-compose.yml file. Start with the contents in our mounted disk example and adjust it for your host and environment. Pay close attention to how we use bind mounts in the volumes section. Many of the required values may be found in the replicated application configuration backup taken earlier. See the Configuration Reference for more information on all of the available configuration settings for Flexible Deployment Options.

$ cat <PATH_TO_HASHICORP_LICENSE_FILE> |  docker login --username terraform images.releases.hashicorp.com --password-stdin

Pull the Terraform Enterprise image from the registry.

$ docker pull images.releases.hashicorp.com/hashicorp/terraform-enterprise:<vYYYYMM-#>

Create a new systemd service for Terraform Enterprise by creating a /etc/systemd/system/terraform-enterprise.service file with the contents on the Docker Installation guide. Update this systemd unit file if any of the following are true:

The name or the path of the docker-compose.yml has changed.
The path to the docker binary and the command to invoke docker compose is different because Docker Engine 1.13.1 is installed.

Step 5: Stop Replicated and migrate

Next, you can stop Replicated and migrate your Terraform Enterprise installation to Docker.

SSH into your Terraform Enterprise (Replicated) instance.
Stop Terraform Enterprise (Replicated). Ensure the application has fully stopped before proceeding.
```
$ replicatedctl app stop
```
Back up your data. If you have already backed up your data proceed forward. If not, or if you want another backup for safe keeping, do the following.
- Retrieve your mounted disk path.
```
$ replicatedctl app-config export --template '{{ .disk_path.Value }}' | tr -d '\r'
```
- We will refer to this path as the ${DISK_PATH} going forward. Next, archive your mounted disk data.
```
$ tar -zcvf data.tar.gz -C ${DISK_PATH} aux postgres
```
- Copy your data.tar.gz archive to a safe place.
Start (and enable on start up) the Docker Compose based Terraform Enterprise.
```
$ systemctl enable --now terraform-enterprise
```
Check the status of your service with systemctl status terraform-enterprise, or use docker ps to find your container's name then run docker logs [name]. You can also run curl https://[hostname]/_health_check to check the health check endpoint. Terraform Enterprise should now be running using Docker Compose, so your Replicated services can be shut down and disabled.

Shut down Replicated.

$ systemctl disable --now replicated replicated-ui replicated-operator

Next, clean up the extra unneccessary Replicated containers.

$ docker stop replicated-premkit

$ docker stop replicated-statsd

$ docker rm -f replicated replicated-ui replicated-operator replicated-premkit replicated-statsd retraced-api retraced-processor retraced-cron retraced-nsqd retraced-postgres

Note: some of the docker stop commands may return “Container not found” errors because not every Replicated install has every container.

Step 6: Validate migration success

Finally, test that your new Terraform Enterprise installation works properly. If you have an existing suite of release acceptance tests, you can use those instead of doing the following steps.

Execute a plan and apply it from the CLI, testing several subsystems. Ensuring that proxies are correctly configured, certificates are properly configured, and the instance can download Terraform binaries and execute runs.
Execute a plan and apply it from VCS, testing that webhooks are working and certificates are in place on both sides.
Publish a new module to the private module registry.
Execute a plan and apply it with a module or provider from the private registry to ensure the registry is functioning.
(Optional) Execute a plan and apply it with Sentinel and cost estimation, ensuring run tasks and cost estimation work.
(Optional) Execute a plan and apply it on a workspace that uses an agent pool, testing that external agents can connect and run jobs successfully.

Mounted Disk rollback steps

In the unlikely event you encounter issues and need to roll back, you can revert back to Terraform Enterprise (Replicated) using the following commands.

Stop and disable Terraform Enterprise (Docker).

$ systemctl disable --now terraform-enterprise

Start and enable Replicated.

$ systemctl enable --now replicated replicated-ui replicated-operator

Start Terraform Enterprise (Replicated)
```
$ replicatedctl app start
```

External services to Docker

When Terraform Enterprise is operating in active-active mode, you can scale directly up to your target number of nodes after the migration is complete. You do not need to scale to one node before scaling to all nodes.

If at any point you need to revert your settings, see the rollback steps.

Migration steps

Step 1: Backup your data tier

Step 2: Upgrade TFE to Flexible Deployment Options compatible section

Step 3: Verify Docker Engine verson

If Docker Engine cannot be upgraded or installed in a supported docker configuration then Flexible Deployment Options should be installed and migrated to on a different host.

Step 4: Prepare the host and install your new Terraform Enterprise

To make migration smoother and faster, we recommend using the same host as your current Replicated instance.

Note: If you want to use a separate host for your new docker-based Terraform Enterprise, we can provide alternative steps. Reach out to support for assistance.

Create a directory called /etc/terraform-enterprise on the host for your docker-compose.yml. Refer to our example for guidance.
Place a docker-compose.yml file in your new directory and adjust it for your host and environment. Ensure that you update and verify the values below.
- Hostname
- Encryption Password
- Certificates and their locations
- Custom CA Bundle
- Mounted disk data path
- Proxy and no_proxy settings

Use the external services credentials from your Replicated installation for the external services installation. Specifically the following in your Compose file:

The TFE_OBJECT_STORAGE_TYPE and TFE_OBJECT_STORAGE_* variables should specify the object storage type and the container or bucket credentials from your Replicated installation.
The TFE_DATABASE_* variables should specify database credentials from the Replicated installation.
If you use an external Redis for your Replicated installation, you can use the same credentials for the external services installation.
If you use an external Vault on your Replicated installation, replace the TFE_VAULT_* values in the Compose file with the Replicated values.
Your TFE_ENCRYPTION_PASSWORD value should match your Replicated installation value. You can get this from your Replicated instance via SSH by running the following command:
```
$ replicatedctl app-config export --template '{{ .enc_password.Value }}'
```
To pull a terraform-enterprise Flexible Deployment Options image, first log-in to the registry, and then use docker to pull the image version.
```
$ docker login --username terraform images.releases.hashicorp.com
```
Note: When prompted for password, use the contents of your Hashicorp license file as your password.
```
$ docker pull images.releases.hashicorp.com/hashicorp/terraform-enterprise:<vYYYYMM-#>
```
Create a new systemd service for Terraform Enterprise by creating a file named, /etc/systemd/system/terraform-enterprise.service with these contents. If you changed the name of the directory holding your docker-compose.yml, update the systemd service file path to match.

Note: If you want to use a separate host for your new docker-based Terraform Enterprise, we can provide alternative steps. Reach out to support for assistance.

Step 5: Stop Replicated and migrate

Next, you can stop Replicated and migrate your Terraform Enterprise installation to Docker.

SSH into your Terraform Enterprise (Replicated) instance.
Stop Terraform Enterprise (Replicated). Ensure the application has fully stopped before proceeding.
```
$ replicatedctl app stop
```
Start (and enable on start up) the Docker Compose based Terraform Enterprise.
```
$ systemctl enable --now terraform-enterprise
```
Check the status of your service with systemctl status terraform-enterprise, or use docker ps to find your container's name then run docker logs [name]. You can also run curl https://[hostname]/_health_check to check the health check endpoint. Terraform Enterprise should now be running using Docker Compose, so your Replicated services can be shut down and disabled.

Shut down Replicated.

$ systemctl disable --now replicated replicated-ui replicated-operator

Next, clean up the extra unneccessary Replicated containers.

$ docker stop replicated-premkit

$ docker stop replicated-statsd

$ docker rm -f replicated replicated-ui replicated-operator replicated-premkit replicated-statsd retraced-api retraced-processor retraced-cron retraced-nsqd retraced-postgres

Note: Some of the docker stop commands may return “Container not found” errors because not every Replicated install has every container.

Step 6: Validate migration success

Finally, test that your new Terraform Enterprise installation works properly. If you have an existing suite of release acceptance tests, you can use those instead of doing the following steps.

Execute a plan and apply it from the CLI, testing several subsystems. Ensuring that proxies are correctly configured, certificates are properly configured, and the instance can download Terraform binaries and execute runs.
Execute a plan and apply it from VCS, testing that webhooks are working and certificates are in place on both sides.
Publish a new module to the private module registry.
Execute a plan and apply it with a module or provider from the private registry to ensure the registry is functioning.
(Optional) Execute a plan and apply it with Sentinel and cost estimation, ensuring run tasks and cost estimation work.
(Optional) Execute a plan and apply it on a workspace that uses an agent pool, testing that external agents can connect and run jobs successfully.

External Services rollback steps

In the unlikely event you encounter issues and need to roll back, you can revert back to Terraform Enterprise (Replicated) using the following commands.

Stop and disable Terraform Enterprise (Docker).

$ systemctl disable --now terraform-enterprise

Start and enable Replicated.

$ systemctl enable --now replicated replicated-ui replicated-operator

Start Terraform Enterprise (Replicated)
```
$ replicatedctl app start
```

Mounted disk to Cloud-managed Kubernetes

In order to meet the Terraform Enterprise Kubernetes Installation Requirements, you must provide an external Postgres server, external object storage, and external Redis storage.

If you currently use the mounted disk operational mode for Terraform Enterprise on Replicated, you do not meet the above requirements. You must first migrate to external services mode, and then follow the external services to Kubernetes migration guide as well as deploy an external Redis server. At a high level, this process involves:

Backing up your data.
Restoring your data to external services.
Testing that the external services migration succeeded.
Deploying external redis.
Follow the guide for External Services to Kubernetes migration.

Contact your HashiCorp account representative or HashiCorp support if you have additional questions.

External services or Active/Active to Cloud-managed Kubernetes

Redis is a required service for running Terraform Enterprise in Kubernetes. If you currently use the external services operational mode, you need to deploy an external Redis server.

If using the Active/Active operational mode, you have all the required service dependencies for migrating to Kubernetes. Refer to Terraform Enterprise Kubernetes Installation Requirements for more details on the requirements for running Terraform Enterprise in Kubernetes.

Before proceeding with this migration guide, make sure all the prerequisites are met, and that a Flexible Deployment Options license file has been provided by your Hashicorp business partner. We do not encourage moving forward with this guide if any of the prerequisites is not fulfilled.

If at any point you need to revert your settings, see the rollback steps.

Migration steps

Step 1: Backup your data tier

Step 2: Upgrade TFE to Flexible Deployment Options compatible section

Step 3: Prepare the custom Helm Values file for Terraform Enterprise

On Terraform Enterprise (Replicated), view existing configuration:
```
$ replicatedctl app-config export
```
Create a custom Helm Values file e.g overrides.yaml to override the default values in the Terraform Enterprise Helm chart.
On the env.secrets and env.variables sections of the overrides values file, use the external services credentials from the Replicated installation for the Kubernetes installation. Specifically the following:
1. The TFE_OBJECT_STORAGE_TYPE and TFE_OBJECT_STORAGE_* variables should specify the object storage type and the container or bucket credentials from your Replicated installation.
2. The TFE_DATABASE_* variables should specify database credentials from the Replicated installation.
3. The TFE_REDIS_* values on the Helm chart should specify the same credentials from the external Redis in your Replicated installation.
4. If there is an external vault, the TFE_VAULT_* values on the Helm chart should specify the same credentials from the external Vault in your Replicated installation.
5. The TFE_ENCRYPTION_PASSWORD value should match the Replicated installation value. You can get this from your Replicated instance via SSH by running the following command:
```
$ replicatedctl app-config export --template '{{ .enc_password.Value }}'
```
  See the Replicated to Flexible Deployment Options configurations mapping for more information on how the Replicated configuration maps to the variables and secrets on Terraform Enterprise Helm chart.

Step 4: Migrate to Kubernetes

Stop your Replicated installation by executing the following command:
```
$ replicatedctl app stop
```
Wait for the application to stop:
```
$ replicatedctl app status
```
Install Terraform Enterprise on Kubernetes using Helm.

Step 5: Validate migration success

Finally, test that your new Terraform Enterprise installation works properly. If you have an existing suite of release acceptance tests, you can use those instead of doing the following steps. You should be able to log in to your new Terraform Enterprise installation with the credentials previously used for your Replicated installation.

Execute a plan and apply it from the CLI, testing several subsystems. Ensuring that proxies are correctly configured, certificates are properly configured, and the instance can download Terraform binaries and execute runs.
Execute a plan and apply it from VCS, testing that webhooks are working and certificates are in place on both sides.
Publish a new module to the private module registry.
Execute a plan and apply it with a module or provider from the private registry to ensure the registry is functioning.
(Optional) Execute a plan and apply it with Sentinel and cost estimation, ensuring run tasks and cost estimation work.
(Optional) Execute a plan and apply it on a workspace that uses an agent pool, testing that external agents can connect and run jobs successfully.

Kubernetes rollback steps

In the unlikely event you encounter issues that cannot be worked around, you can rollback to Terraform Enterprise (Replicated).

If it is possible to exec into the pods, run the node drain command to stop Terraform Enterprise from executing further instructions.
```
$ tfectl node drain --all
```
Uninstall the deployment.
```
$ helm uninstall terraform-enterprise
```
Restart Terraform Enterprise on Replicated using the same external services.
```
$ replicatedctl app start
```

Troubleshooting

Please read the Flexible Deployment Options monitoring documentation to ensure you have uninterrupted visibility into the health of the application:

Common Issues

Below are a list of common migration issues and symptoms of those issues.

Self signed certificates CA not in CA bundle

Symptoms:

Plans Fail
See errors in /var/log/terraform-enterprise/task-worker.log and /var/log/terraform-enterprise/atlas.log (particularly when making calls to Archivist) where the certificate is from an unknown issuer

Fix:

Bring the additional certificates from the full chain certificate into the CA Bundle
Note: Replicated did some of this automatically, for Terraform Enterprise Flexible Deployment Options you may need to manually concatenate the certificates from the full chain certificate into the CA Bundle for the instance to talk to itself.

Required CA not in CA bundle

Symptom:

Setting up VCS fails with unknown certificate issuer error

Fix:

Include the CA in the CA Bundle

Internal calls to instance or AWS Metadata Endpoint unnecessarily proxied

Symptom:

Plans May fail, Logs may fail to load (but not always)
Traffic that shouldn’t be proxied, is

Fix:

Under Replicated we would build much of the default no_proxy/NO_PROXY address list. At the moment this is completely managed by the customer. This means in addition to what was once in their “Additional No Proxy List” some extra entries need to be added: localhost,127.0.0.1,169.254.169.254,[FQDN of instance],[Rest of no_proxy list]