Migrate Alternative Worker Images to Agents: v202302-1

Summary

Terraform Enterprise v202302-1 introduced the agent run pipeline mode which changes the container image used to perform Terraform runs. The previous run pipeline mode is now referred to as the legacy run pipeline mode.

If you currently use the default image, you do not need to take action. Terraform Enterprise will automatically migrate you to the new default agent image when you adopt v202302-1 or later.

If you currently use a legacy custom image Terraform Enterprise will continue using your legacy custom image until you migrate to an agent custom image and enable the agent run pipeline mode. Before adopting v202306-1, you must build an agent custom image. If your legacy custom image also executes custom scripts, you must update migrate your customization agent hooks instead.

Terraform Enterprise will stop supporting legacy custom images in v202306-1.

Continue reading to determine if this change impacts you and what action to take.

What are alternative worker images?

An alternative worker image is the custom image Terraform Enterprise uses to perform Terraform runs.

What is changing and why?

We are replacing three components of the run pipeline, terraform-build-worker, terraform-build-manager, and tfe-rabbitmq with local Terraform Cloud/Enterprise agents. If you use an alternative worker image to execute custom logic within the run lifecycle, you will need to migrate that logic to an agent custom image.

Like Terraform runs that use alternative worker images, runs that use custom agents will still execute in isolated, short-lived Docker containers. This change does not affect the Terraform Runs API and UI.

This change is part of an ongoing effort to refresh the architecture of Terraform Enterprise, improve performance and reliability of runs, and support new application-level features. Some immediate benefits of this change include:

The run pipeline contains fewer components, making it easier to understand and debug.
Local runs and remote agent runs can now use the same base images, and the same pre- and post- run hooks.
Runs are more evenly distributed across Active/Active nodes by eliminating a redundant queue.
You can now access previously inaccessible run logs (Sentinel, cost estimation, and plan-export-worker logs) via Docker logs and any tool that can interact with it. This includes the docker logs command, the Docker API, and TFE log forwarding.

In v202302-1, Terraform Enterprise installations using the default worker image (not an alternative worker image) will automatically use the new run pipeline.

Installations that use an alternative worker image will continue to use the legacy run pipeline when you upgrade to v202302-1. You must migrate your Terraform Enterprise installation to use an agent custom image before adopting v202306-1, which requires rebuilding your alternative worker image as an agent custom image.

Once you rebuild your image, you must manually switch to the new run pipeline by setting the run_pipeline_mode = "agent". You can revert back to the legacy run pipeline at any time before you adopt v202306-1 by setting run_pipeline_mode = "legacy". The legacy option will become unavailable in v202306-1 and later.

Check if you use an alternative worker image

By default, Terraform Enterprise uses the standard image included in a release. You can verify whether your pipeline uses an alternative worker image by reviewing your settings.json file or your admin console.

In the admin console's Settings tab, check if Provide the location of a custom image is selected. If it is, your pipeline uses the alternative worker image specified in the Custom image tag field.

The `custom_image_tag` setting in the user interface.

You can also review your settings.json file to check if your pipeline uses alternative worker image. If your file includes the custom_image_tag field, your pipeline uses the worker image specified.

Migrate to a custom agent image

To migrate to a custom agent image, first review the customizations in your worker image. Then build your new agent image, add your customizations, and test the new, customized agent image.

Step 1: Evaluate customizations in your current image

Before making any changes, you must evaluate your current image and familiarize yourself with all of the implemented customizations.

Consider the following:

Which custom tools does your image include?
Does the image use the initialize and/or finalize hook(s)?
If yes, what do your initialize and/or finalize script(s) do?

Step 2: Build your new custom agent image

Starting with v202302-1, the Docker container that Terraform Cloud and Enterprise uses for Terraform operations is based on the hashicorp/tfc-agent Docker image. Like the legacy worker image, the agent image supports customizations to perform custom logic, but there are several implementation differences between worker images and agent images to consider.

The following tables describe the differences between the legacy alternative worker image and the new custom agent image.

Note: You must build custom images from hashicorp/tfc-agent:1.6 or later.

Use an image

	Alternative worker image	Custom agent image
Building a custom image	Documentation	Documentation
Executing custom scripts	Documentation	Documentation
`settings.json` parameter	Set the `custom_image_tag` field to your custom worker image	Set the `custom_agent_image_tag` field to your custom agent image.
Image location	The image must be located on the host.	If Terraform Enterprise cannot find the image on the host, it will try to pull it from the specified location.

Customize an image

	Alternative Worker Image	Custom Agent Image
Base image	The base image must be `ubuntu:bionic` or an official RHEL7 image (e.g., `registry.access.redhat.com/ubi7/ubi-minimal`).	The base image must be the `hashicorp/tfc-agent`.
Image location	The image must exist on the Terraform Enterprise host. You can add it by running `docker pull` from a local registry or any other similar method.	We recommend that the image exist locally, but if it does not, the `tfe-task-worker` Docker driver will pull the image from the specified source.
Required software	The image must contain the software packages defined here.	N/A
Certificates	The `/usr/local/share/ca-certificates` directory must contain all necessary PEM-encoded CA certificates. Each file added to this directory must end with the `.crt` extension. Terraform Enterprise will not automatically add the CA certificates configured in the CA Bundle settings to the image at runtime.	The `/usr/local/share/ca-certificates` directory must contain all necessary PEM-encoded CA certificates. Each file added to this directory must end with the `.crt` extension. Terraform Enterprise will not automatically add the CA certificates configured in the CA Bundle settings to the image at runtime. This will be similar to the certificate config in this Dockerfile example.
Terraform binary	The image cannot include Terraform. Terraform Enterprise installs Terraform at runtime.	The image cannot include Terraform. Terraform Enterprise installs Terraform at runtime.
Rhel 7	In addition to the packages that `yum` installs, `curl` installs the `envdir` tool Python port because it is a dependency of the Terraform Build Worker service. If the alternative worker image does not include `envdir`, Terraform runs will fail within Terraform Enterprise.	N/A
Run before plans and applies	The same initialize script runs before a `terraform init` during both plans and applies.	You can have separate hooks that run before and after plans and applies: `terraform-pre-plan` `terraform-pre-apply`.
Run after plans and applies	The same finalize script runs after `terraform plan` and `terraform apply` complete.	You can have separate hooks that run before and after plans and applies: `terraform-post-plan` `terraform-post-apply`
Configuration	Ensure your worker image contains an executable shell script at: `/usr/local/bin/init_custom_worker.sh` `/usr/local/bin/finalize_custom_worker.sh`	Store hooks in a hooks directory, and name them according to their purpose. For example: `~/.tfc-agent/hooks/terraform-pre-plan` `~/.tfc-agent/hooks/terraform-post-plan` `~/.tfc-agent/hooks/terraform-pre-apply` `~/.tfc-agent/hooks/terraform-post-apply`
Non-zero exit codes	If the script exits with a non-zero exit code, the Terraform Enterprise run will immediately fail with an error.	If a `terraform apply` does not complete successfully, the hook will still run. If a hook exits with a non-zero exit code, the Terraform run will fail immediately.
`stdout`/`stderr`	The build worker logs whether or not a custom script has executed but does not log the script's standard output and standard error. If the custom script exits non-zero, then its standard output and standard error print to the UI logs.	The standard output and standard error from the hook will print alongside the Terraform run output in the Terraform Cloud/Enterprise user interface, but not in the UI logs.
File locations	You cannot customize the name or location of the script.	The hook name must match one of the supported hooks. You cannot customize or change these names. Because of this, you can only configure one hook of each type for each agent. For example, you could create a pre-plan and pre-apply hook, but you cannot create two pre-plan hooks.
Script permissions	The script must have execute permissions.	Each hook must have the execute permission set.
Execution timeout	The execution of the script does not have a timeout. It is up to the Terraform Enterprise administrator to ensure scripts execute in a timely fashion.	Each hook has a 60 second timeout. If a hook times out the Terraform run will fail immediately.
Execution context	The execution of the script is not sandboxed. The script executes in the same container where `terraform` runs, and both can access the same set of environment variables.	Like with alternative worker images, the execution of the script is not sandboxed. The script executes in the same container where `terraform` runs, and both can access the same set of environment variables.
Reading and writing variables	Users can "export" environment variables for subsequent Terraform commands (plan, apply, etc) to use, by writing their environment variable value to a file `/env/FOO` where `FOO` is the name of the environment variable.	Writing to environment variables requires a minimum agent version of 1.9.0. Instead of directly calling “export”, write to environment variables using “echo”. Terraform parses and sets the environment variable later when other Terraform commands execute. For example, write `echo FOO=bar >> $TFC_AGENT_ENV`, instead of `export FOO=bar`.
Running in Docker	Example Dockerfile snippet: `ADD init_custom_worker.sh /usr/local/bin/init_custom_worker.sh`	When running `tfc-agent` using Docker, you must build a new Docker image containing the hooks. To configure hooks, follow these instructions.
Execute/running as a binary	Scripts always run in the worker container. To execute, initialize and finalize scripts and all the commands they invoke, ensure your worker image contains an executable shell script at `/usr/local/bin/<init or finalize>_custom_worker.sh`.	If you want to run agents outside of containers, you must create your own remote agent pool. To run hooks as a binary, you need to change the permissions to the hook scripts.

Step 3: Test your custom agent image

We strongly recommend that you test your custom agent image before applying it to your production installation of Terraform Enterprise.

Test on Terraform Enterprise v202301 or earlier

Terraform Enterprise v202301 and earlier use worker images, but you can still test your agent image by configuring a test workspace to use a remote agent. You will need to install your agent on the host of your choice, and configure your workspace to use that agent. Follow the process described here for more details.

Test on Terraform Enterprise v202302 or later

To test in v202302 or later, install a test installation of Terraform Enterprise, define a custom_agent_image_tag, and set run_pipeline_mode = "agent" in your test installation’s settings.json.

Step 4: Adopt the new image

After building and testing the new custom agent image, adopt the new image in your production Terraform Enterprise installation (v202302 or later) by defining a custom_agent_image_tag, and setting run_pipeline_mode = "agent" in your settings.json file.

Frequently Asked Questions

Q. I currently use an alternative worker image, but I no longer need the customizations. What should I do?

A. If you want to use the default agent image going forward instead of a customized image, please remove your alternative worker image from use before you upgrade to v202302-1. If you want to remove the alternative worker image after upgrading to v202302-1 or later, set run_pipeline_mode = "agent" in your settings.json file without specifying a custom agent image.

Q. I already use a custom image for my remote agents. Can I re-use that image for my local agent image?

A. Yes. If your remote agent image accomplishes everything you want, you can specify that image as your custom agent image.

Q. I use an alternative worker image, when do I need to complete my migration to a custom agent image?

A. Terraform Enterprise will stop supporting workers in v202306-1. To maintain your customizations, you must complete your migration to a custom agent image before upgrading to v202306-1 or later.

Q. Can I switch back to the legacy run pipeline?

A. Yes, until v202306-1 you can switch between workers and agents. To switch back to the worker run pipeline, set run_pipeline_mode = "legacy" in settings.json. We will remove legacy mode in v202306-1.

Q. I use an alternative worker image to dynamically manage run credentials. What is the best way to accomplish this with a custom agent image?

A. You can dynamically manage credentials using workspace variables and/or agent hooks. Upcoming enhancements to Terraform Enterprise will make this easier. Please reach out to your HashiCorp account representative for more information.

Q. I have a custom workflow on my alternative worker image that does not seem possible on a custom agent image. What should I do?

A. Please reach out to your HashiCorp account representative for further guidance.