Manage private environments with HCP Terraform agents

HCP Terraform allows you to manage isolated, private, or on-premises infrastructure using self-hosted HCP Terraform agents. The agent polls HCP Terraform or Terraform Enterprise for any changes to your configuration and executes the changes locally, so you do not need to allow public ingress traffic to your resources. Agents allow you to control infrastructure in private environments without modifying your network perimeter.

In this tutorial, you will create an HCP Terraform agent pool, launch local Dockerized agents on your machine, and configure an HCP Terraform workspace to use the agents to manage an Nginx container on your machine. You will also manage tokens for multiple HCP Terraform agents. By the end of the tutorial, you will understand how to set up a similar configuration in your own environment.

Prerequisites

For this tutorial, you will need:

• An HCP Terraform account
• A GitHub account
• Docker

Create an agent pool

Agent pools are groups of agents that can share tokens. When you associate a workspace with an agent pool, any agent in the pool can execute a run in that workspace.

To create an agent pool, navigate to the "Agents" panel within your HCP Terraform's "Settings" page and click "Create agent pool."

Enter education as the name, then click "Continue."

Next, HCP Terraform will prompt you to generate a token for the agent pool. Add agent1 as the description and click "Create token."

The next page displays your agent token and example commands for getting your agent running. This page only displays the token on creation — leave it open for now.

The page suggests a command to run the Docker agent that passes in environment variables with individual -e flags. In this tutorial, you will use the --env-file parameter instead, which makes it easier to manage multiple variables within a single file, so you will not have to re-enter them each time you need to relaunch the container. Using an environment variable file also prevents credentials from appearing in your process tables, granting an extra layer of security.

You can configure multiple tokens per agent pool, or have one shared token across all agents in the pool. By using unique tokens, you can revoke the token of one agent and stop its work without disrupting other agents in the pool, which you will do later in this tutorial.

Configure a workspace to use agents

Now configure a version-control-driven HCP Terraform workspace to use the agent pool.

In a new browser window, create a fork of the demo repository in your own GitHub account. This repository contains sample configuration to spin up an Nginx Docker container.

In that same browser window, create a new learn-terraform-cloud-agents workspace associated with your forked repository. For a detailed example of how to configure a VCS integration in HCP Terraform, revisit the Use VCS-Driven Workflow tutorial.

Next, navigate to your workspace's "General" settings.

Under "Execution Mode," select "Custom" and then select "Agent". Select "education" in the drop down under "Agent Pool."

Make sure to click "Save Settings" at the bottom of the page.

Launch the agent

Create a file called agent1.list file and paste in the contents below. Update the value of TFC_AGENT_TOKEN with the token you created in the previous step. Name the agent agent1 for this exercise. This name will show up in the agent management UI and on runs, allowing you to identify specific agents in the future. If you plan on using unique tokens for each agent, use the same name for the agent and token for clarity.

TFC_AGENT_TOKEN=<YOUR TOKEN>
TFC_AGENT_NAME=agent1

The only required environment variable is TFC_AGENT_TOKEN, but the agent loads all variables in your Docker environment. You can use this as an alternative to storing credentials and environment variables in your HCP Terraform Workspace's configuration.

Start your first containerized agent on your local machine. For this exercise, you are mounting the Docker socket using -v /var/run/docker.sock:/var/run/docker.sock. Mounting the socket allows the containerized agent to use the Docker provider to manage other containers on your machine.

$ docker run --name tfc_agent --env-file agent1.list -v /var/run/docker.sock:/var/run/docker.sock hashicorp/tfc-agent:latest
[INFO]  agent: Starting: name=agent1 version=0.4.1
[INFO]  core: Starting: version=0.4.1
[INFO]  core: Agent registered successfully with HCP Terraform: id=agent-9F5TXbYJoA7s7c18 pool-id=apool-wqx8cnrSrFn2AUfi
[INFO]  agent: Core version is up to date: version=0.4.1
[INFO]  core: Waiting for next job

Keep this terminal open.

Because this scenario uses the containerized Terraform to manage other Docker containers, the agent container needs to access the Docker socket. The HCP Terraform Agent runs as the non-root tfc-agent user within the container, so you need to explicitly modify the permissions for the Docker socket.

$ grep docker /etc/group
docker:x:281:root

$ docker exec -it -u 0 tfc_agent /bin/bash

$ groupadd -g <GROUP_ID> docker

$ usermod -aG docker tfc-agent

$ exit

$ docker exec -it -u 0 tfc_agent /bin/bash

$ chmod 666 /var/run/docker.sock

$ exit

Once the agent container launches, verify that it has registered with the pool in the HCP Terraform interface.

Agents are available as Docker containers and as standalone x86 binaries. For guidance on using the binaries, refer to the agent documentation.

Create a local Nginx container

In your browser, navigate back to your HCP Terraform learn-terraform-cloud-agents workspace and trigger a run by clicking "Queue plan."

The run "Execution Mode" shows that it is running in your local Docker agent. The plan details show the names of the agent pool and agent responsible for the run.

In your terminal, the Docker agent logs display the agent's Terraform actions.

[INFO]  core: Job received: type=plan id=run-NpfMDfGTsnY9ai6A
[INFO]  terraform: Handling run: id=run-NpfMDfGTsnY9ai6A type=plan org=hashicorp-training workspace=learn-terraform-cloud-agents
[INFO]  terraform: Extracting Terraform from release archive
[INFO]  terraform: Terraform CLI details: version=0.14.5
[INFO]  terraform: Downloading Terraform configuration
[INFO]  terraform: Running terraform init
[INFO]  terraform: Running terraform plan
[INFO]  terraform: Generating and uploading plan JSON
[INFO]  terraform: Generating and uploading provider schemas JSON
[INFO]  terraform: Persisting filesystem to remote storage
[INFO]  terraform: Finished handling run
[INFO]  core: Waiting for next job

Approve the proposed changes in HCP Terraform by clicking on "Confirm & Apply," and then confirming the plan.

The agent will now execute the work. The agent is self-contained and will not spin up additional containers for the Terraform execution itself — it downloads the Terraform version your configuration specifies and any providers necessary to complete the work task. In this tutorial, the agent launches an additional Docker container because that is the resource declared in the sample Terraform configuration.

[INFO]  terraform: Handling run: id=run-NpfMDfGTsnY9ai6A type=apply org=hashicorp-training workspace=learn-terraform-cloud-agents
[INFO]  core: Job received: type=apply id=run-NpfMDfGTsnY9ai6A
[INFO]  terraform: Extracting Terraform from release archive
[INFO]  terraform: Terraform CLI details: version=0.14.5
[INFO]  terraform: Recovering filesystem from remote storage
[INFO]  terraform: Running terraform init
[INFO]  terraform: Running terraform apply
[INFO]  terraform: Finished handling run
[INFO]  core: Waiting for next job

Once the apply is complete, open a new terminal window and confirm that your Nginx container is running by using docker ps or visiting localhost:8000.

$ docker ps
CONTAINER ID   IMAGE                        COMMAND                  CREATED              STATUS              PORTS                  NAMES
c9469d90663e   f6d0b4767a6c                 "/docker-entrypoint.…"   About a minute ago   Up About a minute   0.0.0.0:8000->80/tcp   nginx
515ed219aa66   hashicorp/tfc-agent:latest   "/bin/tfc-agent"         28 minutes ago       Up 28 minutes                              admiring_swartz

Though this tutorial targets your local machine, you can use the same configuration on any instance of your infrastructure. While running, the agent polls HCP Terraform for any new workloads it needs to complete. Since this is a pull-based pattern, you only need to allow TCP/443 egress traffic and DNS resolution. This is a lightweight way to use HCP Terraform within your isolated network segments without needing to configure your own installation of Terraform Enterprise.

Configure a second agent

Each agent is single-threaded and can only execute the work of one run at a time. For concurrent workloads, you must provision multiple agents. Agents within a pool can share tokens, or use unique ones so you can easily revoke the token of one agent without disrupting others.

Launch a second agent with a unique token.

1. Navigate back to your HCP Terraform organization settings.
2. Select the "Agents" panel.
3. Click on the "education" agent pool.
4. Click "+ New token."
5. Name the new agent agent2.

This will take you through the same steps as your first token creation. You will use this second token to launch another agent.

In a new terminal, create a file named agent2.list and open it. Copy and paste the contents below, and set the value of TFC_AGENT_TOKEN to be your new token.

TFC_AGENT_TOKEN=<YOUR TOKEN>
TFC_AGENT_NAME=agent2

Now launch your second agent.

$ docker run --env-file agent2.list -v /var/run/docker.sock:/var/run/docker.sock hashicorp/tfc-agent:latest

The "Agents" page in HCP Terraform will now display 2 agents ready to retrieve and process workloads.

Any other containers that have exited will appear here as well, but do not count toward your allotted number of agents. Refer to the HCP Terraform Agent documentation for a detailed list of possible agent statuses and how they count toward your agent totals.

You can configure additional agent pools as well, allowing you to maintain agents in each of your data centers and network segments.

Revoke a token

Next, revoke the token of one of your agents. On the education agent pool page, click "Revoke Token" for the agent2 token.

Once you revoke the token, the agent2 container will log its graceful shutdown and exit.

[ERROR] agent: Unrecoverable error, shutting down: error="GET https://app.terraform.io/api/agent/jobs: unexpected status code (401 Unauthorized): Agent token invalid"
[INFO]  agent: Shutting down
[INFO]  agent: Core plugin is shutting down
[ERROR] core: Failed updating status: error="PUT https://app.terraform.io/api/agent/status: unexpected status code (401 Unauthorized): Agent token invalid"
[INFO]  core: Shutdown complete
Graceful shutdown complete

Meanwhile, the agent1 container will continue running. This demonstrates the advantage of using unique tokens for your agents: you can revoke a token without disrupting any other agents. The state of agent2 will change in the "Agents" dashboard in HCP Terraform.

The exited agent is in an "Exited" state but will expire out of the list and your agent allocation count within 2 hours. If this agent comes back online before that timeout, it will return to an "Idle" state. Only agents in the "Idle," "Busy," or "Exited" states count against your purchased agent limit.

Destroy resources

Now that you have created and used HCP Terraform agents to create resources in an isolated environment, clean up all resources created in this tutorial.

Destroy the infrastructure

Prior to removing the agent pool, remove the resources created by queueing a destroy plan in your workspace. Follow the Destroy Infrastructure step of the clean up tutorial for step-by-step instructions if needed.

Your agent container will log the destruction plan as well.

[INFO]  core: Waiting for next job
[INFO]  core: Job received: type=apply id=run-dVh4azMHBwS49QR5
[INFO]  terraform: Handling run: id=run-dVh4azMHBwS49QR5 type=apply org=hashicorp-training workspace=learn-terraform-cloud-agents
[INFO]  terraform: Extracting Terraform from release archive
[INFO]  terraform: Terraform CLI details: version=0.14.6
[INFO]  terraform: Recovering filesystem from remote storage
[INFO]  terraform: Running terraform init
[INFO]  terraform: Running terraform apply
[INFO]  terraform: Finished handling run
[INFO]  core: Waiting for next job

Disassociate the agent pool

To delete an agent pool, you must first disassociate it from all workspaces. In your workspace settings, change the "Execution Mode" to "Remote." Be sure to click "Save Settings."

Delete the agent pool

On the "Agent Pool" page in your HCP Terraform organization settings, scroll to the bottom for the "Delete Agent Pool" section. Click "Delete agent pool" on the "Agents" page and confirm the prompt "Yes, delete agent pool."

Next steps

HCP Terraform's self-hosted agents allow you to manage more of your resources using HCP Terraform or Terraform Enterprise without modifying your network perimeter. Since run execution modes are workspace-specific, you can continue using remote execution for any publicly accessible-resources and use the self-hosted agent for resources that require extra security.

• Learn more in the HCP Terraform Agents documentation.

• Review the configuration details and differences for using HCP Terraform Agents with Terraform Enterprise.

• Explore the HCP Terraform/Enterprise provider.