Use hcdiag with Terraform

12min
|
Enterprise
Terraform

HashiCorp Diagnostics — hcdiag — is a troubleshooting data-gathering tool that you can use to collect and archive important data from Consul, Nomad, Vault, and TFE server environments. The information gathered by hcdiag is well-suited for sharing with teams during incident response and troubleshooting.

For Terraform Enterprise, the hcdiag tool executes a series of commands to collect system and production information by querying replicatedctl and the Terraform Enterprise API. We recommend installing hcdiag when you are experiencing an incident. By default, hcdiag will gather up to 72 hours of server logs. This ensures that you have the latest version of hcdiag.

In this tutorial, you will:

Connect to your Terraform Enterprise application
Install and configure hcdiag
Use hcdiag to gather troubleshooting information about your Terraform Enterprise application

In the process, you will learn the most important hcdiag commands and features, and get comfortable with the contents of the data bundle created by the tool.

Prerequisites

To perform the steps in this tutorial, you need:

A Terraform Enterprise application (Standalone or Active/Active deployment)
sudo permissions on the host (this will make it possible for you to install hcdiag, and for hcdiag to gather all relevant system data)

Create the TFE API token

The hcdiag tool requires a user API token to authenticate and query information from the Terraform Enterprise API. Go to the "Tokens" page of your Terraform Enterprise application and create an API token named hcdiag.

API token named hcdiag in Terraform Enterprise application

Save this token in a safe place, you will use it later in the tutorial to authenticate hcdiag to the Terraform Enterprise API.

Connect to your TFE application

Open a shell session to the server that your Terraform Enterprise application is running on to use hcdiag. Use SSH or your cloud provider's session manager to connect to the Terraform Enterprise application's instances.

Once you have connected to an instance of your Terraform Enterprise application, switch to the root user. Several commands in this tutorial are only available to the root user.

$ sudo su -

Configure the environment

Before you download and run hcdiag, you must first configure your environment so that the tool knows how to communicate with and authenticate to Terraform Enterprise.

First, set the TFE_HTTP_ADDR environment variable to your Terraform Enterprise URL. This tells hcdiag where to query the API.

$ export TFE_HTTP_ADDR=

Then, set the TFE_TOKEN environment variable to the API token you created earlier. This allows hcdiag to authenticate to the API and query it for information.

$ export TFE_TOKEN=

Install dependencies and hcdiag

Before you can download hcdiag, you need to install a few tools.

Note This tutorial shows package installation commands from the perspective of a TFE host running Ubuntu Linux. If you are running Red Hat Linux or another Linux distribution or operating system, replace what you see in this section with the standard package installation commands for your OS.

Update apt-get and install the necessary dependencies.

$ apt-get update && apt-get install -y wget gpg

Add the HashiCorp repository.

$ wget -O- https://apt.releases.hashicorp.com/gpg | gpg --dearmor > /usr/share/keyrings/hashicorp-archive-keyring.gpg && echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com jammy main" | tee /etc/apt/sources.list.d/hashicorp.list

Update the package cache and install the latest stable version of hcdiag.

$ apt-get update && apt-get install -y hcdiag

Note If you cannot use your operating system's package repositories to download hcdiag, you can also get binaries from the HashiCorp releases repository. For instructions on this and other installation methods, see the hcdiag installation documentation

Gather troubleshooting data

Now that it's installed, use hcdiag to gather all troubleshooting data for your Terraform Enterprise application. This may take several minutes.

$ sudo hcdiag -terraform-ent
2022-10-19T13:50:49.082Z [INFO]  hcdiag: Ensuring destination directory exists: directory=.
2022-10-19T13:50:49.083Z [INFO]  hcdiag: Checking product availability
2022-10-19T13:50:49.083Z [INFO]  hcdiag: Gathering diagnostics
2022-10-19T13:50:49.083Z [INFO]  hcdiag.product: Running operations for: product=host
2022-10-19T13:50:49.084Z [INFO]  hcdiag.product: running operation: product=host runner="uname -v"
2022-10-19T13:50:49.087Z [INFO]  hcdiag.product: Running operations for: product=terraform-ent
2022-10-19T13:50:49.087Z [INFO]  hcdiag.product: running operation: product=terraform-ent runner="replicatedctl support-bundle"
2022-10-19T13:50:49.090Z [INFO]  hcdiag.product: running operation: product=host runner=disks
2022-10-19T13:50:49.095Z [INFO]  hcdiag.product: running operation: product=host runner=info
2022-10-19T13:50:49.097Z [INFO]  hcdiag.product: running operation: product=host runner=memory
2022-10-19T13:50:49.097Z [INFO]  hcdiag.product: running operation: product=host runner=process
2022-10-19T13:50:49.124Z [INFO]  hcdiag.product: running operation: product=host runner=network
2022-10-19T13:50:49.214Z [INFO]  hcdiag.product: running operation: product=host runner=/etc/hosts
2022-10-19T13:50:49.218Z [INFO]  hcdiag.product: running operation: product=host runner=iptables
2022-10-19T13:50:49.231Z [INFO]  hcdiag.product: running operation: product=host runner="/proc/ files"
2022-10-19T13:50:49.278Z [INFO]  hcdiag.product: running operation: product=host runner=/etc/fstab
2022-10-19T13:50:49.301Z [INFO]  hcdiag: Product done: product=host statuses=map[success:10]
2022-10-19T13:55:24.615Z [INFO]  hcdiag.product: running operation: product=terraform-ent runner="copy /var/lib/replicated/support-bundles/replicated-support*.tar.gz"
2022-10-19T13:55:24.615Z [INFO]  hcdiag: copying: path=/var/lib/replicated/support-bundles/replicated-support795392873.tar.gz to=/home/tfeuser/hcdiag-2022-10-19T135049Z4003256385/var/lib/replicated/support-bundles/replicated-support795392873.tar.gz
2022-10-19T13:55:40.043Z [INFO]  hcdiag: copying: path=/var/lib/replicated/support-bundles/replicated-support977688192.tar.gz to=/home/tfeuser/hcdiag-2022-10-19T135049Z4003256385/var/lib/replicated/support-bundles/replicated-support977688192.tar.gz
2022-10-19T13:55:55.481Z [INFO]  hcdiag.product: running operation: product=terraform-ent runner="GET /api/v2/admin/customization-settings"
2022-10-19T13:55:55.500Z [INFO]  hcdiag.product: running operation: product=terraform-ent runner="GET /api/v2/admin/general-settings"
2022-10-19T13:55:55.503Z [INFO]  hcdiag.product: running operation: product=terraform-ent runner="GET /api/v2/admin/organizations"
2022-10-19T13:55:55.505Z [INFO]  hcdiag.product: running operation: product=terraform-ent runner="GET /api/v2/admin/terraform-versions"
2022-10-19T13:55:55.508Z [INFO]  hcdiag.product: running operation: product=terraform-ent runner="GET /api/v2/admin/twilio-settings"
2022-10-19T13:55:55.510Z [INFO]  hcdiag.product: running operation: product=terraform-ent runner="GET /api/v2/admin/workspaces?page[size]=1"
2022-10-19T13:55:55.512Z [INFO]  hcdiag.product: running operation: product=terraform-ent runner="docker -v"
2022-10-19T13:55:55.538Z [INFO]  hcdiag.product: running operation: product=terraform-ent runner="replicatedctl app status --output json"
2022-10-19T13:55:56.253Z [INFO]  hcdiag.product: running operation: product=terraform-ent runner="lsblk --json"
2022-10-19T13:55:56.263Z [INFO]  hcdiag.product: running operation: product=terraform-ent runner=getenforce
2022-10-19T13:55:56.265Z [WARN]  hcdiag.product: result: runner=getenforce status=unknown
  result=
  | /bin/bash: getenforce: command not found
   error="exec error, command=getenforce, error=exit status 127"
2022-10-19T13:55:56.265Z [INFO]  hcdiag.product: running operation: product=terraform-ent runner="env | grep -i proxy"
2022-10-19T13:55:56.271Z [WARN]  hcdiag.product: result: runner="env | grep -i proxy" status=unknown result="" error="exec error, command=env | grep -i proxy, error=exit status 1"
2022-10-19T13:55:56.271Z [INFO]  hcdiag: Product done: product=terraform-ent statuses="map[success:11 unknown:2]"
2022-10-19T13:55:56.271Z [INFO]  hcdiag: Recording manifest
2022-10-19T13:55:56.273Z [INFO]  hcdiag: Created Results.json file: dest=/home/tfeuser/hcdiag-2022-10-19T135049Z4003256385/Results.json
2022-10-19T13:55:56.273Z [INFO]  hcdiag: Created Manifest.json file: dest=/home/tfeuser/hcdiag-2022-10-19T135049Z4003256385/Manifest.json
2022-10-19T13:56:03.082Z [INFO]  hcdiag: Compressed and archived output file: dest=hcdiag-2022-10-19T135049Z.tar.gz
2022-10-19T13:56:03.123Z [INFO]  hcdiag: Writing summary of products and ops to standard output
product        success  fail  unknown  total
host           10       0     0        10
terraform-ent  11       0     2        13

If you invoke hcdiag with no flags, it will autodetect which products are installed on the host, and gather all available environment and product information.

For a full list of options supported by hcdiag, run hcdiag -h.

$ hcdiag -h

Explore the troubleshooting data

Notice that hcdiag generates an archive with troubleshooting data about the Terraform Enterprise application. First, list the .tar.gz archive files to discover the file that hcdiag created.

$ ls -lh *.tar.gz
-rw-r--r-- 1 root    root     64M Oct 19 15:18 hcdiag-2022-10-19T151612Z.tar.gz

The filename contains an hcdiag prefix, followed by a timestamp. Unpack the archive to further examine its contents.

$ tar zxvf hcdiag-*.tar.gz
hcdiag-2022-10-19T151612Z/Manifest.json
hcdiag-2022-10-19T151612Z/Results.json
hcdiag-2022-10-19T151612Z/var/lib/replicated/support-bundles/replicated-support727809798.tar.gz

The archive extracts three files into a temporary directory.

The Manifest.json file contains information describing the hcdiag run, including configuration options used, run duration, and a count of any errors encountered.
The Results.json file contains information about the environment and the output from the replicatedctl support-bundle command.
The /var/lib/replicated/support-bundles/replicated-*.tar.gz file contains the output from the replicatedctl support-bundle command

You can further explore the replicated-*.tar.gz archive by unpacking it and examining its contents.

$ tar zxvf hcdiag-2022-10-19T151612Z/var/lib/replicated/support-bundles/replicated-support727809798.tar.gz
replicated-support727809798/
replicated-support727809798/10.0.32.4/
replicated-support727809798/10.0.32.4/VERSION.json
replicated-support727809798/10.0.32.4/app/
replicated-support727809798/10.0.32.4/app/container-logs/
replicated-support727809798/10.0.32.4/app/container-logs/logs/
replicated-support727809798/10.0.32.4/app/container-logs/logs/20220926180055-8537bea47703.stderr.log.gz
replicated-support727809798/10.0.32.4/app/container-logs/logs/20220926180055-8537bea47703.stdout.log.gz
replicated-support727809798/10.0.32.4/app/container-logs/logs/20220926180059-ec6fe778dc09.stderr.log.gz
replicated-support727809798/10.0.32.4/app/container-logs/logs/20220926180127-ab61b1748f9f.stdout.log.gz
replicated-support727809798/10.0.32.4/app/container-logs/logs/20220926180201-b6cac52d9968.stderr.log.gz
replicated-support727809798/10.0.32.4/app/container-logs/logs/20220926180219-6abf017aa2f0.stderr.log.gz
replicated-support727809798/10.0.32.4/app/container-logs/logs/20220926180226-87f38c51d814.stderr.log.gz
replicated-support727809798/10.0.32.4/app/container-logs/logs/20220926180226-87f38c51d814.stdout.log.gz
## ...

Warning

We encourage you to inspect the bundle to ensure it contains only information that is appropriate to share. The hcdiag tool has default redactions which obscure common secrets or sensitive information, but we encourage you to verify the data before sending it to our support team.

When creating a support request for Terraform Enterprise, attach the hcdiag-generated .tar.gz bundle -- because it contains data that support engineers need, attaching it reduces the time it will take to troubleshoot your problem.

The hcdiag tool only works locally and does not export or share the diagnostic bundle with anyone. You must use other tools to transfer it to a secure location so you can share it with specific support staff who need to view it. For information on common workflows to process and send the hcdiag bundle, see our documentation

Configure hcdiag behavior

You can configure hcdiag's behavior with a HashiCorp Configuration Language (HCL) formatted file. Using this file, you can configure behavior by adding your own custom runners, redacting sensitive content using regular expressions, excluding commands, and more.

To run hcdiag with a custom configuration file, just create the file and point hcdiag at it with the -config flag:

$ hcdiag -config /path/to/your/configfile

Tip

This minimal environment doesn't ship with most common command-line text editors, so you'll want to install one with apt-get install nano or apt-get install vim, depending on which one you prefer.

Here is a small configuration file, which does two things:

It adds an agent-level (global) redaction which instructs hcdiag to redact all content that matches the pattern of a Vault Token.
It instructs hcdiag to skip the replicatedctl support-bundle command.

agent {
    redact "regex" {
        match = "hvs\\.[A-Za-z0-9]{24}"
        replace = "<VAULT TOKEN REDACTED>"
    }
}

product "terraform-ent" {
  excludes = [
    "replicatedctl support-bundle",
    "copy /var/lib/replicated/support-bundles/replicated-support*.tar.gz"
  ]
}

If you created this file as diag.hcl and executed hcdiag as follows, then you could expect output like this:

$ sudo hcdiag -terraform-ent -config diag.hcl
2022-10-19T15:39:43.764Z [INFO]  hcdiag: Ensuring destination directory exists: directory=.
2022-10-19T15:39:43.765Z [INFO]  hcdiag: Checking product availability
2022-10-19T15:39:43.765Z [INFO]  hcdiag: Gathering diagnostics
2022-10-19T15:39:43.765Z [INFO]  hcdiag.product: Running operations for: product=host
2022-10-19T15:39:43.765Z [INFO]  hcdiag.product: running operation: product=host runner="uname -v"
2022-10-19T15:39:43.765Z [INFO]  hcdiag.product: Running operations for: product=terraform-ent
2022-10-19T15:39:43.766Z [INFO]  hcdiag.product: running operation: product=terraform-ent runner="GET /api/v2/admin/customization-settings"
2022-10-19T15:39:43.770Z [INFO]  hcdiag.product: running operation: product=host runner=disks
2022-10-19T15:39:43.773Z [INFO]  hcdiag.product: running operation: product=host runner=info
2022-10-19T15:39:43.786Z [INFO]  hcdiag.product: running operation: product=host runner=memory
2022-10-19T15:39:43.786Z [INFO]  hcdiag.product: running operation: product=host runner=process
2022-10-19T15:39:43.798Z [INFO]  hcdiag.product: running operation: product=host runner=network
2022-10-19T15:39:43.801Z [INFO]  hcdiag.product: running operation: product=terraform-ent runner="GET /api/v2/admin/general-settings"
2022-10-19T15:39:43.804Z [INFO]  hcdiag.product: running operation: product=host runner=/etc/hosts
2022-10-19T15:39:43.806Z [INFO]  hcdiag.product: running operation: product=terraform-ent runner="GET /api/v2/admin/organizations"
2022-10-19T15:39:43.810Z [INFO]  hcdiag.product: running operation: product=terraform-ent runner="GET /api/v2/admin/terraform-versions"
2022-10-19T15:39:43.813Z [INFO]  hcdiag.product: running operation: product=terraform-ent runner="GET /api/v2/admin/twilio-settings"
2022-10-19T15:39:43.815Z [INFO]  hcdiag.product: running operation: product=host runner=iptables
2022-10-19T15:39:43.815Z [INFO]  hcdiag.product: running operation: product=terraform-ent runner="GET /api/v2/admin/workspaces?page[size]=1"
2022-10-19T15:39:43.818Z [INFO]  hcdiag.product: running operation: product=terraform-ent runner="docker -v"
2022-10-19T15:39:43.825Z [INFO]  hcdiag.product: running operation: product=host runner="/proc/ files"
2022-10-19T15:39:43.836Z [INFO]  hcdiag.product: running operation: product=host runner=/etc/fstab
2022-10-19T15:39:43.838Z [INFO]  hcdiag: Product done: product=host statuses=map[success:10]
2022-10-19T15:39:43.847Z [INFO]  hcdiag.product: running operation: product=terraform-ent runner="replicatedctl app status --output json"
2022-10-19T15:39:44.115Z [INFO]  hcdiag.product: running operation: product=terraform-ent runner="lsblk --json"
2022-10-19T15:39:44.120Z [INFO]  hcdiag.product: running operation: product=terraform-ent runner=getenforce
2022-10-19T15:39:44.122Z [WARN]  hcdiag.product: result: runner=getenforce status=unknown
  result=
  | /bin/bash: getenforce: command not found
   error="exec error, command=getenforce, error=exit status 127"
2022-10-19T15:39:44.122Z [INFO]  hcdiag.product: running operation: product=terraform-ent runner="env | grep -i proxy"
2022-10-19T15:39:44.126Z [WARN]  hcdiag.product: result: runner="env | grep -i proxy" status=unknown result="" error="exec error, command=env | grep -i proxy, error=exit status 1"
2022-10-19T15:39:44.126Z [INFO]  hcdiag: Product done: product=terraform-ent statuses="map[success:9 unknown:2]"
2022-10-19T15:39:44.126Z [INFO]  hcdiag: Recording manifest
2022-10-19T15:39:44.128Z [INFO]  hcdiag: Created Results.json file: dest=/home/tfeuser/hcdiag-2022-10-19T153943Z3317277613/Results.json
2022-10-19T15:39:44.129Z [INFO]  hcdiag: Created Manifest.json file: dest=/home/tfeuser/hcdiag-2022-10-19T153943Z3317277613/Manifest.json
2022-10-19T15:39:44.132Z [INFO]  hcdiag: Compressed and archived output file: dest=hcdiag-2022-10-19T153943Z.tar.gz
2022-10-19T15:39:44.132Z [INFO]  hcdiag: Writing summary of products and ops to standard output
product        success  fail  unknown  total
host           10       0     0        10
terraform-ent  9        0     2        11

Unarchiving and decompressing the resulting .tar.gz file shows that this hcdiag bundle no longer includes the replicated bundle:

$ tar zxvf hcdiag-*.tar.gz
hcdiag-2022-10-19T153943Z/Manifest.json
hcdiag-2022-10-19T153943Z/Results.json

Custom configuration is quite powerful, and can help you gather data from complex or nonstandard configurations. Please reference hcdiag's custom configuration documentation for more detailed configuration examples.

Production usage tips

By default, the hcdiag tool includes files from up to 72 hours before the current time. You can specify the desired time range using the -include-since flag.

Deploying hcdiag in production involves a workflow similar to the following:

Place the hcdiag binary on the host targeted by hcdiag.
When running with a configuration file and the -config flag, ensure that the specified configuration file is readable by the user that executes hcdiag.
Ensure that the current directory or that specified by the -dest flag is writable by the user that executes hcdiag.
Ensure connectivity to the HashiCorp products that hcdiag needs to connect to during the run. Export any required environment variables for establishing connection or passing authentication tokens as necessary.
Decide on a duration for information gathering, noting that the default is to gather for up to 72 hours back in server log output. Adjust your needs as necessary with the -include-since flag. For example, to include only 24 hours of log output, invoke as:
```
$   hcdiag -terraform-ent -include-since 24h
```
Use redactions to prevent sensitive information like keys or passwords from reaching hcdiag's output or the generated bundle files.
Use the -dryrun flag to observe what hcdiag will do without anything actually being done for testing configuration and options.

Next steps

In this tutorial, you installed, configured, and used hcdiag to gather troubleshooting information about your Terraform Enterprise application.

Visit the hcdiag repository to learn more about hcdiag. This repository contains information on additional configuration flags, along with detailed documentation on all hcdiag features.
If you are using Terraform Enterprise and are experiencing an incident, you can submit a request with your hcdiag bundle on this page. Visit this Support Article to learn how to add bundles and files to support tickets through the support portal.

Forward logs to Datadog

Next Collection

Terraform Enterprise

This tutorial also appears in:

4 tutorials

Customize Terraform Enterprise
Customize Terraform Enterprise to better collaborate with infrastructure. Follow these tutorials to enable logging and setting up single sign on (SSO) on your Terraform Enterprise instance. Debug and generate support bundles for Terraform Enterprise using hcdiags.
- Terraform