Implement logging

10min
|
Terraform

In this tutorial, you will implement log messages in your provider and filter special values from the log output. Then you will manage log output to view those log statements when executing Terraform. To do this, you will:

Add log messages.
This creates provider-defined log messages in Terraform's logs.
Add structured log fields.
This enhances logging data with provider-defined key-value pairs for greater consistency across multiple logs and easier log viewing.
Add log filtering.
This redacts certain log messages or structured log field data from being included in the log output.
View all Terraform log output during commands.
This shows all Terraform logs in the terminal running a Terraform command.
Save Terraform log output to a file during commands.
This saves all Terraform logs to a file when running a Terraform command.
View specific Terraform log output.
This manages Terraform log output to show only certain logs.

Prerequisites

To follow this tutorial, you need:

Go 1.21+ installed and configured.
Terraform v1.8+ installed locally.
Docker and Docker Compose to run an instance of HashiCups locally.

Navigate to your terraform-provider-hashicups directory.

Your code should match the 03-read-coffees directory from the example repository.

If you haven't already done so, clone the Terraform HashiCups Provider repository.

$ git clone https://github.com/hashicorp/terraform-provider-hashicups

Change into the 03-read-coffees directory of cloned repository.

$ cd terraform-provider-hashicups/03-read-coffees

Then, install all the provider's dependencies.

$ go mod tidy

Build the provider

Terraform installs providers and verifies their versions and checksums when you run terraform init. Terraform will download your providers from either the provider registry or a local registry. However, while building your provider you will want to test Terraform configuration against a local development build of the provider. The development build will not have an associated version number or an official set of checksums listed in a provider registry.

Terraform allows you to use local provider builds by setting a dev_overrides block in a configuration file called .terraformrc. This block overrides all other configured installation methods.

Terraform searches for the .terraformrc file in your home directory and applies any configuration settings you set.

First, find the GOBIN path where Go installs your binaries. Your path may vary depending on how your Go environment variables are configured.

$ go env GOBIN
/Users/<Username>/go/bin

If the GOBIN go environment variable is not set, use the default path, /Users/<Username>/go/bin.

Create a new file called .terraformrc in the root directory (~), then add the dev_overrides block below. Change the <PATH> to the value returned from the echo $GOBIN command above.

provider_installation {

  dev_overrides {
      "hashicorp.com/edu/hashicups" = "<PATH>"
  }

  # For all other providers, install them directly from their origin provider
  # registries as normal. If you omit this, Terraform will _only_ use
  # the dev_overrides block, and so no other providers will be available.
  direct {}
}

You must create your CLI configuration file in your user %APPDATA% root directory. The physical location of this directory depends on your Windows version and system configuration.

Search for the user environment variables in PowerShell to find its location on your system.

$ $env:APPDATA

In another terminal, change into the directory returned from the command above.

Find the GOBIN path where go installs your binaries. Your path may vary depending on how your Go environment variables are configured.

$ go env GOBIN
/Users/<Username>/go/bin

If the GOBIN go environment variable is not set, use the default path, /Users/<Username>/go/bin.

Create a new file called terraform.rc and add the dev_overrides block below. Change the <PATH> to the value returned from the go env GOBIN command above.

provider_installation {

  dev_overrides {
      "hashicorp.com/edu/hashicups" = "<PATH>"
      }

  # For all other providers, install them directly from their origin provider
  # registries as normal. If you omit this, Terraform will _only_ use
  # the dev_overrides block, and so no other providers will be available.
  direct {}
}

Start HashiCups locally

Your HashiCups provider requires a running instance of HashiCups. Navigate to the docker_compose directory.

$ cd docker_compose

Run docker-compose up to spin up a local instance of HashiCups on port 19090.

$ docker-compose up

Leave this process running in your terminal window. The HashiCups service will print out log messages in this terminal.

In the original terminal window, verify that HashiCups is running by sending a request to its health check endpoint. The HashiCups service will respond with ok.

$ curl localhost:19090/health/readyz
ok

Create a HashiCups user

HashiCups requires a username and password to generate a JSON web token (JWT) which is used to authenticate against protected endpoints. You will use this user to authenticate to the HashiCups provider to manage your orders.

Create a user on HashiCups named education with the password test123.

$ curl -X POST localhost:19090/signup -d '{"username":"education", "password":"test123"}'
{"UserID":1,"Username":"education","token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTEwNzgwODUsInVzZXJfaWQiOjIsInVzZXJuYW1lIjoiZWR1Y2F0aW9uIn0.CguceCNILKdjOQ7Gx0u4UAMlOTaH3Dw-fsll2iXDrYU"}

Set the HASHICUPS_TOKEN environment variable to the token you retrieved from invoking the /signup endpoint. You will use this later in the tutorial to verify that Terraform has created your HashiCups order.

$ export HASHICUPS_TOKEN=ey...

The terminal containing your HashiCups logs will record the sign up operation.

api_1  | 2020-12-10T09:19:50.601Z [INFO]  Handle User | signup

Now that the HashiCups app is running, you are ready to start working on the Terraform provider.

Implement log messages

Providers support logging through the tflog package of the github.com/hashicorp/terraform-plugin-log Go module. This package implements structured logging and filtering capabilities.

Open the internal/provider/provider.go file.

Update the top of the Configure method logic with the following.

func (p *hashicupsProvider) Configure(ctx context.Context, req provider.ConfigureRequest, resp *provider.ConfigureResponse) {
    tflog.Info(ctx, "Configuring HashiCups client")

    // Retrieve provider data from configuration
    var config hashicupsProviderModel
    /* ... */

Replace the import statement at the beginning of the file with the following.

import (
    "context"
    "os"

    "github.com/hashicorp-demoapp/hashicups-client-go"
    "github.com/hashicorp/terraform-plugin-framework/datasource"
    "github.com/hashicorp/terraform-plugin-framework/path"
    "github.com/hashicorp/terraform-plugin-framework/provider"
    "github.com/hashicorp/terraform-plugin-framework/provider/schema"
    "github.com/hashicorp/terraform-plugin-framework/resource"
    "github.com/hashicorp/terraform-plugin-framework/types"
    "github.com/hashicorp/terraform-plugin-log/tflog"
)

Implement structured log fields

The tflog package supports adding additional key-value pairs to logging for consistency and tracing flow. These pairs can be added for the rest of the provider request with the tflog.SetField() call or inline as a final parameter with any logging calls.

Open the internal/provider/provider.go file.

Inside your provider's Configure method, set three logging fields and a log message immediately before the hashicups.NewClient() call with the following.

    /* ... */
    if resp.Diagnostics.HasError() {
        return
    }

    ctx = tflog.SetField(ctx, "hashicups_host", host)
    ctx = tflog.SetField(ctx, "hashicups_username", username)
    ctx = tflog.SetField(ctx, "hashicups_password", password)

    tflog.Debug(ctx, "Creating HashiCups client")

    // Create a new HashiCups client using the configuration values
    client, err := hashicups.NewClient(&host, &username, &password)
    /* ... */

Add a log message at the end of the Configure method with the following.

    /* ... */
    // Make the HashiCups client available during DataSource and Resource
    // type Configure methods.
    resp.DataSourceData = client
    resp.ResourceData = client

    tflog.Info(ctx, "Configured HashiCups client", map[string]any{"success": true})
}

Implement log filtering

Add a filter to mask the user's password before the tflog.Debug(ctx, "Creating HashiCups client") call in the Configure method with the following.

    /* ... */
    ctx = tflog.SetField(ctx, "hashicups_host", host)
    ctx = tflog.SetField(ctx, "hashicups_username", username)
    ctx = tflog.SetField(ctx, "hashicups_password", password)
    ctx = tflog.MaskFieldValuesWithFieldKeys(ctx, "hashicups_password")

    tflog.Debug(ctx, "Creating HashiCups client")
    /* ... */

Build and install the updated provider.

$ go install .

View all Terraform log output

Log output from Terraform is controlled by various environment variables, such as TF_LOG or otherwise prefixed with TF_LOG_.

Navigate to the examples/coffees directory.

$ cd examples/coffees

Run a Terraform plan with the TF_LOG environment variable set to TRACE.

Terraform will output a large amount of log entries for all components within Terraform itself, the Terraform Plugin Framework, and any provider logging.

$ TF_LOG=TRACE terraform plan
##...
2022-09-19T09:33:34.487-0500 [INFO]  provider.terraform-provider-hashicups: Configuring HashiCups client: tf_provider_addr=hashicorp.com/edu/hashicups tf_req_id=UUID tf_rpc=ConfigureProvider @caller=PATH @module=hashicups_pf timestamp=2022-09-19T09:33:34.487-0500
2022-09-19T09:33:34.487-0500 [DEBUG] provider.terraform-provider-hashicups: Creating HashiCups client: @module=hashicups_pf hashicups_password=*** tf_req_id=UUID @caller=PATH hashicups_host=http://localhost:19090 hashicups_username=education tf_provider_addr=hashicorp.com/edu/hashicups tf_rpc=ConfigureProvider timestamp=2022-09-19T09:33:34.487-0500
2022-09-19T09:33:34.517-0500 [INFO]  provider.terraform-provider-hashicups: Configured HashiCups client: tf_rpc=ConfigureProvider hashicups_password=*** hashicups_username=education success=true tf_provider_addr=hashicorp.com/edu/hashicups tf_req_id=UUID @caller=PATH @module=hashicups_pf hashicups_host=http://localhost:19090 timestamp=2022-09-19T09:33:34.517-0500
##...

Save all Terraform log output

Viewing log output inline with the running command is helpful if you plan on looking through the logs in the same terminal session or want to pipe the output to other shell commands, such as grep. Terraform can instead write this output to a log file on the local filesystem for opening in text editors or for archiving purposes.

Run a Terraform plan with both the TF_LOG and TF_LOG_PATH environment variables set.

$ TF_LOG=TRACE TF_LOG_PATH=trace.txt terraform plan

Open the examples/coffees/trace.txt file and verify it contains the log messages you added to your provider's Configure method.

##...
2022-09-30T16:23:38.515-0500 [DEBUG] provider.terraform-provider-hashicups: Calling provider defined Provider Configure: tf_provider_addr=hashicorp.com/edu/hashicups tf_req_id=12541d93-279d-1ec0-eac1-d2d2fcfd4030 tf_rpc=ConfigureProvider @caller=/Users/YOU/go/pkg/mod/github.com/hashicorp/terraform-plugin-framework@v0.13.0/internal/fwserver/server_configureprovider.go:12 @module=sdk.framework timestamp=2022-09-30T16:23:38.515-0500
2022-09-30T16:23:38.515-0500 [INFO]  provider.terraform-provider-hashicups: Configuring HashiCups client: tf_req_id=12541d93-279d-1ec0-eac1-d2d2fcfd4030 tf_rpc=ConfigureProvider @caller=/Users/YOU/code/terraform-provider-hashicups/hashicups/provider.go:66 @module=hashicups_pf tf_provider_addr=hashicorp.com/edu/hashicups timestamp=2022-09-30T16:23:38.515-0500
2022-09-30T16:23:38.515-0500 [DEBUG] provider.terraform-provider-hashicups: Creating HashiCups client: hashicups_password=*** tf_req_id=12541d93-279d-1ec0-eac1-d2d2fcfd4030 tf_provider_addr=hashicorp.com/edu/hashicups tf_rpc=ConfigureProvider @caller=/Users/YOU/code/terraform-provider-hashicups/hashicups/provider.go:171 @module=hashicups_pf hashicups_host=http://localhost:19090 hashicups_username=education timestamp=2022-09-30T16:23:38.515-0500
2022-09-30T16:23:38.524-0500 [INFO]  provider.terraform-provider-hashicups: Configured HashiCups client: @module=hashicups_pf hashicups_password=*** hashicups_username=education tf_req_id=12541d93-279d-1ec0-eac1-d2d2fcfd4030 tf_rpc=ConfigureProvider @caller=/Users/YOU/code/terraform-provider-hashicups/hashicups/provider.go:190 hashicups_host=http://localhost:19090 success=true tf_provider_addr=hashicorp.com/edu/hashicups timestamp=2022-09-30T16:23:38.524-0500
2022-09-30T16:23:38.524-0500 [DEBUG] provider.terraform-provider-hashicups: Called provider defined Provider Configure: @module=sdk.framework tf_provider_addr=hashicorp.com/edu/hashicups tf_req_id=12541d93-279d-1ec0-eac1-d2d2fcfd4030 tf_rpc=ConfigureProvider @caller=/Users/YOU/go/pkg/mod/github.com/hashicorp/terraform-plugin-framework@v0.13.0/internal/fwserver/server_configureprovider.go:20 timestamp=2022-09-30T16:23:38.524-0500
##...

Remove the examples/coffees/trace.txt file.

$ rm trace.txt

View specific Terraform log output

The previous examples used the TRACE logging level. Trace logs are the most verbose level of logging available and may contain an overwhelming amount of information that is only relevant if you deeply understand certain internal components of Terraform or the Terraform Plugin Framework. You can instead lower the logging level to DEBUG, INFO, WARN, or ERROR.

Run a Terraform plan with the TF_LOG environment variable set to INFO.

$ TF_LOG=INFO terraform plan
2022-09-30T16:27:38.446-0500 [INFO]  Terraform version: 1.3.0
2022-09-30T16:27:38.447-0500 [INFO]  Go runtime version: go1.19.1
2022-09-30T16:27:38.447-0500 [INFO]  CLI args: []string{"terraform", "plan"}
##...

You can enable log output for certain components, such as only provider logs and not Terraform CLI logs.

Run a Terraform plan with the TF_LOG_PROVIDER environment variable set to INFO.

$ TF_LOG_PROVIDER=INFO terraform plan
##...
2022-12-14T10:39:33.247-0600 [INFO]  provider.terraform-provider-hashicups: Configuring HashiCups client: @caller=/Users/YOU/code/terraform-provider-hashicups/hashicups/provider.go:61 @module=hashicups_pf tf_provider_addr=hashicorp.com/edu/hashicups tf_req_id=45969718-ed46-42b3-2cb9-847635d5aebb tf_rpc=ConfigureProvider timestamp=2022-12-14T10:39:33.247-0600
2022-12-14T10:39:33.255-0600 [INFO]  provider.terraform-provider-hashicups: Configured HashiCups client: success=true hashicups_host=http://localhost:19090 hashicups_username=education tf_provider_addr=hashicorp.com/edu/hashicups tf_req_id=45969718-ed46-42b3-2cb9-847635d5aebb tf_rpc=ConfigureProvider @caller=/Users/YOU/code/terraform-provider-hashicups/hashicups/provider.go:184 @module=hashicups_pf timestamp=2022-12-14T10:39:33.255-0600
data.hashicups_coffees.edu: Reading...
data.hashicups_coffees.edu: Read complete after 0s
##...

Navigate to the terraform-provider-hashicups directory.

cd ../..

Next steps

Congratulations! You have implemented logging in the provider, viewed it, and refined the output.

If you were stuck during this tutorial, checkout the 04-logging directory in the example repository to see the code implemented in this tutorial.

To learn more about the Terraform Plugin Framework, refer to the Terraform Plugin Framework documentation.
For a full capability comparison between the SDKv2 and the Plugin Framework, refer to the Which SDK Should I Use? documentation.
The example repository contains directories corresponding to each tutorial in this collection.
Submit any Terraform Plugin Framework bug reports or feature requests to the development team in the Terraform Plugin Framework Github repository.
Submit any Terraform Plugin Framework questions in the Terraform Plugin Framework Discuss forum.

Data sources

Create and read