Telemetry

HCP Terraform Agents may be configured to emit helpful telemetry data, including both trace spans and metrics.

Telemetry architecture

HCP Terraform Agents emit telemetry data using the OpenTelemetry protocol. The agent is a simple producer of telemetry data, and relies on an OpenTelemetry Collector to perform aggregation, post-processing, and exporting of telemetry data to upstream technology and/or vendors. OpenTelemetry is an extensible telemetry protocol and open source platform, which enables the HCP Terraform Agent to remain vendor-agnostic while producing high-value telemetry data for a wide array of platforms.

Agent Configuration

To configure your agent to emit telemetry data, you must include the -otlp-address flag or TFC_AGENT_OTLP_ADDRESS environment variable. This should be set to the host:port address of an OpenTelemetry collector. This address should be a GRPC server running an OLTP collector.

Optionally, you can pass the -otlp-cert-file or TFC_AGENT_OTLP_CERT_FILE. The agent will use a certificate at the path supplied to encrypt the client connection to the OpenTelemetry collector. When omitted, client connections are not secured.

OpenTelemetry Collector

The OpenTelemetry Collector has a pluggable architecture. This opens up the ecosystem and makes it possible to leverage a wide array of telemetry platforms and technologies. The OpenTelemetry project distributes the collector in two flavors: core and contrib. For most uses, the contrib distribution will be most useful, as it includes a wide array of integrations for specific telemetry platforms and technologies which are not available in the core distribution. The contrib distribution is available as a pre-packaged Docker container labeled otel/opentelemetry-collector-contrib.

Compatibility

Each released version of the HCP Terraform Agent ships with a specific version of the OpenTelemetry client libraries built in. As these libraries and implementations change, it is possible that certain versions of the agent will become incompatible with newer versions of the OpenTelemetry collector. To track version compatibility, the table below will be updated whenever these version constraints become known.

Running the collector

By default, the contrib distribution accepts telemetry from a wide variety of sources, and dumps information about the metrics and traces collected to stdout. To start the collector with default settings, run the following command:

docker run -p 127.0.0.1:4317:4317/tcp otel/opentelemetry-collector-contrib:0.42.0

This will start an OpenTelemetry collector, and expose the gRPC interface (which the HCP Terraform Agent can talk to) on port 4317 of the local machine.

Configuring the collector

The OpenTelemetry collector takes configuration in the form of a YAML file. Refer to the default YAML configuration file as a starting point for writing your own configuration.

To tweak the configuration, save your custom YAML configuration to a file on disk called collector.yaml, and run the collector container with the following command:

docker run \
  --mount type=bind,source=${PWD}/collector.yml,target=/etc/otelcol-contrib/config.yaml \
  -p 127.0.0.1:4317:4317/tcp \
  otel/opentelemetry-collector-contrib:0.42.0

The collector should start with your modified configuration.

DataDog Agent

The HCP Terraform Agent may target a DataDog agent directly instead of an OpenTelemetry collector. To configure the DataDog agent to accept OTLP connections, refer to the DataDog OpenTelemetry documentation. Note that the tfc-agent talks to a gRPC OTLP receiver only. Be sure to configure the DataDog agent to accept a gRPC OTLP connection. Once the DataDog agent is listening for gRPC connections, [configure the tfc-agent] using the address of the DataDog agent's gRPC listener in place of the OpenTelemetry collector address.

Tracing

The agent emits tracing spans which are useful in understanding and debugging various operations performed by the HCP Terraform Agent.

See the Tracing documentation for details.

Metrics

HCP Terraform Agents emit granular metrics which are useful in understanding the behavior and performance of various operations in a high-level, aggregated view.

See the Metrics documentation for details.

Categories

Telemetry data can be broken down into three categories, global, Terraform, and policy.

Global

Once an Agent has registered, global fields are present on telemetry.

Terraform

Terraform fields are present on telemetry once an Agent begins handling a Terraform run.

Policy

Policy fields are present on telemetry once an Agent begins handling policy evaluation.