Container metrics reference

This topic provides reference information about the metrics that Terraform Enterprise can expose when enabled. Refer to Monitor Terraform Enterprise for instructions on how to enable metrics.

Container metrics

The following table describes metrics report runtime information about Terraform Enterprise containers.

Exposed Metric	Metrics Type	Description
`tfe.container.cpu.usage.user`	`counter`	Running count, in nanoseconds, of the total amount of time processes in the container have spent in userspace
`tfe.container.cpu.usage.kernel`	`counter`	Running count, in nanoseconds, of the total amount of time processes in the container have spent in kernel space
`tfe.container.memory.used_bytes`	`gauge`	The amount of memory allocated to the container in bytes, minus memory that is used for page cache
`tfe.container.memory.limit`	`gauge`	The maximum amount of memory in bytes that can be allocated by the container
`tfe.container.network.rx_bytes_total`	`counter`	Running count of the number of network bytes received by the container
`tfe.container.network.rx_packets_total`	`counter`	Running count of the number of network packets received by the container
`tfe.container.network.tx_bytes_total`	`counter`	Running count of the number of network bytes transmitted by the container
`tfe.container.network.tx_packets_total`	`counter`	Running count of the number of network packets transmitted by the container
`tfe.container.disk.io_op_read_total`	`counter`	Running count of the number of read disk operations executed by the container
`tfe.container.disk.io_op_write_total`	`counter`	Running count of the number of write disk operations executed by the container
`tfe.container.disk.io_bytes_read_total`	`counter`	Running count of the number of disk bytes read by the container
`tfe.container.disk.io_bytes_write_total`	`counter`	Running count of the number of disk bytes written by the container
`tfe.container.process_count`	`gauge`	The number of processes active within the container
`tfe.container.process_limit`	`gauge`	The maximum number of processes that can be executed inside the container

The following metadata labels will be added to each container metric emitted:

id: The container ID
name: The container name
image: The container image

Note: tfe.container.* metrics are not emitted on Kubernetes installations.

Worker container metrics include four additional labels: run_type, run_id, workspace_name, and organization_name. You can use these labels to associate a worker container with its type, run, workspace, and organization, respectively. Metrics for long-running service containers will not include these labels.

In addition to the per-container metrics, the following global metrics are exposed:

Exposed Metric	Metrics Type	Description
`tfe.run.count`	`gauge`	Number of running containers being used for Terraform runs.
`tfe.run.limit`	`gauge`	Maximum number of runs as defined by the `TFE_CAPACITY_CONCURRENCY` environment variable.
`tfe.run.current.count`	`gauge`	Number of active Terraform runs labeled by organization, workspace, and status.

The name and ID for worker containers are unique for each run, and worker container names take the form of a UUID. Be aware of this when planning for Prometheus storage capacity requirements that relate to metric cardinality. Environments that do not need to track resource consumption of individual build containers or runs can use Prometheus metric relabelling to remove the unique ID, name, and run type labels from container metrics. This reduces cardinality within the dataset while still retaining the ability to associate resource usage with a given workspace and organization.

Grafana dashboard

This template Grafana dashboard demonstrates how you can use Grafana and Prometheus to visualize exported Terraform Enterprise metrics.