Troubleshooting common application errors

This topic describes the errors Terraform Enterprise may report if your deployment is misconfigured and how to resolve them.

Kubernetes Fails to Pull Image

Symptom

Kubernetes pods are failing to pull the container image with a BackOff error.

Signals

kubectl describe pod is stuck in the Waiting state with the ErrImagePull reason.

$ kubectl describe pod terraform-enterprise-7f649f6598-2k79b
...
Containers:
  terraform-enterprise:
    State:          Waiting
      Reason:       ErrImagePull
...

Solution

Update the image pull policy for the deployment to always.

Empty S3 static credentials

Symptom

Application fails to start.

Signals

Logs show the following S3 prefix detection error.

2023-05-10T23:38:18.100Z [ERROR] terraform-enterprise: startup: error="failed detecting s3 prefix: could not list objects: operation error S3: ListObjectsV2, failed to sign request: failed to retrieve credentials: failed to refresh cached credentials, static credentials are empty"

Solution

Set TFE_OBJECT_STORAGE_S3_USE_INSTANCE_PROFILE to true when using IAM auth for S3.

Unknown certificate with VCS integration

Symptom

You cannot configure a VCS connection within Terraform Enterprise.

Signals

Setting up VCS fails with unknown certificate issuer error.

Solution

Include the CA certificate for your VCS server in the CA Bundle. Ensure the TFE_TLS_CA_BUNDLE_FILE is set to a path pointing to your CA bundle.

Unknown certificate with failing Terraform runs

Symptom

Terraform plans and applies fail.

Signals

Logs for task worker and archivist show an x509 error.

Solution

Include the CA certificates for all hosts that Terraform must communicate with, including your Terraform Enterprise server itself, in the CA Bundle. Ensure the TFE_TLS_CA_BUNDLE_FILE is set to a path pointing to your CA bundle.

Unable to fetch Terraform binary

Symptom

Terraform plans and applies fail with failed downloading terraform.

Signals

Terraform run logs contain.

Operation failed: failed fetching Terraform: failed downloading terraform: failed downloading "https://releases.hashicorp.com/terraform/1.3.2/terraform_1.3.2_linux_amd64.zip": GET https://releases.hashicorp.com/terraform/1.3.2/terraform_1.3.2_linux_amd64.zip giving up after 5 attempt(s): failed making temp file: open /tmp/terraform/8c23e18ed1846a552fc22ed5ee80eec9.download-67d5219a-aa5c-cd41-3262-2b9d57c1bfe2: read-only file system

Solution

Ensure the TFE_DISK_CACHE_PATH location is properly backed by a writable volume.

Unable to write to database after a failover

Symptom

If Terraform Enteprise is connected to a PostgreSQL database cluster, you may experience issues after a failover.

Signals

The Vault logs contain the following entry:

cannot execute INSERT in a read-only transaction (SQLSTATE 25006)

Solutions

You may need to manually address issues after a failover to return to functionality. The Vault process may still be connected to a read-only instance if the affected instance can not process runs. Perform the following actions to resolve this issue:

Restart Terraform Enterprise.
Reopen sealed Vault processes. If the Vault process seals before the failover resolves, either restart Terraform Enterprise or restart the Vault process. Use the following command to restart the Vault process:
```
$ supervisorctl restart tfe:vault
```