Kubernetes and OpenShift (Beta) requirements

OpenShift Beta

Terraform Enterprise operating on Red Hat OpenShift is still in beta. Do not deploy beta features in production environments. Provision a dedicated test environment before installing Terraform Enterprise on OpenShift. If you have questions or feedback about the OpenShift beta, contact your HashiCorp account representative. Refer toOperate on Red Hat OpenShift for additional information.

We recommend that developers have a deep understanding of Kubernetes or OpenShift before deploying Terraform Enterprise to a production environment.

Example of Kubernetes Architecture

Kubernetes and OpenShift deployments have different operational and observability considerations than Replicated- and Docker-based deployments. External service dependencies should be deployed outside the cluster and scale reliably to accommodate Terraform Enterprise workloads.

External services

Terraform Enterprise requires the following external services to install on Kubernetes or OpenShift:

PostgreSQL
Blob Storage (AWS S3, Azure Cloud Storage, Google Cloud Storage, or any S3-compatible storage service)
Redis version 6 or 7 (Redis Cluster and Redis Sentinel are not currently supported)

Runtime

Terraform Enterprise requires the following to deploy in a Kubernetes or OpenShift runtime:

A hostname for Terraform Enterprise
A valid TLS certificate and private key provisioned and matching the hostname selected in pem format
License as the password for Terraform Enterprise FDO container registry: images.releases.hashicorp.com
Install the Helm CLI version 3.0 or above. Learn more about Helm.

Network

Refer to the Network requirements

Configuration

You must create a custom values file (e.g., /tmp/overrides.yaml) to override the default values in the terraform-enterprise helm chart. Refer to Application configuration for a full list of customizable settings.

OpenShift requirements

The OpenShift default restricted security context constraints require that containers run under a unique user ID. There is no exception to these security constraints for tfc-agent. To plan and apply workspaces in Terraform Enterprise on OpenShift , you need a custom tfc-agent image that creates the default working directory for tfc-agent and assigns permissions to the root group.

FROM hashicorp/tfc-agent

USER root

RUN mkdir /.tfc-agent && \
    chmod 770 /.tfc-agent

USER tfc-agent

Place your custom tfc-agent image in a container registry accessible from the OpenShift nodes that will host Terraform Enterprise. Custom tfc-agent sourced images cannot use automatic CA certificate injection, and you may need to add CA certificate injection configuration to the Dockerfile below. Enterprise. Automatic CA certificate injection is not available when a custom image source is used for the tfc-agent image and may be required as an additional configuration in this Dockerfile. Use the following settings to configure Terraform Enterprise to use this image.

Example configurations

The below examples for each cloud-platform are based on cloud native hosted PostgreSQL, storage, or Redis cache services. Please customize the values in angle brackets before using these examples for you configuration.

The following is true for all of the below YAML examples:

Values under .env.variables are set as a ConfigMap and mounted as Terraform Enterprise environment variables.
Values under .env.secrets are set as Kubernetes secrets and mounted as Terraform Enterprise environment variables.
Extend the env.configMapRefs[] or env.secretRefs[] with your own resources to add additional ConfigMap or Secret resources within your environment configuration.

Note: In the below examples, any values marked BASE_64_ENCODED* indicates that the value given must be base 64 encoded. If you are using this certificate configuration to host Terraform Enterprise web traffic, this value must be valid with the env.TFE_HOSTNAME, or match the wildcard pattern.

AWS Elastic Kubernetes Service (EKS)

Below is an example configuration for AWS Elastic Kubernetes Services.

replicaCount: <Number of replicas e.g 3>
tls:
  certData: <BASE_64_ENCODED_CERTIFICATE_PEM_FILE>
  keyData: <BASE_64_ENCODED_CERTIFICATE_PRIVATE_KEY_PEM_FILE>
  caCertData: <BASE_64_ENCODED_CERTIFICATE_CA_CERTIFICATE_PEM_FILE>
image:
  repository: images.releases.hashicorp.com
  name: hashicorp/terraform-enterprise
  tag: <vYYYYMM-#>
env:
  variables:
    TFE_HOSTNAME: <TFE hostname>
    TFE_IACT_SUBNETS: <IACT subnet, eg. 10.0.0.0/8,192.168.0.0/24>

    # Database settings.
    TFE_DATABASE_HOST: <Database hostname with port e.g "xxx.us-west-2.rds.amazonaws.com:5432">
    TFE_DATABASE_NAME: <Database name>
    TFE_DATABASE_PARAMETERS: <Database extra params e.g "sslmode=disable">
    TFE_DATABASE_USER: <Database user>

    # Redis settings.
    TFE_REDIS_HOST: <Redis host, eg. 10.101.0.4>
    TFE_REDIS_USE_TLS: <To use tls? eg. "false">
    TFE_REDIS_USE_AUTH: <To use customized credential to authenticate? eg. "true">
    TFE_REDIS_USER: <Redis username>

    # S3 settings. For Server Side Encryption settings, see to the configuration reference.
    TFE_OBJECT_STORAGE_TYPE: s3
    TFE_OBJECT_STORAGE_S3_BUCKET: <S3 bucket name>
    TFE_OBJECT_STORAGE_S3_REGION: <S3 region>
    TFE_OBJECT_STORAGE_S3_USE_INSTANCE_PROFILE: <To use the pod's credentials to authenticate with AWS? e.g. false>
  secrets:
    TFE_DATABASE_PASSWORD: <Database password>
    TFE_OBJECT_STORAGE_S3_ACCESS_KEY_ID: <Required if TFE_OBJECT_STORAGE_S3_USE_INSTANCE_PROFILE is false>
    TFE_OBJECT_STORAGE_S3_SECRET_ACCESS_KEY: <Required if TFE_OBJECT_STORAGE_S3_USE_INSTANCE_PROFILE is false>
    TFE_REDIS_PASSWORD: <Redis password>
    TFE_LICENSE: <Hashicorp license>
    TFE_ENCRYPTION_PASSWORD: <Encryption password>

Google Kubernetes Engine (GKE)

Below is an example configuration for Google Kubernetes Engine.

replicaCount: <Number of replicas e.g 3>
tls:
  certData: <BASE_64_ENCODED_CERTIFICATE_PEM_FILE>
  keyData: <BASE_64_ENCODED_CERTIFICATE_PRIVATE_KEY_PEM_FILE>
  caCertData: <BASE_64_ENCODED_CERTIFICATE_CA_CERTIFICATE_PEM_FILE>
image:
 repository: images.releases.hashicorp.com
 name: hashicorp/terraform-enterprise
 tag: <vYYYYMM-#>
env:
  variables:
    TFE_HOSTNAME: <TFE hostname>
    TFE_IACT_SUBNETS: <IACT subnet, eg. 10.0.0.0/8,192.168.0.0/24>

    # Database settings.
    TFE_DATABASE_HOST: <Database hostname with port e.g 11.22.33.44:5432>
    TFE_DATABASE_NAME: <Database name>
    TFE_DATABASE_PARAMETERS: <Database extra params e.g "sslmode=require">
    TFE_DATABASE_USER: <Database user>

    # Redis settings.
    TFE_REDIS_HOST: <Redis host, eg. 10.101.0.4>
    TFE_REDIS_USE_TLS: <To use tls? eg. "false">
    TFE_REDIS_USE_AUTH: <To use customized credential to authenticate? eg. "true">
    TFE_REDIS_USER: <Redis username>

    # Google Cloud Storage settings.
    TFE_OBJECT_STORAGE_TYPE: google
    TFE_OBJECT_STORAGE_GOOGLE_BUCKET: <Bucket name>
    TFE_OBJECT_STORAGE_GOOGLE_PROJECT: <GCP project ID>
  secrets:
    TFE_DATABASE_PASSWORD: <Database password>
    TFE_OBJECT_STORAGE_GOOGLE_CREDENTIALS: <BASE_64_ENCODED_SERVICE_ACCOUNT_CREDENTIALS>
    TFE_REDIS_PASSWORD: <Redis password>
    TFE_LICENSE: <Hashicorp license>
    TFE_ENCRYPTION_PASSWORD: <Encryption password>

Azure Kubernetes Service (AKS)

Below is an example configuration for Azure Kubernetes Service.

replicaCount: <Number of replicas e.g 3>
tls:
  certData: <BASE_64_ENCODED_CERTIFICATE_PEM_FILE>
  keyData: <BASE_64_ENCODED_CERTIFICATE_PRIVATE_KEY_PEM_FILE>
  caCertData: <BASE_64_ENCODED_CERTIFICATE_CA_CERTIFICATE_PEM_FILE>
image:
 repository: images.releases.hashicorp.com
 name: hashicorp/terraform-enterprise
 tag: <vYYYYMM-#>
env:
  variables:
    TFE_HOSTNAME: <TFE hostname (DNS) e.g. terraform.example.com>
    TFE_IACT_SUBNETS: <IACT subnet, eg. 10.0.0.0/8,192.168.0.0/24>

    # Database settings.
    TFE_DATABASE_HOST: <Database hostname with port e.g "xxx.postgres.database.azure.com:5432">
    TFE_DATABASE_NAME: <Database name>
    TFE_DATABASE_PARAMETERS: <Database extra params e.g "sslmode=disable">
    TFE_DATABASE_USER: <Database user>

    # Redis settings.
    TFE_REDIS_HOST: <Redis host, eg. 10.101.0.4>
    TFE_REDIS_USE_TLS: <To use tls? eg. "false">
    TFE_REDIS_USE_AUTH: <To use customized credential to authenticate? eg. "true">
    TFE_REDIS_USER: <Redis username>

    # Azure container storage settings.
    TFE_OBJECT_STORAGE_TYPE: azure
    TFE_OBJECT_STORAGE_AZURE_ACCOUNT_NAME: <Azure storage account name>
    TFE_OBJECT_STORAGE_AZURE_CONTAINER: <Azure storage container name>
    TFE_OBJECT_STORAGE_AZURE_ENDPOINT: <Azure storage endpoint>
  secrets:
    TFE_DATABASE_PASSWORD: <Database password>
    TFE_OBJECT_STORAGE_AZURE_ACCOUNT_KEY: <Azure storage account key>
    TFE_REDIS_PASSWORD: <Redis password>
    TFE_LICENSE: <Hashicorp license>
    TFE_ENCRYPTION_PASSWORD: <Encryption password>

OpenShift in Azure with hosted external services

Below is an example configuration for an OpenShift environment hosted in Microsoft Azure.

replicaCount: <Number of replicas e.g 3>
tls:
  certData: <BASE_64_ENCODED_CERTIFICATE_PEM_FILE>
  keyData: <BASE_64_ENCODED_CERTIFICATE_PRIVATE_KEY_PEM_FILE>
  caCertData: <BASE_64_ENCODED_CERTIFICATE_CA_CERTIFICATE_PEM_FILE>
image:
  repository: images.releases.hashicorp.com
  name: hashicorp/terraform-enterprise
  tag: <vYYYYMM-#>
openshift:
  enabled: true
env:
  variables:
    TFE_HOSTNAME: <TFE hostname (DNS) e.g. terraform.example.com>
    TFE_IACT_SUBNETS: <IACT subnet, eg. 10.0.0.0/8,192.168.0.0/24>

    # Database settings.
    TFE_DATABASE_HOST: <Database hostname with port e.g "xxx.postgres.database.azure.com:5432">
    TFE_DATABASE_NAME: <Database name>
    TFE_DATABASE_PARAMETERS: <Database extra params e.g "sslmode=disable">
    TFE_DATABASE_USER: <Database user>

    # Redis settings.
    TFE_REDIS_HOST: <Redis host, eg. 10.101.0.4>
    TFE_REDIS_USE_TLS: <To use tls? eg. "false">
    TFE_REDIS_USE_AUTH: <To use customized credential to authenticate? eg. "true">
    TFE_REDIS_USER: <Redis username>

    # Azure container storage settings.
    TFE_OBJECT_STORAGE_TYPE: azure
    TFE_OBJECT_STORAGE_AZURE_ACCOUNT_NAME: <Azure storage account name>
    TFE_OBJECT_STORAGE_AZURE_CONTAINER: <Azure storage container name>
    TFE_OBJECT_STORAGE_AZURE_ENDPOINT: <Azure storage endpoint>

    # Terraform Enterprise on OpenShift Required settings
    TFE_RUN_PIPELINE_IMAGE: <URL and path to the custom tfc-agent image>
    TFE_RUN_PIPELINE_KUBERNETES_IMAGE_PULL_SECRET_NAME: <The name of an imagePullSecret created in the agents namespace for the tfc-agent image>

  secrets:
    TFE_DATABASE_PASSWORD: <Database password>
    TFE_OBJECT_STORAGE_AZURE_ACCOUNT_KEY: <Azure storage account key>
    TFE_REDIS_PASSWORD: <Redis password>
    TFE_LICENSE: <Hashicorp license>
    TFE_ENCRYPTION_PASSWORD: <Encryption password>

Enabling OpenShift (beta)

To deploy Terraform Enterprise on OpenShift, you'll need to set the Values.openshift.enabled option to true.

openshift:
  enabled: true

Below are additional reference materials for setting up these value files:

Terraform Enterprise Helm repository
Tag (release version)
Generic reference for values file to override the default values in the helm chart.

Follow the Kubernetes installation guide to install Terraform Enterprise application using helm.

Security context configuration

Modify the .securityContext helm chart value to set pod security configuration for Terraform Enterprise Flexible Deployment Options. Modify the .container.securityContext helm chart value to set the container security configuration. The allowPrivilegeEscalation container security context option must be omitted or set to true in order for Terraform Enterprise to function properly.