Deploy HCP Consul Dedicated with ECS using Terraform

13min
|
HCP
Terraform
Consul

In the previous tutorial, you learned how to deploy a new HCP Consul Dedicated cluster and to deploy your workload in an EC2 run time created in the same operation with Terraform.

In this tutorial, you will learn how to deploy a new HCP Consul Dedicated cluster and deploy your demo application workload in an ECS cluster, created in the same operation with Terraform.

HCP Consul Dedicated deployment workflow ECS

Prerequisites

To complete this tutorial you will need the following.

Basic command line access
Terraform v1.0.0+ CLI installed
Git installed
Admin access to the HashiCorp Cloud Platform (HCP) Consul portal
Note
HCP Admin access is necessary to create the Service Principal credentials used by Terraform to interact with HCP. If you already have a Service Principal key and client id provided by your admin, you only require Contributor access. If you are an Admin and would like to create a Service Principal, check Deploy HCP Consul Dedicated with Terraform tutorial for instructions on how to create a Service Principal.

An AWS account and AWS Access Credentials configured locally.

You can configure the AWS credentials using environment variables.

export AWS_ACCESS_KEY_ID=<your AWS access key ID>
export AWS_SECRET_ACCESS_KEY=<your AWS secret access key>
export AWS_SESSION_TOKEN=<your AWS session token>

Generate Terraform template

You can generate a Terraform template for this example directly from the Overview page in your HCP organization.

HCP UI Consul - Deploy with Terraform

HCP UI Consul - Deploy with Terraform - ECS with existing VPC

Note

Click on the Where can I find this? links in the UI to get help in locating the right values.

Authenticate Terraform to HCP

To authenticate Terraform to HCP you need a Service Principal with Contributor permissions. If you are logged with an Admin account you can create one during this step.

In the Authenticate Terraform to HCP section click on the Generate Service Principal and Key.

HCP UI Consul - Deploy with Terraform - Download TF code for ECS

HCP will generate a new set of credentials for you and you can copy them using the Copy code button and export them in your terminal.

export HCP_CLIENT_ID=<your client id>
export HCP_CLIENT_SECRET=<the key generated>

Note

If you are not an Admin on HCP you should contact your administrator and obtain valid Service Principal credentials before proceeding with the tutorial.

Get Terraform code

Once you have filled in all the options in the bottom side of the page you will find the generated Terraform code.

HCP UI Consul - Deploy with Terraform - Download TF code for ECS

Click on Copy code to copy it to your clipboard and save it in a file named main.tf.

Click here for an example configuration

Note

Content should resemble the example below. This example is not guaranteed to be up to date. Always refer to the template file provided by HCP UI after the configuration.

locals {
  vpc_region             = "us-west-2"
  hvn_region             = "us-west-2"
  cluster_id             = "consul-quickstart-1642458594797"
  hvn_id                 = "consul-quickstart-1642458594797-hvn"
  vpc_id                 = "{{ .VPCID }}"
  public_route_table_id  = "{{ .PublicRouteTableID }}"
  private_route_table_id = "{{ .PrivateRouteTableID }}"
  public_subnet1         = "{{ .PublicSubnet1 }}"
  public_subnet2         = "{{ .PublicSubnet2 }}"
  private_subnet1        = "{{ .PrivateSubnet1 }}"
  private_subnet2        = "{{ .PrivateSubnet2 }}"
}

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.43"
    }
    hcp = {
      source  = "hashicorp/hcp"
      version = ">= 0.18.0"
    }
  }

}

provider "aws" {
  region = local.vpc_region
}

provider "consul" {
  address    = hcp_consul_cluster.main.consul_public_endpoint_url
  datacenter = hcp_consul_cluster.main.datacenter
  token      = hcp_consul_cluster_root_token.token.secret_id
}

resource "hcp_hvn" "main" {
  hvn_id         = local.hvn_id
  cloud_provider = "aws"
  region         = local.hvn_region
  cidr_block     = "172.25.32.0/20"
}

module "aws_hcp_consul" {
  source  = "hashicorp/hcp-consul/aws"
  version = "~> 0.5.0"

  hvn    = hcp_hvn.main
  vpc_id = local.vpc_id
  subnet_ids = concat(
    [local.private_subnet1, local.private_subnet2],
    [local.public_subnet1, local.public_subnet2],
  )
  route_table_ids = concat(
    [local.private_route_table_id],
    [local.public_route_table_id],
  )
}

resource "hcp_consul_cluster" "main" {
  cluster_id      = local.cluster_id
  hvn_id          = hcp_hvn.main.hvn_id
  public_endpoint = true
  tier            = "development"
}

resource "consul_config_entry" "service_intentions" {
  name = "*"
  kind = "service-intentions"

  config_json = jsonencode({
    Sources = [
      {
        Action     = "allow"
        Name       = "*"
        Precedence = 9
        Type       = "consul"
      },
    ]
  })
}

resource "hcp_consul_cluster_root_token" "token" {
  cluster_id = hcp_consul_cluster.main.id
}

module "aws_ecs_cluster" {
  source  = "hashicorp/hcp-consul/aws//modules/hcp-ecs-client"
  version = "~> 0.5.0"

  private_subnet_ids       = [local.private_subnet1, local.private_subnet2]
  public_subnet_ids        = [local.public_subnet1, local.public_subnet2]
  vpc_id                   = local.vpc_id
  security_group_id        = module.aws_hcp_consul.security_group_id
  allowed_ssh_cidr_blocks  = ["0.0.0.0/0"]
  allowed_http_cidr_blocks = ["0.0.0.0/0"]
  client_config_file       = hcp_consul_cluster.main.consul_config_file
  client_ca_file           = hcp_consul_cluster.main.consul_ca_file
  client_gossip_key        = jsondecode(base64decode(hcp_consul_cluster.main.consul_config_file))["encrypt"]
  client_retry_join        = jsondecode(base64decode(hcp_consul_cluster.main.consul_config_file))["retry_join"]
  region                   = local.vpc_region
  root_token               = hcp_consul_cluster_root_token.token.secret_id
  consul_url               = hcp_consul_cluster.main.consul_private_endpoint_url
  consul_version           = substr(hcp_consul_cluster.main.consul_version, 1, -1)
  datacenter               = hcp_consul_cluster.main.datacenter
}

output "consul_root_token" {
  value     = hcp_consul_cluster_root_token.token.secret_id
  sensitive = true
}

output "consul_url" {
  value = hcp_consul_cluster.main.public_endpoint ? (
    hcp_consul_cluster.main.consul_public_endpoint_url
    ) : (
    hcp_consul_cluster.main.consul_private_endpoint_url
  )
}

output "hashicups_url" {
  value = "http://${module.aws_ecs_cluster.hashicups_url}"
}

Locals

The values you provided in the UI during the creation are used as local variables in the generated Terraform code.

vpc_region - This is the region where you deployed your VPC.
hvn_region - The HashiCorp Virtual Network (HVN) region.
cluster_id - The HCP Consul Dedicated cluster ID. Use a unique name to identify your HCP Consul Dedicated cluster. HCP will pre-populate it with a name following the pattern consul-quickstart-<unique-ID>.
vpc_id - Since you are using an existing VPC you need to instruct Terraform on the VPC ID to be able to use it.
public_route_table_id - A route table contains a set of rules, called routes, that are used to determine where network traffic from your subnet or gateway is directed. This is the route table ID for the public subnets.
private_route_table_id - A route table contains a set of rules, called routes, that are used to determine where network traffic from your subnet or gateway is directed. This is the route table ID for the private subnets.
public_subnet1 - The quickstart demo design requires two public and private subnets. Each pair of public-private subnets must be in different availability zones. The public subnets host external facing ALBs. The value you specified in public subnet field in availability zone 1 is populated.
private_subnet1 - Ensure that this private subnet has internet connectivity. The private subnets require internet connectivity to download demo app container images and to access AWS secrets manager to populate credentials. The value you specified in private subnet field in availability zone 1 is populated.
public_subnet2 - A second public subnet. The value you specified in public subnet field in availability zone 2 is populated.
private_subnet2 - Ensure that this private subnet has internet connectivity. The value you specified in private subnet field in availability zone 2 is populated.

HCP UI Consul - Deploy with Terraform - ECS with new VPC

Authenticate Terraform to HCP

To authenticate Terraform to HCP you need a Service Principal with Contributor permissions. If you are logged with an Admin account you can create one during this step.

In the Authenticate Terraform to HCP section click on the Generate Service Principal and Key.

HCP UI Consul - Deploy with Terraform - Download TF code for ECS

HCP will generate a new set of credentials for you and you can copy them using the Copy code button and export them in your terminal.

export HCP_CLIENT_ID=<your client id>
export HCP_CLIENT_SECRET=<the key generated>

Note

If you are not an Admin on HCP you should contact your administrator and obtain valid Service Principal credentials before proceeding with the tutorial.

Get Terraform code

Once you have filled in all the options in the bottom side of the page you will find the generated Terraform code.

HCP UI Consul - Deploy with Terraform - Download TF code for ECS

Click on Copy code to copy it to your clipboard and save it in a file named main.tf.

Click here for an example configuration

Note

Content should resemble the example below. This example is not guaranteed to be up to date. Always refer to the template file provided by HCP UI after the configuration.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.43"
    }
    hcp = {
      source  = "hashicorp/hcp"
      version = ">= 0.18.0"
    }
  }

  provider_meta "hcp" {
    module_name = "hcp-consul"
  }
}

provider "consul" {
  address    = hcp_consul_cluster.main.consul_public_endpoint_url
  datacenter = hcp_consul_cluster.main.datacenter
  token      = hcp_consul_cluster_root_token.token.secret_id
}

locals {
  vpc_region = "us-west-2"
  hvn_region = "us-west-2"
  cluster_id = "consul-quickstart-1641859870657"
}

provider "aws" {
  region = local.vpc_region
}

data "aws_availability_zones" "available" {}

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "3.10.0"

  name                 = "${local.cluster_id}-vpc"
  cidr                 = "10.0.0.0/16"
  azs                  = data.aws_availability_zones.available.names
  private_subnets      = ["10.0.1.0/24", "10.0.2.0/24", "10.0.3.0/24"]
  public_subnets       = ["10.0.4.0/24", "10.0.5.0/24", "10.0.6.0/24"]
  enable_dns_hostnames = true
  enable_nat_gateway   = true
  single_nat_gateway   = true
}

resource "hcp_hvn" "main" {
  hvn_id         = "${local.cluster_id}-hvn"
  cloud_provider = "aws"
  region         = local.hvn_region
  cidr_block     = "172.25.32.0/20"
}

module "aws_hcp_consul" {
  source = "hashicorp/hcp-consul/aws"

  hvn             = hcp_hvn.main
  vpc_id          = module.vpc.vpc_id
  subnet_ids      = module.vpc.public_subnets
  route_table_ids = module.vpc.public_route_table_ids
}

resource "hcp_consul_cluster" "main" {
  cluster_id      = local.cluster_id
  hvn_id          = hcp_hvn.main.hvn_id
  public_endpoint = true
  tier            = "development"
}

resource "consul_config_entry" "service_intentions" {
  name = "*"
  kind = "service-intentions"

  config_json = jsonencode({
    Sources = [
      {
        Action     = "allow"
        Name       = "*"
        Precedence = 9
        Type       = "consul"
      },
    ]
  })
}

resource "hcp_consul_cluster_root_token" "token" {
  cluster_id = hcp_consul_cluster.main.id
}

module "aws_ecs_cluster" {
  source  = "hashicorp/hcp-consul/aws//modules/hcp-ecs-client"
  version = "~> 0.4.1"

  private_subnet_ids       = module.vpc.private_subnets
  public_subnet_ids        = module.vpc.public_subnets
  vpc_id                   = module.vpc.vpc_id
  security_group_id        = module.aws_hcp_consul.security_group_id
  allowed_ssh_cidr_blocks  = ["0.0.0.0/0"]
  allowed_http_cidr_blocks = ["0.0.0.0/0"]
  client_config_file       = hcp_consul_cluster.main.consul_config_file
  client_ca_file           = hcp_consul_cluster.main.consul_ca_file
  client_gossip_key        = jsondecode(base64decode(hcp_consul_cluster.main.consul_config_file))["encrypt"]
  client_retry_join        = jsondecode(base64decode(hcp_consul_cluster.main.consul_config_file))["retry_join"]
  region                   = local.vpc_region
  root_token               = hcp_consul_cluster_root_token.token.secret_id
  consul_url               = hcp_consul_cluster.main.consul_private_endpoint_url
  datacenter               = hcp_consul_cluster.main.datacenter
  consul_version           = substr(hcp_consul_cluster.main.consul_version, 1, -1)

  depends_on = [module.aws_hcp_consul]
}

output "consul_root_token" {
  value     = hcp_consul_cluster_root_token.token.secret_id
  sensitive = true
}

output "consul_url" {
  value = hcp_consul_cluster.main.public_endpoint ? (
    hcp_consul_cluster.main.consul_public_endpoint_url
    ) : (
    hcp_consul_cluster.main.consul_private_endpoint_url
  )
}

output "hashicups_url" {
  value = "http://${module.aws_ecs_cluster.hashicups_url}"
}

Locals

The HCP Consul Dedicated UI will guide you into picking the right values for the locals. If you want to, you can edit the cluster_id but make sure it does not conflict with other deployments you have in your organization.

vpc_region - This is the region where you deployed your VPC.
hvn_region - The HashiCorp Virtual Network (HVN) region.
cluster_id - The HCP Consul Dedicated cluster ID. Use a unique name to identify your HCP Consul Dedicated cluster. HCP will pre-populate it with a name following the pattern consul-quickstart-<unique-ID>.

Run terraform

With the Terraform manifest files and your custom credentials file, you are now ready to deploy your infrastructure.

Check that the following setup is complete before executing the terraform init step:

Your AWS credentials are populated as environment variables and Terraform install is complete (refer to prerequisites)
You have exported the HCP credentials from the UI as environment variables
If you are deploying in an existing VPC: ensure the two public-private subnet pairs have internet connectivity and that the two pairs are in different availability zones.

Issue the terraform init command from your working directory to download the necessary providers and initialize the backend.

$ terraform init

Initializing the backend...

Initializing provider plugins...
...

Terraform has been successfully initialized!
...

Once Terraform has been initialized, you can verify the resources that will be created using the plan command.

$ terraform plan

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:
...

Finally, you can deploy the resources using the apply command.

$ terraform apply

...
Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value:

Remember to confirm the run by entering yes.

Once you confirm, it will take a few minutes to complete the deploy. Terraform will print the following output if the deployment is successful.

Apply complete! Resources: 37 added, 0 changed, 0 destroyed.

Examine Terraform output

At the end of the execution Terraform will output the following lines:

Outputs:

consul_root_token = <sensitive>
consul_url = "https://consul-quickstart-learn.consul.11eb5071-85f5-1eb2-992c-0242ac110003.aws.hashicorp.cloud"
hashicups_url = "http://consul-ecs-test-frontend-950796231.us-west-2.elb.amazonaws.com/"

As you can notice the consul_root_token is not showed since is a sensitive value.

You can retrieve it using:

$ terraform output consul_root_token

Verify created resources

Consul UI

Visit the Consul UI using the consul_url link in the output values.

Consul UI services HashiCups login request

Login in Consul using the token retrieved in the previous step and verify the services are all present in your Consul datacenter UI

Consul UI services HashiCups

Consul CLI configuration

Using the Terraform output values you can setup your Consul CLI to connect to the datacenter you created.

Setup environment variables:

$ export CONSUL_HTTP_TOKEN=$(terraform output -raw consul_root_token)

$ export CONSUL_HTTP_ADDR=$(terraform output -raw consul_url)

Verify Consul can connect to the datacenter:

$ consul members

Example output:

Node                                         Address             Status  Type    Build       Protocol  DC                               Segment
ip-172-25-41-122                             172.25.41.122:8301  alive   server  1.10.6+ent  2         consul-quickstart-1642014232516  <all>
ip-10-0-1-141.eu-central-1.compute.internal  10.0.1.141:8301     alive   client  1.10.6      2         consul-quickstart-1642014232516  <default>
ip-10-0-1-208.eu-central-1.compute.internal  10.0.1.208:8301     alive   client  1.10.6      2         consul-quickstart-1642014232516  <default>
ip-10-0-1-62.eu-central-1.compute.internal   10.0.1.62:8301      alive   client  1.10.6      2         consul-quickstart-1642014232516  <default>
ip-10-0-2-176.eu-central-1.compute.internal  10.0.2.176:8301     alive   client  1.10.6      2         consul-quickstart-1642014232516  <default>
ip-10-0-3-5.eu-central-1.compute.internal    10.0.3.5:8301       alive   client  1.10.6      2         consul-quickstart-1642014232516  <default>

HashiCups application

The Terraform code deployed an application that exposes a web UI accessible using the hashicups_url URL.

You can access the configurations of the deployed Hashicups app services here. HashiCups UI

Cleanup environment

Use the terraform destroy command to clean up the resources you created.

$ terraform destroy

...
Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value:

Remember to confirm by entering yes.

Once you confirm, it will take a few minutes to complete the removal. Terraform will print the following output if the command is successful.

Destroy complete! Resources: xx destroyed.

Next steps

In this tutorial you learned how to use Terraform to deploy a demo application on AWS ECS using HCP Consul Dedicated as your service mesh.

In the next tutorial you will learn how to edit the Terraform template to deploy a demo application on an existing EKS cluster using HCP Consul Dedicated as your service mesh.

If you encounter any issues, please contact the HCP team at support.hashicorp.com.

Deploy HCP with EC2

Deploy HCP with Azure VM