Access services in your service mesh

18min
|
Consul
Interactive

Workflow

In the previous tutorial, you deployed a Consul datacenter with service mesh enabled and an application running.

In this tutorial, you will learn how to add a Consul API Gateway in your service mesh and secure external network access to applications and services running in your Consul service mesh.

Note

This tutorial is part of the Get Started collection, for this reason all the steps used to configure Consul agents and services are shown and require to be executed manually. If you are setting up a production environment you should codify and automate the installation and deployment process. Refer to the VM production patterns tutorial collection for Consul production deployment best practices.

Tutorial scenario

This tutorial uses HashiCups, a demo coffee shop application made up of several microservices running on VMs.

At the beginning of the tutorial, you have a fully deployed Consul service mesh with Envoy sidecar proxies running alongside each service.

Architectural diagram 03

By the end of this tutorial, you will have enabled Consul API gateway and configured it to permit access to the HashiCups application over port 8443. You will also generate an SSL certificate to be exposed by the application.

Architectural diagram 04

Prerequisites

If you completed the previous tutorial, the infrastructure is already in place with all prerequisites needed.

For this tutorial, you will need:

An AWS account configured for use with Terraform
aws-cli >= 2.0
terraform >= 1.0
consul >= 1.15.0

Clone GitHub repository

Clone the GitHub repository containing the configuration files and resources.

$ git clone https://github.com/hashicorp-education/learn-consul-get-started-vms

Change into the directory that contains the complete configuration files for this tutorial.

$ cd learn-consul-get-started-vms/self-managed/infrastructure/aws

Create infrastructure

With these Terraform configuration files, you are ready to deploy your infrastructure.

Issue the terraform init command from your working directory to download the necessary providers and initialize the backend.

$ terraform init
Initializing the backend...
Initializing provider plugins...
...
Terraform has been successfully initialized!
...

Then, deploy the resources. Confirm the run by entering yes.

$ terraform apply -var-file=./conf/03_service_mesh.tfvars
## ...
Do you want to perform these actions?
 Terraform will perform the actions described above.
 Only 'yes' will be accepted to approve.
 Enter a value: yes
## ...
Apply complete! Resources: 49 added, 0 changed, 0 destroyed.

Tip

The Terraform deployment could take up to 15 minutes to complete. Feel free to explore the next sections of this tutorial while waiting for the environment to complete initialization or learn more about the Raft protocol in a fun, interactive way.

Once the deployment is complete, Terraform will return a list of outputs you can use to interact with the newly created environment.

Outputs:

connection_string = "ssh -i certs/id_rsa.pem admin@`terraform output -raw ip_bastion`"
ip_bastion = "<redacted-output>"
remote_ops = "export BASTION_HOST=<redacted-output>"
retry_join = "provider=aws tag_key=ConsulJoinTag tag_value=auto-join-hcoc"
ui_consul = "https://<redacted-output>:8443"
ui_grafana = "http://<redacted-output>:3000/d/hashicups/hashicups"
ui_hashicups = "http://<redacted-output>"

The Terraform outputs provide useful information, including the bastion host IP address. The following is a brief description of the Terraform outputs:

The ip_bastion provides IP address of the bastion host you will use to run the rest of the commands in this tutorial.
The remote_ops lists the bastion host IP, which you can use access the bastion host.
The retry_join output lists Consul's retry_join configuration parameter. The next tutorial will use this to generate Consul server and client configuration.
The ui_consul output lists the Consul UI address. The Consul UI is not currently running. You will use this in a later tutorial to verify Consul started correctly.
The ui_grafana output lists the Grafana UI address. You will use this in a future tutorial.
The ui_hashicups output lists the HashiCups UI address. You can use it to verify the HashiCups demo application is running properly.

Configure environment

This tutorial and interactive lab environment uses scripts in the tutorial's GitHub repository to generate the Consul configuration files for your client agents.

$ ssh -i certs/id_rsa.pem admin@`terraform output -raw ip_bastion`

Define scenario environment variables.

$ export DATACENTER="dc1"; \
export DOMAIN="consul"; \
export OUTPUT_FOLDER="./assets/scenario/conf/"; \
export CONSUL_CONFIG_DIR="/etc/consul.d/"

Configure the Consul CLI to interact with the Consul server.

$ export CONSUL_HTTP_ADDR="https://consul-server-0:8443" \
export CONSUL_HTTP_SSL=true \
export CONSUL_CACERT="${OUTPUT_FOLDER}secrets/consul-agent-ca.pem" \
export CONSUL_TLS_SERVER_NAME="server.${DATACENTER}.${DOMAIN}"

To interact with Consul, you need to set CONSUL_HTTP_TOKEN to a valid Consul token. For this tutorial, you will use the token you created during the ACL bootstrap.

If you completed the previous tutorial, the bootstrap token is located the home directory in a file named acl-token-bootstrap.json.

$ export CONSUL_HTTP_TOKEN=`cat ./acl-token-bootstrap.json | jq -r ".SecretID"`

If you are starting your journey from this tutorial, the token is located in the configuration directory in a file named acl-token-bootstrap.json.

$ export CONSUL_HTTP_TOKEN=`cat ${OUTPUT_FOLDER}secrets/acl-token-bootstrap.json | jq -r ".SecretID"`

Add API gateway node to Consul datacenter

Consul API Gateway uses the same components as the rest of the service mesh client nodes to join the Consul datacenter. This means that you need a Consul agent running and an Envoy proxy instance to act as a proxy for the services you want to expose outside your service mesh.

Generate Consul configuration for API gateway

Set the retry_join value using the Terraform output to be able to connect to the Consul servers.

$ export CONSUL_RETRY_JOIN="<use value of Terraform output retry_join here>"

With these variables in place, generate Consul agent configuration.

First, define the Consul node name.

$ export NODE_NAME="gateway-api"

Then, generate the Consul configuration for the API Gateway node.

$ bash ./ops/scenarios/99_supporting_scripts/generate_consul_client_config.sh
[generate_consul_client_config.sh] - - [gateway-api]
-- Parameter Check
-- Cleaning Scenario before apply.
-- [WARN] Removing pre-existing configuration in ./assets/scenario/conf/
-- Generate folder structure
-- Copy available configuration
-- Generating configuration for Consul agent gateway-api

To complete Consul agent configuration, you need to setup tokens for the client. For this tutorial, you are using the bootstrap token. We recommend using more restrictive tokens for your Consul client agents in production.

$ tee ${OUTPUT_FOLDER}${NODE_NAME}/agent-acl-tokens.hcl > /dev/null << EOF
acl {
  tokens {
    agent  = "${CONSUL_HTTP_TOKEN}"
    default  = "${CONSUL_HTTP_TOKEN}"
    config_file_service_registration = "${CONSUL_HTTP_TOKEN}"
  }
}
EOF

Check generated files

Once you have generated your configuration files, your directory should look like the following:

$ tree ${OUTPUT_FOLDER}gateway-api
./assets/scenario/conf/gateway-api
├── agent-acl-tokens.hcl
├── agent-gossip-encryption.hcl
├── consul-agent-ca.pem
└── consul.hcl

0 directories, 4 files

The scripts generated multiple configuration files to separate the configuration so it is easier to read and tune them for your environment. The following are the generated files and a description of what they do:

The agent-acl-tokens.hcl file contains tokens for the Consul agent.
The agent-gossip-encryption.hcl file configures gossip encryption.
The consul-agent-ca.pem file is the public certificate for Consul CA.
The consul.hcl file contains node specific configuration and it is needed, with this specific name, if you want to configure Consul as a systemd daemon.

Visit the agent configuration documentation to interpret the files or to modify them when applying them to your environment.

After the script generates the client configuration, you will copy these files into the API gateway node.

Tip

In the lab environment, the node has a running SSH server. As a result, you can use ssh and scp commands to perform the following operations. If the nodes in your personal environment does not have an SSH server, you may need to use a different approach to create the configuration directories and copy the files.

First, define the Consul configuration directory.

$ export CONSUL_REMOTE_CONFIG_DIR=/etc/consul.d/

Then, remove existing configuration from the VM.

$ ssh -i certs/id_rsa gateway-api "sudo rm -rf ${CONSUL_REMOTE_CONFIG_DIR}*"

Finally copy the configuration files into the remote VM.

$ scp -i certs/id_rsa ${OUTPUT_FOLDER}/gateway-api/* gateway-api:${CONSUL_REMOTE_CONFIG_DIR}

Start Consul on API GW

$ ssh -i certs/id_rsa gateway-api

Define the Consul configuration and data directories.

$ export CONSUL_CONFIG_DIR=/etc/consul.d/ \
export CONSUL_DATA_DIR=/opt/consul/

Ensure your user has write permission to the Consul data directory.

$ sudo chmod g+w ${CONSUL_DATA_DIR}

Finally, start the Consul server process.

$ consul agent -config-dir=${CONSUL_CONFIG_DIR} > /tmp/consul-client.log 2>&1 &

The process is started in background to not lock the terminal. Consul server log can be accessed in the /tmp/consul-client.log file.

Verify Consul API Gateway successfully joined the datacenter using the consul members command.

$ consul members

The output should show an extra node, named gateway-api among the datacenter members.

Node                Address          Status  Type    Build   Protocol  DC   Partition  Segment
consul-server-0     10.0.4.141:8301  alive   server  1.16.1  2         dc1  default    
gateway-api         10.0.4.142:8301  alive   client  1.16.1  2         dc1  default    
hashicups-api       10.0.4.140:8301  alive   client  1.16.1  2         dc1  default    
hashicups-db        10.0.4.165:8301  alive   client  1.16.1  2         dc1  default    
hashicups-frontend  10.0.4.168:8301  alive   client  1.16.1  2         dc1  default    
hashicups-nginx     10.0.4.33:8301   alive   client  1.16.1  2         dc1  default

Once the Consul agent is started, exit the ssh session to return to the bastion host.

$ exit

Generate API Gateway rules

Now that the Consul agent for the API Gateway successfully joined the datacenter, it is time to create the configuration for the Consul Data Plane.

Consul API Gateway is configured using Consul global configuration entries so you can configure it from a remote node. For this scenario, you will use the bastion host VM to generate, store, and apply the configuration.

To configure a Consul API Gateway you need two configuration entries:

An API Gateway configuration entry, defining the listeners that the gateway exposes externally and the certificates associated with them.
An inline certificate to make the certificate available to the gateway.

Note

Consul v1.19 introduces the file-system-certificate configuration entry to secure the Consul API Gateway on VMs. This configuration entry specifies the file path to a certificate and private key that must be present in the file system of the API gateway. The Consul server never sees the contents of these files. File system certificates require that you have access to the gateway's file system in order to place the certificate or update it.

Generate API Gateway configuration

The following API Gateway configuration entry includes listener configuration and a reference to a TLS certificate that the gateway exposes.

$ tee ${OUTPUT_FOLDER}config-gateway-api.hcl > /dev/null << EOF
Kind = "api-gateway"
Name = "gateway-api"

// Each listener configures a port which can be used to access the Consul cluster
Listeners = [
    {
        Port = 8443
        Name = "api-gw-listener"
        Protocol = "tcp"
        TLS = {
            Certificates = [
                {
                    Kind = "inline-certificate"
                    Name = "api-gw-certificate"
                }
            ]
        }
    }
]
EOF

Generate API Gateway certificate

The gateway configuration includes a reference to api-gw-certificate, a TLS certificate. You can create the certificate using an internal or public CA so your services can be compliant with your internal standards.

For this tutorial, you will use openssl to generate a valid certificate for the HashiCups application.

Define the certificate common name.

$ export COMMON_NAME="hashicups.hashicorp.com"

Create a configuration file for openssl.

$ tee ${OUTPUT_FOLDER}gateway-api-ca-config.cnf > /dev/null << EOF
[req]
default_bit = 4096
distinguished_name = req_distinguished_name
prompt = no

[req_distinguished_name]
countryName             = US
stateOrProvinceName     = California
localityName            = San Francisco
organizationName        = HashiCorp
commonName              = ${COMMON_NAME}
EOF

Generate a private key.

$ openssl genrsa -out ${OUTPUT_FOLDER}gateway-api-cert.key  4096 2>/dev/null

Create a certificate signing request.

$ openssl req -new \
  -key ${OUTPUT_FOLDER}gateway-api-cert.key \
  -out ${OUTPUT_FOLDER}gateway-api-cert.csr \
  -config ${OUTPUT_FOLDER}gateway-api-ca-config.cnf 2>/dev/null

Finally, sign the certificate and save it to a crt file.

$ openssl x509 -req -days 3650 \
  -in ${OUTPUT_FOLDER}gateway-api-cert.csr \
  -signkey ${OUTPUT_FOLDER}gateway-api-cert.key \
  -out ${OUTPUT_FOLDER}gateway-api-cert.crt 2>/dev/null

You can now generate the certificate configuration using the .key and .crt files.

First populate two environment variables with the files content.

$ export API_GW_KEY=`cat ${OUTPUT_FOLDER}gateway-api-cert.key`; \
  export API_GW_CERT=`cat ${OUTPUT_FOLDER}gateway-api-cert.crt`

Then populate the configuration file with the inline certificate and key.

$ tee ${OUTPUT_FOLDER}config-gateway-api-certificate.hcl > /dev/null << EOF
Kind = "inline-certificate"
Name = "api-gw-certificate"

Certificate = <<EOT
${API_GW_CERT}
EOT

PrivateKey = <<EOT
${API_GW_KEY}
EOT
EOF

Apply the configuration to Consul datacenter

You can now apply the configuration to Consul datacenter.

$ consul config write ${OUTPUT_FOLDER}config-gateway-api.hcl; \
  consul config write ${OUTPUT_FOLDER}config-gateway-api-certificate.hcl

Example output:

Config entry written: api-gateway/gateway-api
Config entry written: inline-certificate/api-gw-certificate

Start API gateway

Now that you have configured API Gateway, you can start the Envoy process that will serve external requests.

$ ssh -i certs/id_rsa gateway-api

Then, configure the token for the Envoy process.

$ export CONSUL_AGENT_TOKEN=`cat /etc/consul.d/agent-acl-tokens.hcl | grep agent | awk '{print $3}'| sed 's/"//g'`

Finally, start the Envoy process.

$ /usr/bin/consul connect envoy \
  -gateway api \
  -register \
  -service gateway-api \
  -token=${CONSUL_AGENT_TOKEN} \
  -envoy-binary /usr/bin/envoy > /tmp/api-gw-proxy.log 2>&1 &

Once the Envoy process is started, exit the ssh session to return to the bastion host and complete the configuration.

$ exit

Apply route

At this point, Consul API Gateway is ready to serve requests but no route is configured to expose services in the mesh externally.

For this tutorial, you will expose the HashiCups application using the hashicups-nginx service as entry point.

Generate API Gateway route

From the bastion host, create a route to redirect ingress traffic to the hashicups-nginx service.

$ tee ${OUTPUT_FOLDER}config-gateway-api-tcp-route.hcl > /dev/null << EOF
Kind = "tcp-route"
Name = "hashicups-tcp-route"

Services = [
  {
    Name = "hashicups-nginx"
  }
]

Parents = [
  {
    Kind = "api-gateway"
    Name = "gateway-api"
    SectionName = "api-gw-listener"
  }
]
EOF

Then, apply the configuration to Consul.

$ consul config write ${OUTPUT_FOLDER}config-gateway-api-tcp-route.hcl
Config entry written: tcp-route/hashicups-tcp-route

Create intention for service access

To allow access to your NGINX service that serves the HashiCups application, create an intention that allows traffic from gateway-api service to hashicups-nginx service.

First make sure the folder exists.

$ mkdir -p ${OUTPUT_FOLDER}global

Then create the intention configuration file.

$ tee ${OUTPUT_FOLDER}global/intention-nginx.hcl > /dev/null << EOF
Kind = "service-intentions"
Name = "hashicups-nginx"
Sources = [
  {
    Name   = "gateway-api"
    Action = "allow"
  }
]
EOF

After creating the configuration file for the intention, apply it.

$ consul config write ${OUTPUT_FOLDER}global/intention-nginx.hcl 
Config entry written: service-intentions/nginx

Verify HashiCups is now reachable using API Gateway

After applying the route, you are able to access the HashiCups application using the Consul API gateway address.

First, retrieve the API Gateway address.

For this scenario, you can get the API Gateway public IP directly from the bastion host.

$ echo "https://`cat /etc/hosts | grep gateway-api-public | awk '{print $1}'`:8443"

The output will be similar to the following:

https://35.87.44.101:8443

Then, open the address in a browser.

HashiCups UI

You now have exposed HashiCups using Consul API Gateway and the application is now TLS protected. Verify the certificate exposed by the application is the one you created earlier in this tutorial.

$ openssl s_client -connect \
    `cat /etc/hosts | grep gateway-api-public | awk '{print $1}'`:8443 </dev/null 2>/dev/null | \
    openssl x509 -inform pem -text

The output will be similar to the following:

Certificate:
    Data:
        Version: 1 (0x0)
        Serial Number:
            3a:eb:01:5b:72:8c:40:47:e3:8b:c5:49:a4:bb:2f:50:ff:97:3e:c8
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, ST = California, L = San Francisco, O = HashiCorp, CN = hashicups.hashicorp.com
        Validity
            Not Before: Jun  1 09:36:12 2023 GMT
            Not After : May 29 09:36:12 2033 GMT
        Subject: C = US, ST = California, L = San Francisco, O = HashiCorp, CN = hashicups.hashicorp.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (4096 bit)
                Modulus:
                    ## ...
                Exponent: 65537 (0x10001)
    Signature Algorithm: sha256WithRSAEncryption
         ## ...
-----BEGIN CERTIFICATE-----
## ...
-----END CERTIFICATE-----

The public access to your HashiCups application instance is now TLS secured.

Remove direct access to hashicups-nginx

At this point, the HashiCups application is still accessible using the old, insecure, endpoint.

The last step is to ensure hashicups-nginx service will only serve local requests. You can achieve this using iptables.

$ ssh -i certs/id_rsa hashicups-nginx "bash -c \
    'sudo iptables -A INPUT -i lo -p tcp --dport 80 -j ACCEPT; \
     sudo iptables -v -A INPUT -p tcp -s 0/0 --dport 80 -j REJECT'"

The insecure endpoint is not available anymore. You can only access your application securely through the Consul API Gateway.

Next steps

With this tutorial, you completed an important step towards the Zero Trust security model for a VM based infrastructure. You started from a scenario where the HashiCups application was deployed with no security features and introduced Consul service mesh gradually to the scenario.

At this moment:

the HashiCups application is accessible only using the Consul API Gateway
the internal communication across the different services that compose the application is fully secured
you can expose a custom certificate for the externally exposed services and rotate it using Consul

(Optional) Destroy infrastructure

If you want to stop at this tutorial, you can destroy the infrastructure now.

From the ./self-managed/infrastruture/aws folder of the repository, use terraform to destroy the infrastructure.

$ terraform destroy --auto-approve

In the next tutorial, you will learn how to monitor the services in your Consul service mesh using the Grafana suite.

For more information about the topics covered in this tutorial, refer to the following resources:

If you want to learn more about the File system certificate, introduced by Consul 1.9, refer to File system certificate configuration reference.

Securely connect your services

Observe service mesh traffic