HashiConf 2024 Now streaming live from Boston! Attend for free
Consul
Connect services

Monitor your application health with distributed checks

  • 23min
  • |
  • Consul
  • Interactive

Workflow

In modern distributed systems, health checks play a significant role in the maintenance of complex deployments. Health checks provide a way to monitor and verify the status of your service fleet and can be used to automatically react to failures or excessive load. Consul uses distributed health checks to maintain an updated catalog of all services and nodes registered within the datacenter and to periodically verify services' behavior.

The Consul DNS service provides a standard interface to query the Consul catalog. Using Consul FQDN for service configuration, instead of IPs, you can simplify application configuration. One further simplification provided by Consul is the load-balancing feature provided across multiple instances of the same service. This is completely transparent to the downstream service instances and does not require extra configuration. Consul DNS service prevents unhealthy service instances to be resolved and simplifies the configuration process for your applications that do not have to change configuration in case of service failure.

Consul DNS service is based on the service definition files that the different Consul agents provide and uses a distributed health check system to ensure reliability of DNS results.

In this tutorial you will learn how to write a service definition to register your service in Consul and how to create health checks to make sure your service is available only when fully functional.

Specifically, you will:

  • Review the ACL permissions required to register a Consul service
  • Generate a token for a Consul service using the service-identity policy
  • Create a service configuration for the hashicups-api service
  • Reload Consul configuration to include the newly configured service
  • Verify service registration
  • Verify the load-balancing feature provided by Consul DNS

Tutorial scenario

This tutorial uses HashiCups, a demo coffee shop application made up of several microservices running on VMs.

At the beginning of the tutorial, you have a fully deployed Consul datacenter with an instance of the HashiCups application, composed by four services, NGINX, Frontend, API, and Database, deployed and registered in Consul catalog.

In the Consul datacenter, there is an extra node that you want to use to host a second instance of the API service for HashiCups, that you want to include in the datacenter.

Architecture Initial scenario Architecture Initial scenario

By the end of this tutorial, you will have configured the API service to be part of Consul catalog.

This tutorial will focus specifically on service and health checks definitions to register a service in Consul catalog.

Architecture Final scenario Architecture Final scenario

Once the service is registered in Consul, it will automatically appear in Consul DNS results and Consul will automatically load balance traffic across the different instances of the service.

All operations in this tutorial, after the scenario deployment, will be performed from an additional node, a bastion host, that will be deployed as part of the scenario. The tutorial provides instructions on how to connect to the bastion host at the end of the deployment.

Prerequisites

This tutorial assumes you are already familiar with Consul service discovery and its core functionalities. If you are new to Consul refer to refer to the Consul Getting Started tutorials collection.

This tutorial uses an interactive lab to guide you through the process of setting up a Consul datacenter on VM workloads. The lab environment includes all required binaries and sample configurations.

Launch Terminal

This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.

Once the lab is loaded, Open the Bastion Host tab to login to the bastion host.

Configure CLI to interact with Consul

Configure your bastion host to communicate with your Consul environment using the two dynamically generated environment variable files.

$ source "/home/admin/assets/scenario/env-scenario.env" && \
  source "/home/admin/assets/scenario/env-consul.env"

That will produce no output.

After loading the needed variables, verify you can connect to your Consul datacenter.

$ consul members

That will produce an output similar to the following.

Node                  Address          Status  Type    Build   Protocol  DC   Partition  Segment
consul-server-0       172.22.0.8:8301  alive   server  1.17.1  2         dc1  default    <all>
hashicups-api-0       172.22.0.2:8301  alive   client  1.17.1  2         dc1  default    <default>
hashicups-api-1       172.22.0.3:8301  alive   client  1.17.1  2         dc1  default    <default>
hashicups-db-0        172.22.0.4:8301  alive   client  1.17.1  2         dc1  default    <default>
hashicups-frontend-0  172.22.0.5:8301  alive   client  1.17.1  2         dc1  default    <default>
hashicups-nginx-0     172.22.0.6:8301  alive   client  1.17.1  2         dc1  default    <default>

Use consul catalog services to query Consul catalog for the services present in the datacenter. 

$ consul catalog services -tags
consul                  
hashicups-api           inst_0
hashicups-db            inst_0
hashicups-frontend      inst_0
hashicups-nginx         inst_0

Every service for the HashiCups application is tagged with the inst_0 tag to indicate they are the first instance of the service.

Create ACL token for service registration

When ACLs are enabled, service registration requires a token with write permission over the service name you want to register. In this case service name is hashicups-api.

Service identities let you to quickly construct policies for services, rather than manually creating identical polices for each service instance and for their companion sidecar proxy. Generate a new token for hashicups-api-1 using service-identity.

$ consul acl token create \
  -description="SVC HashiCups API token" \
  --format json \
  -service-identity="hashicups-api" | tee /home/admin/assets/scenario/conf/secrets/acl-token-svc-hashicups-api-1.json

That will produce an output similar to the following.

{
  "CreateIndex": 97,
    "ModifyIndex": 97,
    "AccessorID": "cdf3909b-16cf-a5ea-bcb8-3b3b19fe0bd0",
    "SecretID": "925cee23-8b47-5ef2-2f20-1fb40cdcbf0d",
    "Description": "SVC HashiCups API token",
    "ServiceIdentities": [
      {
        "ServiceName": "hashicups-api"
        }
    ],
    "Local": false,
    "CreateTime": "2024-01-17T13:43:09.166273172Z",
    "Hash": "RtpgRlE69uEOWjeuaBUG2ezUM37J4OxeUhFaaRJaEac="
}

Set your newly generated token as the CONSUL_AGENT_TOKEN environment variable. You will use this variable later in the tutorial to generate the service definition file. 

$ export CONSUL_AGENT_TOKEN=`cat /home/admin/assets/scenario/conf/secrets/acl-token-svc-hashicups-api-1.json | jq -r ".SecretID"`

Review service configuration for Consul services

Consul allows multiple ways to register a service in the service catalog:

  1. Creating a service definition file in the agent's configuration directory
  2. Using the consul services register CLI command
  3. Calling the /agent/service/register HTTP API endpoint

Each of these methods will persist the registration in the Consul data folder.

To make sure your service registration survives data folder corruption, in this tutorial you will configure a service by placing a service definition file inside the agent's configuration directory.

Each service configuration is composed by two parts, service definitions and health checks definitions.

Service definition

The service definition requires the following parameters to be defined:

  • name - the service name to register. Multiple instances of the same service share the same service name.
  • id - the service id. Multiple instances of the same service require a unambiguous id to be registered.
  • tags - [optional] - tags to assign to the service instance. Useful for blue-green deployment, canary deployment or to identify services inside Consul datacenter.
  • port - the port your service is exposing.
  • token - a token to be used during service registration. This is the token you created in the previous section.

Below an example configuration for the service definition.

service {
  name = "hashicups-api"
  id = "hashicups-api-1"
  tags = [ "inst_1" ]
  port = 8081
  token = "925cee23-8b47-5ef2-2f20-1fb40cdcbf0d"
}

Read more on service definitions at Services configuration reference

Checks definition.

Consul provides distributed monitoring for your services when health checks for the service are configured. Health checks configurations are nested in the service block. They can be defined using the following parameters:

  • id - unique string value that specifies an ID for the check.
  • name - required string value that specifies the name of the check.
  • service_id - specifies the ID of a service instance to associate with a service check.
  • interval - specifies how frequently to run the check.
  • timeout - specifies how long unsuccessful requests take to end with a timeout.

The other parameter required to define the check is the type. Consul supports multiple check types, but for this tutorial you will use the TCP and HTTP check types.

A tcp check establishes connections to the specified IPs or hosts. If the check successfully establishes a connection, the service status is reported as success. If the IP or host does not accept the connection, the service status is reported as critical.

An example of tcp check for the hashicups-api service, listening on port 8081 is the following:

{
  id =  "check-hashicups-api.public",
  name = "hashicups-api.public status check",
  service_id = "hashicups-api-1",
  tcp  = "localhost:8081",
  interval = "5s",
  timeout = "5s"
}

HTTP checks send an HTTP request to the URL specified in the definition and report the service health based on the HTTP response code.

An example of tcp check for the hashicups-api service, which exposes an health endpoint to test service status, is the following:

{
  id =  "check-hashicups-api.public.http",
  name = "hashicups-api.public  HTTP status check",
  service_id = "hashicups-api-1",
  http  = "http://localhost:8081/health",
  interval = "5s",
  timeout = "5s"
}

Create service configuration for HashiCups API service

The HashiCups API service is composed by three different services, public-api, product-api, and payments, listening respectively on port 8081, 9090, and 8080. The only service that requires external access is the public-api on port 8081.

Create service configuration file.

$ tee /home/admin/assets/scenario/conf/hashicups-api-1/svc-hashicups-api.hcl > /dev/null << EOF
## -----------------------------
## svc-hashicups-api.hcl
## -----------------------------
service {
  # Name of the service as it will appear in Consul catalog
  name = "hashicups-api"

  # Unambiguous ID, to distinguish the service instance from other instances of the same service
  id = "hashicups-api-1"
  
  # Tags for the service instance. A service instance can be reched using the 
  # <tag>.<service>.service.<datacenter>.<domain> FQDN
  tags = [ "inst_1" ]
  
  # Port number for the service
  port = 8081
  
  # ACL token to present when registering the service 
  token = "${CONSUL_AGENT_TOKEN}"

  checks =[
  {
  id =  "check-hashicups-api.public.http",
  name = "hashicups-api.public  HTTP status check",
  service_id = "hashicups-api-1",
  http  = "http://localhost:8081/health",
  interval = "5s",
  timeout = "5s"
  },
  {
    id =  "check-hashicups-api.public",
    name = "hashicups-api.public status check",
    service_id = "hashicups-api-1",
    tcp  = "localhost:8081",
    interval = "5s",
    timeout = "5s"
  },
  {
    id =  "check-hashicups-api.product",
    name = "hashicups-api.product status check",
    service_id = "hashicups-api-1",
    tcp  = "localhost:9090",
    interval = "5s",
    timeout = "5s"
  },
  {
    id =  "check-hashicups-api.payments",
    name = "hashicups-api.payments status check",
    service_id = "hashicups-api-1",
    tcp  = "localhost:8080",
    interval = "5s",
    timeout = "5s"
  }]
}
EOF

Tip

To simplify tests, all services already present in the Consul catalog have a tag assigned, inst_0. The new instance uses a different tag, inst_1.

Note

The service definition file can be adapted for Consul service mesh by adding a `connect` section in the file. Learn more on this on Securely connect your services with Consul service mesh

Copy the configuration file on the hashicups-api-1 node.

$ scp -r -i /home/admin/certs/id_rsa /home/admin/assets/scenario/conf/hashicups-api-1/svc-hashicups-api.hcl admin@hashicups-api-1:/etc/consul.d/svc-hashicups-api.hcl
svc-hashicups-api.hcl

Open the API-1 tab to login to hashicups-api-1.

Verify the VM contains the files required for Consul configuration.

$ cat /etc/consul.d/svc-hashicups-api.hcl

That will produce an output similar to the following.

## -----------------------------
## svc-hashicups-api.hcl
## -----------------------------
service {
  # Name of the service as it will appear in Consul catalog
  name = "hashicups-api"

  # Unambiguous ID, to distinguish the service instance from other instances of the same service
  id = "hashicups-api-1"
  
  # Tags for the service instance. A service instance can be reched using the 
  # <tag>.<service>.service.<datacenter>.<domain> FQDN
  tags = [ "inst_1" ]
  
  # Port number for the service
  port = 8081
  
  # ACL token to present when registering the service 
  token = "925cee23-8b47-5ef2-2f20-1fb40cdcbf0d"

  checks =[
  {
  id =  "check-hashicups-api.public.http",
  name = "hashicups-api.public  HTTP status check",
  service_id = "hashicups-api-1",
  http  = "http://localhost:8081/health",
  interval = "5s",
  timeout = "5s"
  },
  {
    id =  "check-hashicups-api.public",
    name = "hashicups-api.public status check",
    service_id = "hashicups-api-1",
    tcp  = "localhost:8081",
    interval = "5s",
    timeout = "5s"
  },
  {
    id =  "check-hashicups-api.product",
    name = "hashicups-api.product status check",
    service_id = "hashicups-api-1",
    tcp  = "localhost:9090",
    interval = "5s",
    timeout = "5s"
  },
  {
    id =  "check-hashicups-api.payments",
    name = "hashicups-api.payments status check",
    service_id = "hashicups-api-1",
    tcp  = "localhost:8080",
    interval = "5s",
    timeout = "5s"
  }]
}

Reload Consul to apply the service configuration.

$ consul reload
Configuration reload triggered

Start hashicups-api service using Consul FQDN for upstream services.

$ ~/start_service.sh start --consul

That will show an output similar to the following.

Stop pre-existing instances.
START - Start services on all interfaces.
START CONSUL - Starts the service using Consul service name for upstream services (using LB functionality).
Start service instance.
Service started to listen on all available interfaces.
{
        "db_connection": "host=hashicups-db.service.dc1.consul port=5432 user=postgres password=p05tgr35 dbname=products sslmode=disable",
        "bind_address": ":9090",
        "metrics_address": ":9103"
}
Starting payments application
Starting Product API
Starting Public API

To continue with the tutorial, open the Bastion Host tab to return to the bastion host.

Verify service registration

The service will now start appearing among Consul DNS results when a query is performed. Use the inst_1 tag in the results to recognize the new instance or in the query to filter the DNS results. Consul DNS service can be queried using all interfaces provided by Consul.

API interface and UI can also be used to retrieve information about the health checks configured for the service.

Use the consul catalog command to query the Consul catalog and show all available services.

$ consul catalog services -tags
consul                  
hashicups-api           inst_0,inst_1
hashicups-db            inst_0
hashicups-frontend      inst_0
hashicups-nginx         inst_0

Notice that the hashicups-api instance presents two different tags, inst_0 and inst_1, that indicate that both instances are registered within Consul.

Verify Consul load balancing functionalities

When multiple instances of a service are defined, Consul DNS will automatically provide basic round-robin load balancing capabilities.

To test it, you will make 100 requests to the Consul DNS service asking to resolve the hashicups-api.service.dc1.consul name and count the different results. 

$ for i in `seq 1 100` ; do dig @consul-server-0  hashicups-api.service.dc1.consul +short | head -1; done | sort | uniq -c

That will produce an output similar to the following.

     54 172.22.0.2
     46 172.22.0.3

Notice that the request was balanced across the two different instances of the hashicups-api service.

Next steps

In this tutorial you learned how to register a service in a Consul datacenter by writing a service definition with associated health checks. You created a token with the necessary permissions to register the service, generated a definition for the hashicups-api service including health checks for all the different components of the service, and registered the service in Consul catalog by reloading the Consul agent on the hashicups-api-1 node. You then used all Consul interfaces, CLI, API, DNS, and UI, to verify the service got registered correctly and learned how to identify the different instances of the service using the inst_1 tag. Finally, you tested Consul DNS load-balancing capabilities by performing multiple queries for the same service and verifying the number of request was roughly even across the two instances.

For more information about the topics covered in this tutorial, refer to the following resources:

To learn more about other features provided by Consul DNS service to:

Collection Overview

Connect services

 Next

Monitor external services with Consul ESM

This tutorial also appears in:

  • 1 tutorial
    Service Discovery and Health Monitoring
    Register services with Consul DNS, including services on external nodes and with Terraform.
    • Consul

Workflow