Manage permissions with access control lists (ACLs) on VMs

21min
|
Consul
Interactive

Workflow

Consul service mesh provides secure communication across your services within and across your infrastructure, including on-premises and cloud environments. In a service mesh scenario, you configure the single service instances to listen on the loopback interface and, using the upstreams in the service definition, Envoy sidecar proxies permit communication with other services by adding listeners locally. These communication are protected via an mTLS layer that prevents unauthorized services, inside or outside the mesh, to access them.

Consul provides one extra layer of security, Access Control Lists (ACLs), to prevent malicious actors to register services and hijack communications in your datacenter. A Consul datacenter with ACLs enabled and with default-deny policy, will require a valid token for every operation, including service registration.

In this tutorial you will learn how to create credentials required to join an existing Consul datacenter and register a service to the Consul catalog. After an introductory section about the different tokens available for Consul agents' configuration, you will generate tokens for the Consul node using the node-identiy template policy, and for a Consul service using the service-identity policy. You will also learn how to reuse an existing policy to generate a token for the Consul DNS interface, to allow service resolution locally.

You will use the generated credentials to create a service configuration for the API service, and to add a new Consul agent to Consul service mesh. After the agent joins the Consul service mesh, you will verify the service is correctly registered in the mesh using all the interfaces provided by Consul: CLI, API, DNS, and web UI.

Tutorial scenario

This tutorial uses HashiCups, a demo coffee shop application made up of several microservices running on VMs.

At the beginning of the tutorial, you will deploy Consul service mesh, with Envoy sidecar proxies running alongside each service and an API Gateway configured to permit access to the NGINX service.

In addition, you will deploy an extra node that hosts another HashiCups API instance that you want to add to your datacenter.

Architecture Initial scenario

In this tutorial, you will use ACLs to add the API service to the Consul service mesh.

Architecture Final scenario

Once you have registered the service in Consul, it will automatically appear in Consul DNS results and Consul will automatically load balance traffic across the different instances of the service.

Note

All operations in this tutorial, after the scenario deployment, will be performed from an additional node, a bastion host, that will be deployed as part of the scenario. The tutorial provides instructions on how to connect to the bastion host at the end of the deployment.

Prerequisites

This tutorial assumes you are already familiar with Consul service mesh and its core functionalities. If you are new to Consul refer to the Consul Getting Started tutorials collection.

This tutorial uses an interactive lab to guide you through the process of setting up a service mesh on your VM workloads. The lab environment includes all required binaries and sample configurations.

Launch Terminal

This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.

Once the lab is loaded, open the Bastion Host tab to login to the bastion host.

Configure CLI to interact with Consul

Configure your bastion host to communicate with your Consul environment using the two, dynamically generated, environment variable files.

$ source "/home/admin/assets/scenario/env-scenario.env" && \
  source "/home/admin/assets/scenario/env-consul.env"

After loading the needed variables, verify you can connect to your Consul datacenter.

$ consul members
Node                  Address             Status  Type    Build   Protocol  DC   Partition  Segment
consul-server-0       192.168.80.7:8301   alive   server  1.17.0  2         dc1  default    <all>
gateway-api-0         192.168.80.4:8301   alive   client  1.17.0  2         dc1  default    <default>
hashicups-api-0       192.168.80.8:8301   alive   client  1.17.0  2         dc1  default    <default>
hashicups-db-0        192.168.80.2:8301   alive   client  1.17.0  2         dc1  default    <default>
hashicups-frontend-0  192.168.80.12:8301  alive   client  1.17.0  2         dc1  default    <default>
hashicups-nginx-0     192.168.80.11:8301  alive   client  1.17.0  2         dc1  default    <default>

Notice that bastion and hashicups-api-1 are not part of the datacenter.

Create ACL token for ACL management

A management token is required to manage Consul's ACLs system. We recommend creating a new Consul management token for this purpose instead of using the ACL bootstrap token.

$ consul acl token create \
  -description="SVC HashiCups API token" \
  --format json \
  -policy-name="global-management" | tee /home/admin/assets/scenario/conf/secrets/acl-token-management.json

That will produce an output similar to the following.

{
    "CreateIndex": 115,
    "ModifyIndex": 115,
    "AccessorID": "9cdf707d-8847-9bb2-606c-d4f0849692d6",
    "SecretID": "e4d09a88-f3dc-0191-7ce6-f7a35a7097ae",
    "Description": "SVC HashiCups API token",
    "Policies": [
        {
            "ID": "00000000-0000-0000-0000-000000000001",
            "Name": "global-management"
        }
    ],
    "Local": false,
    "CreateTime": "2023-11-18T13:27:42.735373628Z",
    "Hash": "UfTdHatx/OxAw8z+/8suSKQrY6MgpC3CutkuZZAzPC4="
}

Set your newly generated management token as the CONSUL_HTTP_TOKEN environment variable. This will allow you to perform ACL system operations in the next steps using the new token.

$ export CONSUL_HTTP_TOKEN=`cat /home/admin/assets/scenario/conf/secrets/acl-token-management.json | jq -r ".SecretID"`

Create ACL configuration for Consul agent

Consul provides a fine grained set of permissions to manage your Consul nodes and services. In this section you will create the ACL tokens required to register a new Consul client node in a Consul datacenter. You will then register a service in the Consul service mesh with these ACL tokens.

The initial tutorial environment includes an empty VM named hashicups-api-1 for you. You will create ACLs token to register this Consul client node to the Consul datacenter, and register the service in Consul.

Consul has a section in the configuration to specify all the ACL tokens to be used by the agent.

...
acl {
  tokens {
    agent  = "<Consul agent token>"
    default  = "<Consul default token>"
    config_file_service_registration = "<Consul service registration token>"
  }
}
...

The different tokens are used for different operation performed by the Consul agent:

agent - Used for clients and servers to perform internal operations. This token must have write access to the node name it will register as in order to set any of the node-level information in the catalog.
default - Used for the default operations when agent token is not provided and for the endpoints that do not permit a token being passed. This will be used for the DNS interface of the Consul agent.
config_file_service_registration - Specifies the ACL token the agent uses to register services and checks. This token needs write permission to register all services and checks defined in this agent's configuration.

You will generate the ACL configuration section after these three tokens are generated.

Generate `default` token

Each token must be associated with a policy providing the desired permissions. The scenario already contains a policy used to generate the default token for the other nodes that is already suitable for the node default token. The policy is named acl-policy-dns.

Read the policy using the consul acl policy command.

$ consul acl policy read -name acl-policy-dns
ID:           f77b2144-190f-00ee-46b5-cb769d018e8e
Name:         acl-policy-dns
Description:  Policy for DNS endpoints
Datacenters:  
Rules:
# -----------------------------+
# acl-policy-dns.hcl           |
# -----------------------------+

node_prefix "" {
  policy = "read"
}
service_prefix "" {
  policy = "read"
}
# only needed if using prepared queries
query_prefix "" {
  policy = "read"
}
# needed for prometheus metrics
agent_prefix ""
{
  policy = "read"
}

The acl-policy-dns policy, used to generate the default token, grants read access to all services and nodes so DNS query can work. It also grants read permissions for prepared queries and agent metrics endpoints.

Generate a new token using the acl-policy-dns policy.

$ consul acl token create \
  -description="hashicups-api-1 default token" \
  --format json \
  -policy-name="acl-policy-dns" | tee /home/admin/assets/scenario/conf/secrets/acl-token-default-hashicups-api-1.json

That will produce an output similar to the following.

{
    "CreateIndex": 116,
    "ModifyIndex": 116,
    "AccessorID": "ba4533ad-6e44-7279-41d3-08bca235306f",
    "SecretID": "391bc1b8-2837-db2c-6466-613e16051050",
    "Description": "hashicups-api-1 default token",
    "Policies": [
        {
            "ID": "f77b2144-190f-00ee-46b5-cb769d018e8e",
            "Name": "acl-policy-dns"
        }
    ],
    "Local": false,
    "CreateTime": "2023-11-18T13:27:42.871333836Z",
    "Hash": "xPN3IX5yLnk7SDzHYXIBCPKpQtAWGP/VoLfZBmiK9kA="
}

Generate `agent` token

In order for the agent to set any node-level information in Consul catalog, the agent token must have write access to the node name it will register as. One minimal example for such a policy is the following.

# Allow the agent to register its own node in the Catalog and update its network coordinates
node "<node name>" {
  policy = "write"
}

# Allows the agent to detect and diff services registered to itself. This is used during
# anti-entropy to reconcile difference between the agents knowledge of registered
# services and checks in comparison with what is known in the Catalog.
service_prefix "" {
  policy = "read"
}

You can use node-identity to help you generate agent tokens for Consul nodes. Node identities let you to quickly construct policies for nodes, rather than manually creating identical polices for each node. Generate a new token for hashicups-api-1 using node-identity.

$ consul acl token create \
  -description="hashicups-api-1 agent token" \
  --format json \
  -node-identity="hashicups-api-1:dc1" | tee /home/admin/assets/scenario/conf/secrets/acl-token-agent-hashicups-api-1.json

That will produce an output similar to the following.

{
    "CreateIndex": 117,
    "ModifyIndex": 117,
    "AccessorID": "715cad9a-ebc5-21cd-0919-ac08effdbf3d",
    "SecretID": "3c470114-1214-d4fe-e3c5-3dd1f9515373",
    "Description": "hashicups-api-1 agent token",
    "NodeIdentities": [
        {
            "NodeName": "hashicups-api-1",
            "Datacenter": "dc1"
        }
    ],
    "Local": false,
    "CreateTime": "2023-11-18T13:27:42.918717795Z",
    "Hash": "dn0S8locXJeqcWcmqHURzQDKSO4UZKwV49QO0YCCbv8="
}

Generate `config_file_service_registration` token

The config_file_service_registration token needs write permission to register all services and checks defined in this agent's configuration. For Consul service mesh, the token also needs permissions to register the Envoy sidecar proxy for the service as an independent service in Consul catalog. One minimal example for such a policy is the following.

# Allow the service and its sidecar proxy to register into the catalog.
service "<service name>" {
    policy = "write"
}
service "<service name>-sidecar-proxy" {
    policy = "write"
}

# Allow for any potential upstreams to be resolved.
service_prefix "" {
    policy = "read"
}
node_prefix "" {
    policy = "read"
}

You can use service-identity to help you generate config_file_service_registration tokens for Consul nodes. Service identities let you to quickly construct policies for services, rather than manually creating identical polices for each service instance and for their companion sidecar proxy. Generate a new token for hashicups-api-1 using service-identity.

$ consul acl token create \
  -description="hashicups-api-1 config_file_service_registration token" \
  --format json \
  -service-identity="hashicups-api:dc1" | tee /home/admin/assets/scenario/conf/secrets/acl-token-svc-hashicups-api-1.json

That will produce an output similar to the following.

{
    "CreateIndex": 118,
    "ModifyIndex": 118,
    "AccessorID": "8cf68272-25ce-0d87-8dec-713890e8a0ba",
    "SecretID": "75906ad5-2162-efcb-cf95-63a2b6396840",
    "Description": "hashicups-api-1 config_file_service_registration token",
    "ServiceIdentities": [
        {
            "ServiceName": "hashicups-api",
            "Datacenters": [
                "dc1"
            ]
        }
    ],
    "Local": false,
    "CreateTime": "2023-11-18T13:27:42.963791211Z",
    "Hash": "WCtk+YLUNIm5VR0zLykM9oo5TG4xdBbwAIBb3fSZ2qA="
}

Verify you have created the three tokens and the files containing the tokens are present in the assets/ folder.

$ ls ~/assets/scenario/conf/secrets/ | grep hashicups-api-1
acl-token-agent-hashicups-api-1.json
acl-token-default-hashicups-api-1.json
acl-token-svc-hashicups-api-1.json

Generate Consul agent ACL configuration section

With the three tokens you created, generate the Consul agent ACL configuration.

$ tee /home/admin/assets/scenario/conf/hashicups-api-1/agent-acl-tokens.hcl > /dev/null << EOF
acl {
  tokens {
    agent  = "`cat /home/admin/assets/scenario/conf/secrets/acl-token-agent-hashicups-api-1.json | jq -r ".SecretID"`"
    default  = "`cat /home/admin/assets/scenario/conf/secrets/acl-token-default-hashicups-api-1.json | jq -r ".SecretID"`"
    config_file_service_registration = "`cat /home/admin/assets/scenario/conf/secrets/acl-token-svc-hashicups-api-1.json | jq -r ".SecretID"`"
  }
}
EOF

Generate hashicups-api service configuration

Generate hashicups-api service configuration.

$ tee /home/admin/assets/scenario/conf/hashicups-api-1/svc-hashicups-api.hcl > /dev/null << EOF
## -----------------------------
## svc-hashicups-api.hcl
## -----------------------------
service {
  name = "hashicups-api"
  id = "hashicups-api-1"
  tags = [ "inst_1" ]
  port = 8081

  token = "`cat /home/admin/assets/scenario/conf/secrets/acl-token-svc-hashicups-api-1.json | jq -r ".SecretID"`"

  connect {
    sidecar_service {
        proxy {
          upstreams = [
            {
              destination_name = "hashicups-db"
              local_bind_port = 5432
            }
          ]
        }
     }
  }

  checks =[
  {
  id =  "check-hashicups-api.public.http",
  name = "hashicups-api.public  HTTP status check",
  service_id = "hashicups-api-1",
  http  = "http://localhost:8081/health",
  interval = "5s",
  timeout = "5s"
  },
  {
    id =  "check-hashicups-api.public",
    name = "hashicups-api.public status check",
    service_id = "hashicups-api-1",
    tcp  = "localhost:8081",
    interval = "5s",
    timeout = "5s"
  },
  {
    id =  "check-hashicups-api.product",
    name = "hashicups-api.product status check",
    service_id = "hashicups-api-1",
    tcp  = "localhost:9090",
    interval = "5s",
    timeout = "5s"
  },
  {
    id =  "check-hashicups-api.payments",
    name = "hashicups-api.payments status check",
    service_id = "hashicups-api-1",
    tcp  = "localhost:8080",
    interval = "5s",
    timeout = "5s"
  }]
}
EOF

The tutorial shows an example of how to configure a service in Consul service mesh. The same concepts apply to a Consul service discovery. If you are trying to register a new service for service discovery, remove the connect section in the service configuration file.

Add new service to Consul service mesh with ACLs

In the previous paragraphs, you created agent-acl-tokens.hcl, containing token configuration for the Consul agent, and svc-hashicups-api.hcl, containing service configuration for the hashicups-api service instance. The bastion host already contains the remaining configuration needed to configure Consul on the hashicups-api-1 node. Review the full configuration.

$ ls /home/admin/assets/scenario/conf/hashicups-api-1
agent-acl-tokens.hcl
agent-gossip-encryption.hcl
consul-agent-ca.pem
consul.hcl
svc-hashicups-api.hcl

Copy the configuration on the remote node.

$ scp -r -i /home/admin/certs/id_rsa /home/admin/assets/scenario/conf/hashicups-api-1/* admin@hashicups-api-1:/etc/consul.d/
agent-acl-tokens.hcl                  ...
agent-gossip-encryption.hcl           ...
consul-agent-ca.pem                   ...
consul.hcl                            ...
svc-hashicups-api.hcl                 ...

Start Consul process

Open the API-1 tab to login to hashicups-api-1.

Verify the VM contains the files required for Consul configuration.

$ ls /etc/consul.d
agent-acl-tokens.hcl
agent-gossip-encryption.hcl
consul-agent-ca.pem
consul.hcl
svc-hashicups-api.hcl

Start Consul agent.

$ /usr/bin/consul agent \
  -retry-join=consul \
  -log-file=/tmp/consul-client \
  -config-dir=/etc/consul.d > /tmp/consul-client.log 2>&1 &

The command starts the Consul server in the background to not lock the terminal. You can access the Consul server log through the /tmp/consul-server.log file.

Start Envoy sidecar proxy

To make the API service instance join Consul service mesh, your node requires an Envoy proxy acting as a sidecar for the service. You need a valid token with the permission to register the sidecar as a service in Consul catalog to start the proxy process.

Use the token created previously using the service-identity policy.

$ export _agent_token=`cat /etc/consul.d/agent-acl-tokens.hcl | grep -Po "(?<=registration = \")[^\"]+(?=\")"` && echo ${_agent_token}

That will produce an output similar to the following.

75906ad5-2162-efcb-cf95-63a2b6396840

Start Envoy sidecar proxy for hashicups-api service.

$ /usr/bin/consul connect envoy \
  -token=${_agent_token} \
  -envoy-binary /usr/bin/envoy \
  -sidecar-for hashicups-api-1 > /tmp/sidecar-proxy.log 2>&1 &

The command starts the Envoy sidecar proxy in the background to not lock the terminal. You can access the Envoy log through the /tmp/sidecar-proxy.log file.

Start the API service instance

Once the Consul node joined the Consul datacenter and the Envoy sidecar is configured and ready to route traffic to and from your service, start the hashicups-api service to listen on the local network interface.

$ ~/start_service.sh local

That will show an output similar to the following.

Starting service on local interface.
{
    "db_connection": "host=localhost port=5432 user=postgres password=p05tgr35 dbname=products sslmode=disable",
    "bind_address": "localhost:9090",
    "metrics_address": "localhost:9103"
}
Starting payments application
Starting Product API
Starting Public API

To continue with the tutorial, open the Bastion Host tab to return to the bastion host.

Verify service registration

After you registered the service, it will start appearing among the results when Consul catalog is queried.

Tip

To simplify tests all services already present in the Consul service mesh had a tag assigned, inst_0 while the new instance got registered using a different tag, inst_1.

Verify it using the different interfaces exposed by Consul.

Use the consul catalog command to query the Consul catalog and show all available services.

$ consul catalog services -tags

The command will produce an output similar to the following.

consul                                
gateway-api                           
hashicups-api                         inst_0,inst_1
hashicups-api-sidecar-proxy           inst_0,inst_1
hashicups-db                          inst_0
hashicups-db-sidecar-proxy            inst_0
hashicups-frontend                    inst_0
hashicups-frontend-sidecar-proxy      inst_0
hashicups-nginx                       inst_0
hashicups-nginx-sidecar-proxy         inst_0

Notice that the hashicups-api and hashicups-api-sidecar-proxy instances present two different tags, inst_0 and inst_1, that indicate that both instances are registered within Consul.

Use the /v1/catalog/service endpoint to query the Consul catalog and show all instances for a specific service.

$ curl --silent \
   --header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
   --connect-to server.${CONSUL_DATACENTER}.${CONSUL_DOMAIN}:8443:consul-server-0:8443 \
   --cacert ${CONSUL_CACERT} \
   https://server.${CONSUL_DATACENTER}.${CONSUL_DOMAIN}:8443/v1/catalog/service/hashicups-api | jq -r .

That will produce an output similar to the following.

[
  {
    "ID": "94899a63-3f43-4105-1adb-60ecde44f167",
    "Node": "hashicups-api-0",
    "Address": "192.168.80.8",
    "Datacenter": "dc1",
    "TaggedAddresses": {
        "lan": "192.168.80.8",
    "lan_ipv4": "192.168.80.8",
    "wan": "192.168.80.8",
    "wan_ipv4": "192.168.80.8"
    },
    "NodeMeta": {
        "consul-network-segment": "",
    "consul-version": "1.17.0"
    },
    "ServiceKind": "",
    "ServiceID": "hashicups-api-0",
    "ServiceName": "hashicups-api",
    "ServiceTags": [
        "inst_0"
    ],
    "ServiceAddress": "",
    "ServiceWeights": {
        "Passing": 1,
    "Warning": 1
    },
    "ServiceMeta": {},
    "ServicePort": 8081,
    "ServiceSocketPath": "",
    "ServiceEnableTagOverride": false,
    "ServiceProxy": {
        "Mode": "",
    "MeshGateway": {},
    "Expose": {}
    },
    "ServiceConnect": {},
    "ServiceLocality": null,
    "CreateIndex": 73,
    "ModifyIndex": 73
  },
  {
    "ID": "e8b2907a-bfca-4dc5-75d1-50ddb0faacbd",
    "Node": "hashicups-api-1",
    "Address": "192.168.80.6",
    "Datacenter": "dc1",
    "TaggedAddresses": {
        "lan": "192.168.80.6",
      "lan_ipv4": "192.168.80.6",
      "wan": "192.168.80.6",
      "wan_ipv4": "192.168.80.6"
    },
    "NodeMeta": {
        "consul-network-segment": "",
      "consul-version": "1.17.0"
    },
    "ServiceKind": "",
    "ServiceID": "hashicups-api-1",
    "ServiceName": "hashicups-api",
    "ServiceTags": [
        "inst_1"
    ],
    "ServiceAddress": "",
    "ServiceWeights": {
        "Passing": 1,
      "Warning": 1
    },
    "ServiceMeta": {},
    "ServicePort": 8081,
    "ServiceSocketPath": "",
    "ServiceEnableTagOverride": false,
    "ServiceProxy": {
        "Mode": "",
      "MeshGateway": {},
      "Expose": {}
    },
    "ServiceConnect": {},
    "ServiceLocality": null,
    "CreateIndex": 124,
    "ModifyIndex": 124
  }
]

Use the dig command to query Consul DNS service.

$ dig @consul-server-0  hashicups-api.service.dc1.consul

That will produce an output similar to the following.


; <<>> DiG 9.18.19-1~deb12u1-Debian <<>> @consul-server-0 hashicups-api.service.dc1.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 56699
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;hashicups-api.service.dc1.consul. IN   A

;; ANSWER SECTION:
hashicups-api.service.dc1.consul. 0 IN  A   192.168.80.8
hashicups-api.service.dc1.consul. 0 IN  A   192.168.80.6

;; Query time: 1 msec
;; SERVER: 192.168.80.7#53(consul-server-0) (UDP)
;; WHEN: Sat Nov 18 13:27:53 UTC 2023
;; MSG SIZE  rcvd: 93

Notice the output reports two instances for the service.

Click on the Consul UI tab

The default page, Services, shows all registered services. Notice the hashicups-api service has two instances.

Services page - Multiple API instances

Click on the hashicups-api service and in the new page, click on the Instances tab, you will get the details of the different service instances.

Service API page - Multiple instances

Finally, select the Nodes page. This will display a list of agents in the Consul datacenter. You can notice the two different agents running for hashicups-api.

Nodes page - Multiple API instances

Next steps

In this tutorial you learned the necessary steps to generate Access Control Lists' tokens to add a new Consul agent to your Consul datacenter and to register a new service to your Consul service mesh.

Specifically you:

Generated a new management token used the built-in global-management policy.
Generated a new token from a pre-existing policy
Generated a new token using the node-identity policy
Generated a new token using the service-identity policy
Verified the successful node and service registration using all the different interfaces provided by Consul: CLI, API, DNS, and UI

For more information about the topics covered in this tutorial, refer to the following resources:

To learn more about other security features provided by Consul service mesh refer to:

Service requests with L7 intentions

Permissive mTLS

Tutorial scenario

Prerequisites

Configure CLI to interact with Consul

Create ACL token for ACL management

Create ACL configuration for Consul agent

Generate default token

Generate agent token

Generate config_file_service_registration token

Generate Consul agent ACL configuration section

Generate hashicups-api service configuration

Add new service to Consul service mesh with ACLs

Start Consul process

Start Envoy sidecar proxy

Start the API service instance

Verify service registration

Next steps

Generate `default` token

Generate `agent` token

Generate `config_file_service_registration` token