Consul
Control traffic communication between services with intentions
In a Consul service mesh scenario you configure the single service instances to listen on the loopback interface and, using the upstreams in the service definition, Envoy sidecar proxies permit communication with other services by adding listeners locally.
These communication are protected via an mTLS layer that prevents unauthorized services, inside or outside the mesh, to access them.
This removes the need to generate ad-hoc firewall rules for each service instance and moves the focus over service identity.
Consul uses service intentions to further limit the traffic only to the services explicitly allowed to communicate. This limits the surface exposed by the single service and constitutes one extra step towards zero-trust security.
In this tutorial, you will learn how to tune service access by using intentions.
Specifically, you will:
- Understand the ACL permissions required to manipulate intentions
- List the current intentions in your Consul service mesh
- Change a service definition to add a new upstream
- Apply intentions to your Consul datacenter to prevent unwanted traffic
Tutorial scenario
This tutorial uses HashiCups, a demo coffee shop application made up of several microservices running on VMs.
At the beginning of the tutorial, you have a fully deployed Consul service mesh, with Envoy sidecar proxies running alongside each service, and an API Gateway configured to permit access to the NGINX service.
The Consul service mesh is initially configured with an allow-all intention that enables every service to communicate with each other.
The only requirement for a service to service communication, in this scenario, is the presence of an upstream in the source service definition.
While this can be acceptable in the onboarding phase of your service mesh, in a production environment you want to transition to a deny-all default intention, where only the necessary communications are explicitly allowed.
In this scenario, the service communications needed for HashiCups to work properly are the following:
- The NGINX service needs to communicate with Frontend and API services
- The API service needs to communicate with the Database service
- The API Gateway needs to communicate with the NGINX service
By the end of this tutorial, you will have configured your service mesh to deny all communications except the ones explicitly allowed by the intentions you defined.
Prerequisites
This tutorial assumes you are already familiar with Consul service mesh and its core functionalities. If you are new to Consul refer to refer to the Consul Getting Started tutorials collection.
This tutorial uses an interactive lab to guide you through the process of setting up a service mesh on your VM workloads. The lab environment includes all required binaries and sample configurations.
Launch Terminal
This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.
Once the lab is loaded, Open the Bastion Host tab to login to the bastion host.
Configure CLI to interact with Consul
Configure your bastion host to communicate with your Consul environment using the two dynamically generated environment variable files.
$ source assets/scenario/env-consul.env && \
source assets/scenario/env-scenario.env
After loading the needed variables, verify you can connect to your Consul datacenter.
$ consul members
Node Address Status Type Build Protocol DC Partition Segment
consul-server-0 10.0.4.207:8301 alive server 1.16.3 2 dc1 default <all>
gateway-api 10.0.4.27:8301 alive client 1.16.3 2 dc1 default <default>
hashicups-api 10.0.4.83:8301 alive client 1.16.3 2 dc1 default <default>
hashicups-db 10.0.4.65:8301 alive client 1.16.3 2 dc1 default <default>
hashicups-frontend 10.0.4.33:8301 alive client 1.16.3 2 dc1 default <default>
hashicups-nginx 10.0.4.130:8301 alive client 1.16.3 2 dc1 default <default>
Create ACL token for intentions management
To configure intentions in a Consul datacenter, when ACLs are enabled, you need a valid token with the proper permissions.
To change intentions for a service, Consul includes the intentions scope for the ACL rules of a service.
Since in this tutorial you will configure services with names starting with hashicups-, the policy will need the intentions = "write" for all those services.
The rules to grant intentions permissions to all and only HashiCups services will be the following.
First define the output folder where to store the files generated.
$ export OUTPUT_FOLDER=assets/scenario/conf/
Then, create a policy file for the ACL rules you want to grant.
$ tee ${OUTPUT_FOLDER}/acl-policy-intentions-hashicups.hcl > /dev/null << EOF
# --------------------------------------+
# acl-policy-intentions-hashicups.hcl |
# --------------------------------------+
service_prefix "hashicups-" {
policy = "read"
intentions = "write"
}
EOF
Tip
Consul grants permissions for creating and managing intentions based on the destination, not the source. When ACLs are enabled, services and operators must present a token linked to a policy that grants the necessary permissions to the destination service.
Create the Consul policy.
$ consul acl policy create \
-name "HashiCups-intentions-policy" \
-rules @${OUTPUT_FOLDER}/acl-policy-intentions-hashicups.hcl
The output should look like the following.
ID: f495b8f2-5e5e-1a0f-78ab-27fc884305d7
Name: HashiCups-intentions-policy
Description:
Datacenters:
Rules:
# --------------------------------------+
# acl-policy-intentions-hashicups.hcl |
# --------------------------------------+
service_prefix "hashicups-" {
policy = "read"
intentions = "write"
}
Then, create a token associated with the policy.
$ consul acl token create \
-description 'HashiCups Intentions Management token' \
-policy-name HashiCups-intentions-policy --format json > ${OUTPUT_FOLDER}secrets/acl-token-intentions-hashicups.json
Extended permissions
Since the Consul datacenter contains a allow-all intention that refers to all services, the rules above are not sufficient to manipulate that intention.
To solve that you will create another policy that grants intentions:write for all services.
$ tee ${OUTPUT_FOLDER}/acl-policy-intentions-all.hcl > /dev/null << EOF
# --------------------------------------+
# acl-policy-intentions-all.hcl |
# --------------------------------------+
service_prefix "" {
policy = "read"
intentions = "write"
}
EOF
With the new rules file, create the policy.
$ consul acl policy create \
-name "all-intentions-policy" \
-rules @${OUTPUT_FOLDER}/acl-policy-intentions-all.hcl
The output should look like the following.
ID: e9e65da3-1616-5ac8-0287-3e60da8fca2f
Name: all-intentions-policy
Description:
Datacenters:
Rules:
# --------------------------------------+
# acl-policy-intentions-all.hcl |
# --------------------------------------+
service_prefix "" {
policy = "read"
intentions = "write"
}
Then, create a token associated with the policy.
$ consul acl token create \
-description 'All Intentions Management token' \
-policy-name all-intentions-policy --format json > ${OUTPUT_FOLDER}secrets/acl-token-intentions-all.json
Note
The second token grants access to intentions for all the services present in the datacenter. The use of the token should be limited to datacenter operators, while the first token can be assigned to application developers to offload some of the maintenance for the single application.
To continue with the tutorial, using minimal permissions, export the first token as an environment variable.
$ export CONSUL_HTTP_TOKEN=`cat ${OUTPUT_FOLDER}secrets/acl-token-intentions-hashicups.json | jq -r ".SecretID"`
Check intentions for a Consul datacenter
Check existing intentions in your Consul datacenter.
Use the consul intention command to check existing intentions.
$ consul intention list
ID Source Action Destination Precedence
* allow * 5
From the output you can verify that all (*) service-to-service connections are allowed.
Check HashiCups UI
In this section you will verify that the HashiCups application is working properly.
Click on the HashiCups tab.
Confirm that HashiCups is reachable from the API Gateway. The Envoy sidecar proxies route each service's local traffic to the relevant upstream.
Enable a new service-to-service communication
In the present scenario, upstreams are already defined for the HashiCups' services to permit the intended traffic flows.
The allow-all intention permits to define new service-to-service communication flows by adding upstream definitions in the service configuration files.
You will now learn how to add a new service-to-service communication flow by adding an upstream that permits the Database service to communicate with the API service.
This is the intended sequence of steps that you will perform in your environment when a new communication flow is identified as necessary for a service.
Check initial status
First, you will check that, in the initial state, the Database service is not configured to communicate with the API service.
Click on the API tab.
Check existing listening processes.
$ netstat -natp | grep LISTEN
The output will be similar to the following, the order of listeners might be different in your scenario.
tcp 0 0 127.0.0.1:9090 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:21000 0.0.0.0:* LISTEN 6422/envoy
tcp 0 0 10.0.4.7:8301 0.0.0.0:* LISTEN 6359/consul
tcp 0 0 127.0.0.1:9103 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:8080 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:8081 0.0.0.0:* LISTEN 6702/public-api
tcp 0 0 127.0.0.1:8500 0.0.0.0:* LISTEN 6359/consul
tcp 0 0 127.0.0.1:8502 0.0.0.0:* LISTEN 6359/consul
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:5432 0.0.0.0:* LISTEN 6422/envoy
tcp 0 0 127.0.0.1:19000 0.0.0.0:* LISTEN 6422/envoy
tcp 0 0 127.0.0.1:8600 0.0.0.0:* LISTEN 6359/consul
tcp 0 0 127.0.0.1:12345 0.0.0.0:* LISTEN 6751/grafana-agent
tcp6 0 0 :::22 :::* LISTEN -
The service you want to reach, public-api, is configured to listen on the loopback interface, on port `8081``, and is not accessible externally.
From the output, you can notice that the VM only exposes three listeners externally:
- the SSH server, on port 22
- the Consul client gossip interface, on port 8301
- the Envoy process, on port 21000
The Envoy process is used by Consul service mesh to route traffic inside the service mesh services.
Among the listeners, you can also notice one Envoy process listening on port 5432. That represents the available connection from the API service (source) and the Database service (destination/upstream).
Add a new upstream to a service definition
In this section, you will add new communication channels across services by changing the service definition and adding upstreams to the services.
Click on the Database tab.
Check existing listening processes, the order of listeners might be different in your scenario.
$ netstat -natp | grep LISTEN
tcp 0 0 127.0.0.1:5432 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:19000 0.0.0.0:* LISTEN 6257/envoy
tcp 0 0 127.0.0.1:8600 0.0.0.0:* LISTEN 6194/consul
tcp 0 0 127.0.0.1:12345 0.0.0.0:* LISTEN 6503/grafana-agent
tcp 0 0 127.0.0.1:12346 0.0.0.0:* LISTEN 6503/grafana-agent
tcp 0 0 0.0.0.0:21000 0.0.0.0:* LISTEN 6257/envoy
tcp 0 0 10.0.4.30:8301 0.0.0.0:* LISTEN 6194/consul
tcp 0 0 127.0.0.1:8500 0.0.0.0:* LISTEN 6194/consul
tcp 0 0 127.0.0.1:8502 0.0.0.0:* LISTEN 6194/consul
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -
tcp6 0 0 :::22 :::* LISTEN -
Among the listeners, the process listening on port 5432 represents the Database service.
You can notice that the only processes exposed externally are the SSH server, Consul gossip, and Envoy.
In this case, there is no Envoy process that represents an upstream service.
You want to add the possibility to connect to the API service from the Database host, to do so modify the hashicups-db service definition to add the following section.
proxy {
upstreams = [
{
destination_name = "hashicups-api"
local_bind_port = 8081
}
]
}
The section describes an upstreams array for the hashicups-db service including one upstream for the hashicups-api service on port 8081.
Note
The port represents the port that will be used on the local listener and not the original port exposed by the hashicups-api service on its node.
First, populate CONSUL_HTTP_TOKEN variable using the token used by the Consul client agent.
$ export CONSUL_HTTP_TOKEN=`cat /etc/consul.d/agent-acl-tokens.hcl | grep agent | awk '{print $3}' | tr -d \"`
Then, generate the new service definition file to include the upstream definition.
$ tee /etc/consul.d/svc-hashicups-db.hcl > /dev/null << EOF
## svc-hashicups-db.hcl
service {
name = "hashicups-db"
id = "hashicups-db-1"
port = 5432
token = "${CONSUL_HTTP_TOKEN}"
connect {
sidecar_service {
proxy {
upstreams = [
{
destination_name = "hashicups-api"
local_bind_port = 8081
}
]
}
}
}
check
{
id = "check-hashicups-db",
name = "hashicups-db status check",
service_id = "hashicups-db-1",
tcp = "localhost:5432",
interval = "5s",
timeout = "5s"
}
}
EOF
Reload Consul to apply the new service configuration.
$ consul reload
Configuration reload triggered
After the Consul agent reloaded the configuration, it will automatically update the Envoy sidecar proxy configuration to include the new upstream on the local node.
$ netstat -natp | grep LISTEN
The output will be similar to the following, the order of listeners might be different in your scenario.
tcp 0 0 127.0.0.1:5432 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:19000 0.0.0.0:* LISTEN 6257/envoy
tcp 0 0 127.0.0.1:8600 0.0.0.0:* LISTEN 6194/consul
tcp 0 0 127.0.0.1:12345 0.0.0.0:* LISTEN 6503/grafana-agent
tcp 0 0 127.0.0.1:12346 0.0.0.0:* LISTEN 6503/grafana-agent
tcp 0 0 0.0.0.0:21000 0.0.0.0:* LISTEN 6257/envoy
tcp 0 0 10.0.4.30:8301 0.0.0.0:* LISTEN 6194/consul
tcp 0 0 127.0.0.1:8081 0.0.0.0:* LISTEN 6257/envoy
tcp 0 0 127.0.0.1:8500 0.0.0.0:* LISTEN 6194/consul
tcp 0 0 127.0.0.1:8502 0.0.0.0:* LISTEN 6194/consul
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -
tcp6 0 0 :::22 :::* LISTEN -
Now, the API service is exposed on the Database node loopback interface, on the port you specified in the service definition, 8081.
Check service connection
Verify you can now connect to the API service from the Database node.
$ curl --silent 'http://localhost:8081/api' \
-H 'Accept-Encoding: gzip, deflate, br' \
-H 'Content-Type: application/json' \
-H 'Accept: application/json' \
-H 'Connection: keep-alive' \
-H 'DNT: 1' \
-H 'Origin: http://localhost:8081' \
--data-binary '{"query":"mutation{ pay(details:{ name: \"nic\", type: \"mastercard\", number: \"1234123-0123123\", expiry:\"10/02\", cv2: 1231, amount: 12.23 }){id, card_plaintext, card_ciphertext, message } }"}' --compressed | jq
A successful API connection will return an output similar to the following:
{
"data": {
"pay": {
"id": "b7912087-80c8-4520-99bb-79a89f1b611e",
"card_plaintext": "1234123-0123123",
"card_ciphertext": "Encryption Disabled",
"message": "Payment processed successfully, card details returned for demo purposes, not for production"
}
}
}
By changing the service definition upstreams and reloading the Consul configuration to include the change, you added a new service-to-service communication flow to your service mesh. The fact you have an allow-all intention in place, is the other factor that allows the communication.
This is a very powerful method to configure services in your network, where the application developer can define the dependencies for each service as upstreams directly in the service definition, and have Consul service mesh make the upstreams available on the service node, directly on the loopback interface, on the specified port.
Tune intentions for your environment
Having the traffic dependencies defined as upstreams, makes easy for developers to deploy their applications from their local environment to production. The services will keep the localhost configuration and Consul will route their requests to the right service.
In a production environment, this simplicity can also bring unwanted behaviors.
A developer adding un-necessary upstreams to their service definition might open communication channels that are not intended.
Consul permits an operator to prevent unwanted traffic using intentions.
For this reason, before deploying a new application in your Consul service mesh, you should prepare an intention diagram that defines intended traffic permissions. This will be used by the operators to identify the necessary intentions your application will need to operate properly.
Understand HashiCups services' upstreams
The following intentions are required for HashiCups:
Tip
Notice these descriptions define traffic starting from the destination, Consul intentions are defined in the same way, the intention is defined for the destination service and defines all the source services that need to communicate wit it.
- The
hashicups-dbservice needs to be reached by thehashicups-apiservice. - The
hashicups-apiservice needs to be reached by thehashicups-nginxservices. - The
hashicups-frontendservice needs to be reached by thehashicups-nginxservice. - The
hashicups-nginxservice needs to be reached by thegateway-apiservice.
Define and apply intentions
Select the Bastion Host tab.
Use the provided script to generate service intentions.
$ export OUTPUT_FOLDER=assets/scenario/conf/
$ bash ops/scenarios/99_supporting_scripts/generate_consul_service_intentions.sh
Parameter Check
Create global proxy configuration
Create intention configuration files
The script creates service-intentions configuration files both in json and hcl format.
$ tree ${OUTPUT_FOLDER}global
assets/scenario/conf/global
|-- config-global-proxy-default.hcl
|-- config-global-proxy-default.json
|-- intention-allow-all.hcl
|-- intention-allow-all.json
|-- intention-api.hcl
|-- intention-api.json
|-- intention-db.hcl
|-- intention-db.json
|-- intention-frontend.hcl
|-- intention-frontend.json
|-- intention-nginx.hcl
`-- intention-nginx.json
0 directories, 12 files
Hera one example of a service-intentions configuration file generated by the script.
intention-api.hcl
Kind = "service-intentions"
Name = "hashicups-api"
Sources = [
{
Name = "hashicups-nginx"
Action = "allow"
}
]
You can notice the intention is defined for the destination service, in this case hashicups-api, specifying the source services that the intention regulates, in this case hashicups-nginx, and a directive, in this case allow, that defines the permission for the connection.
Apply specific intentions
Apply the intentions to your Consul datacenter.
Use config write to create the following intentions.
Create the intentions for the hashicups-db service.
$ consul config write ${OUTPUT_FOLDER}global/intention-db.hcl
Config entry written: service-intentions/hashicups-db
Create the intentions for the hashicups-api service.
$ consul config write ${OUTPUT_FOLDER}global/intention-api.hcl
Config entry written: service-intentions/hashicups-api
Create the intentions for the hashicups-frontend service.
$ consul config write ${OUTPUT_FOLDER}global/intention-frontend.hcl
Config entry written: service-intentions/hashicups-frontend
Create the intentions for the hashicups-nginx service to allow access from the API
gateway.
$ consul config write ${OUTPUT_FOLDER}global/intention-nginx.hcl
Config entry written: service-intentions/hashicups-nginx
Now the Consul datacenter contains, alongside the allow-all intention that permits traffic across all services, he four intentions you just defined.
These four intentions define the HashiCups traffic flow.
Remove default intention
Now that all connections are explicitly specified, you can remove the allow-all intention without affecting the application uptime.
Tip
By removing the *>* intention, Consul will use as default intention the same behavior specified by the ACL default policy. In this scenario the default policy is default_policy = "deny", therefore if no intention is explicitly defined for the communication across two services Consul will deny the connection.
$ consul config delete -kind service-intentions -name "*"
If the token you are using does not include necessary permissions to manipulate intentions for all services, the request is expected to fail with a 403 error.
Error deleting config entry service-intentions/*: Unexpected response code: 403 (Permission denied: token with AccessorID '01bcbd3a-ce74-a615-143f-8780d8391dff' lacks permission 'intention:write' on "*")
To complete the operation assign the correct token to the environment variable and try again.
$ export CONSUL_HTTP_TOKEN=`cat ${OUTPUT_FOLDER}secrets/acl-token-intentions-all.json | jq -r ".SecretID"`
The new attempt should be successful.
$ consul config delete -kind service-intentions -name "*"
Config entry deleted: service-intentions/*
Once you completed the intention deletion, you should revert back to a less privileged token.
$ export CONSUL_HTTP_TOKEN=`cat ${OUTPUT_FOLDER}secrets/acl-token-intentions-hashicups.json | jq -r ".SecretID"`
Check intentions in your Consul datacenter
After applying the desired intention, check the new intentions configuration for your Consul datacenter.
Use the consul intention command to check existing intentions.
$ consul intention list
ID Source Action Destination Precedence
gateway-api allow hashicups-nginx 9
hashicups-api allow hashicups-db 9
hashicups-nginx allow hashicups-api 9
hashicups-nginx allow hashicups-frontend 9
Verify connections are now secured
After applying the intentions and removing the allow-all*` directive you reached the final configuration for the scenario.
Last steps are to check if the HashiCups application is still working and if the connection between Database and API is now blocked.
Check HashiCups UI
Click on the HashiCups tab.
Confirm that HashiCups still works despite its services being configured to
communicate on localhost. The Envoy sidecar proxies route each service's local
traffic to the relevant upstream.
Test service to service connection
Click on the Database tab.
Verify the connection to the API service from the Database node.
$ curl 'http://localhost:8081/api' \
-H 'Accept-Encoding: gzip, deflate, br' \
-H 'Content-Type: application/json' \
-H 'Accept: application/json' \
-H 'Connection: keep-alive' \
-H 'DNT: 1' \
-H 'Origin: http://localhost:8081' \
--data-binary '{"query":"mutation{ pay(details:{ name: \"nic\", type: \"mastercard\", number: \"1234123-0123123\", expiry:\"10/02\", cv2: 1231, amount: 12.23 }){id, card_plaintext, card_ciphertext, message } }"}' --compressed
If the intentions are being enforced properly, the command will return an output similar to the following:
curl: (52) Empty reply from server
Using intentions you removed the risk of unwanted connections from the Database service to the API service without the need of ad-hoc firewall rules, and enforcing a global set of permissions even over a specific configuration made on the service definition itself.
Exit the ssh session to return to the bastion host and complete the tutorial.
$ exit
logout
Connection to hashicups-db closed.
admin@bastion:~$
Next steps
In this tutorial you learned how to configure upstreams for existing services and to open service-to-service communications inside Consul service mesh.
You then learned how to secure communication by limiting connections only across specific services.
In the next tutorial, you will learn how use application aware intentions to tune permissions at a finer level of detail.
For more information about the topics covered in this tutorial, refer to the following resources: