Consul
Control service requests with application-aware intentions
In the previous tutorial you learned how to define intentions for your services in Consul service mesh.
Intentions presented in the previous tutorial allow broad all-or-nothing access control between pairs of services and provide a secure approach for TCP-based services. For this reason they are also referred to as L4 intentions, because they operate at the transport layer of the ISO/OSI model, and are only capable of allowing or denying access to a specific service.
Most modern scenarios, however, contain HTTP-based services, such as APIs or web services, that expose multiple endpoints and that might have different access profiles depending on the source of the request or on some metadata of the request itself. This includes HTTP headers, cookies, and URL paths.
To secure such services, a simple all-or-nothing access control is often insufficient.
Consul allows to specify a different, more powerful, kind of intentions that takes into account request metadata to define request permissions. They are usually referred as L7 intentions, or application-aware intentions, because they operate at the application layer of he ISO/OSI model, and permit a fine grained filtering of the different requests.
In this tutorial, you will learn how to apply application-aware intentions to your Consul datacenter to fine tune allowed requests for an existing service.
Tutorial scenario
This tutorial uses HashiCups, a demo coffee shop application made up of several microservices running on VMs.
At the beginning of the tutorial, you have a fully deployed Consul service mesh, with Envoy sidecar proxies running alongside each service, and an API Gateway configured to permit access to the NGINX service.
The Consul service mesh is initially configured with a deny-all default policy for ACL and intentions, that disables undesired service-to-service communications, and with a set of minimal intentions that permit the HashiCups services to communicate properly.
During the tutorial, you will verify that the API service exposes multiple endpoints, /, /api, and /health, and that the L4 intentions are not suited to selectively block requests on the different endpoints.
By the end of the tutorial, you will have learned how to apply L7 intentions to your Consul service mesh to remove access to the / and /health endpoints while allowing traffic over the /api endpoint needed for the HashiCups application.
Prerequisites
This tutorial assumes you are already familiar with Consul service mesh and its core functionalities. If you are new to Consul refer to refer to the Consul Getting Started tutorials collection.
If you completed the previous tutorial, the infrastructure you already have in place meets all of the prerequisites.
Login into the bastion host VM
Login to the bastion host using ssh.
$ ssh -i certs/id_rsa.pem admin@`terraform output -raw ip_bastion`
#...
admin@bastion:~$
Configure CLI to interact with Consul
Configure your bastion host to communicate with your Consul environment using the two dynamically generated environment variable files.
$ source assets/scenario/env-consul.env && \
source assets/scenario/env-scenario.env
After loading the needed variables, verify you can connect to your Consul datacenter.
$ consul members
Node Address Status Type Build Protocol DC Partition Segment
consul-server-0 10.0.4.207:8301 alive server 1.16.3 2 dc1 default <all>
gateway-api 10.0.4.27:8301 alive client 1.16.3 2 dc1 default <default>
hashicups-api 10.0.4.83:8301 alive client 1.16.3 2 dc1 default <default>
hashicups-db 10.0.4.65:8301 alive client 1.16.3 2 dc1 default <default>
hashicups-frontend 10.0.4.33:8301 alive client 1.16.3 2 dc1 default <default>
hashicups-nginx 10.0.4.130:8301 alive client 1.16.3 2 dc1 default <default>
Setup production intentions
If you completed the previous tutorial, the infrastructure you already have in place meets all of the prerequisites.
Check intentions for a Consul datacenter
After setting up the prerequisites, check the intentions configuration for your Consul datacenter.
Use the consul intention command to check existing intentions.
$ consul intention list
ID Source Action Destination Precedence
gateway-api allow hashicups-nginx 9
hashicups-api allow hashicups-db 9
hashicups-nginx allow hashicups-api 9
hashicups-nginx allow hashicups-frontend 9
Create ACL token for intentions management
To configure intentions for a service, Consul includes the intentions scope for the ACL rules of a service.
In this tutorial you will configure intentions for the hashicups-api service, therefore the policy will need the intentions = "write" for this service.
First define the output folder where to store the files generated.
$ export OUTPUT_FOLDER=assets/scenario/conf/
Then, create a policy file for the ACL rules you want to grant.
$ tee ${OUTPUT_FOLDER}/acl-policy-intentions-hashicups-api-l7.hcl > /dev/null << EOF
# ----------------------------------------+
# acl-token-intentions-hashicups-api-l7 |
# ----------------------------------------+
service_prefix "hashicups-api" {
policy = "write"
intentions = "write"
}
EOF
Tip
Consul grants permissions for creating and managing intentions based on the destination, not the source. When ACLs are enabled, services and operators must present a token linked to a policy that grants the necessary permissions to the destination service.
Create the Consul policy.
$ consul acl policy create \
-name "hashicups-api-l7-intentions-policy" \
-rules @${OUTPUT_FOLDER}/acl-policy-intentions-hashicups-api-l7.hcl
The output should look like the following.
ID: 75ae537c-9445-6b5d-a1ae-f3b60195f0c9
Name: hashicups-api-l7-intentions-policy
Description:
Datacenters:
Rules:
# ----------------------------------------+
# acl-token-intentions-hashicups-api-l7 |
# ----------------------------------------+
service_prefix "hashicups-api" {
policy = "write"
intentions = "write"
}
Then, create a token associated with the policy.
$ consul acl token create \
-description 'HashiCups API L7 Intentions Management token' \
-policy-name hashicups-api-l7-intentions-policy --format json > ${OUTPUT_FOLDER}secrets/acl-token-intentions-hashicups-api-l7.json
To continue with the tutorial, using minimal permissions, export the first token as an environment variable.
$ export CONSUL_HTTP_TOKEN=`cat ${OUTPUT_FOLDER}secrets/acl-token-intentions-hashicups-api-l7.json | jq -r ".SecretID"`
Explore HashiCups API service endpoints
The HashiCups API service is configured to be reachable only from the NGINX node in this scenario. This implementation allows all-or-nothing access control between pairs of services and is a secure approach for TCP-based services.
HTTP-based services, like the HashiCups API service, often offer a more complex range of requests and can expose multiple endpoints to implement different request types or to expose different content.
You will now explore the different endpoints provided by the HashiCups API service.
Explore the HashiCups API /api path
Login to hashicups-nginx from your bastion host.
$ ssh -i certs/id_rsa hashicups-nginx
#..
admin@hashicups-nginx:~$
Verify that the API service is exposed locally on the NGINX node on the port specified in the service upstream definition, 8081.
$ netstat -natp | grep LISTEN
If the service mesh is configured correctly you should get an output similar to the following, with one entry, on port 8081 corresponding to the API service.
tcp 0 0 0.0.0.0:21000 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:8600 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:8502 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:8500 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:19000 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:3000 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.11:35427 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:12346 0.0.0.0:* LISTEN 160/grafana-agent
tcp 0 0 127.0.0.1:12345 0.0.0.0:* LISTEN 160/grafana-agent
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN -
tcp 0 0 192.168.32.11:8301 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:8081 0.0.0.0:* LISTEN -
tcp6 0 0 :::22 :::* LISTEN -
Run a query to the HashiCups API service from the NGINX node.
$ curl --silent 'http://localhost:8081/api' \
-H 'Accept-Encoding: gzip, deflate, br' \
-H 'Content-Type: application/json' \
-H 'Accept: application/json' \
-H 'Connection: keep-alive' \
-H 'DNT: 1' \
-H 'Origin: http://localhost:8081' \
--data-binary '{"query":"mutation{ pay(details:{ name: \"nic\", type: \"mastercard\", number: \"1234123-0123123\", expiry:\"10/02\", cv2: 1231, amount: 12.23 }){id, card_plaintext, card_ciphertext, message } }"}' --compressed | jq
A successful request will return an output similar to the following:
{
"data": {
"pay": {
"id": "b7912087-80c8-4520-99bb-79a89f1b611e",
"card_plaintext": "1234123-0123123",
"card_ciphertext": "Encryption Disabled",
"message": "Payment processed successfully, card details returned for demo purposes, not for production"
}
}
}
In the previous command, you used the /api path for the request, but the API service also exposes other paths.
Explore the HashiCups API root path
The default path, /, provides a number of information about the API service configuration.
$ curl --silent 'http://localhost:8081/'
Example output.
<!DOCTYPE html>
<html>
<head>
<meta charset=utf-8/>
<meta name="viewport" content="user-scalable=no, initial-scale=1.0, minimum-scale=1.0, maximum-scale=1.0, minimal-ui">
<link rel="shortcut icon" href="https://graphcool-playground.netlify.com/favicon.png">
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/graphql-playground-react@1.7.20/build/static/css/index.css"
integrity="sha256-cS9Vc2OBt9eUf4sykRWukeFYaInL29+myBmFDSa7F/U=" crossorigin="anonymous"/>
<link rel="shortcut icon" href="https://cdn.jsdelivr.net/npm/graphql-playground-react@1.7.20/build/favicon.png"
integrity="sha256-GhTyE+McTU79R4+pRO6ih+4TfsTOrpPwD8ReKFzb3PM=" crossorigin="anonymous"/>
<script src="https://cdn.jsdelivr.net/npm/graphql-playground-react@1.7.20/build/static/js/middleware.js"
integrity="sha256-4QG1Uza2GgGdlBL3RCBCGtGeZB6bDbsw8OltCMGeJsA=" crossorigin="anonymous"></script>
<title>Playground</title>
</head>
<body>
<style type="text/css">
html { font-family: "Open Sans", sans-serif; overflow: hidden; }
body { margin: 0; background: #172a3a; }
</style>
<div id="root"/>
<script type="text/javascript">
window.addEventListener('load', function (event) {
const root = document.getElementById('root');
root.classList.add('playgroundIn');
const wsProto = location.protocol == 'https:' ? 'wss:' : 'ws:'
GraphQLPlayground.init(root, {
endpoint: location.protocol + '//' + location.host + '\/api',
subscriptionsEndpoint: wsProto + '//' + location.host + '\/api',
shareEnabled: true,
settings: {
'request.credentials': 'same-origin'
}
})
})
</script>
</body>
</html>
Explore the HashiCups API /health path
The API service also exposes a simple /health endpoint that provides a summary of the state of the service.
$ curl --silent 'http://localhost:8081/health'
If the service is up and running the call will respond with a simple message.
ok
In some cases it can respond with a message informing about the temporary state of one of its sub-components.
error. the following services are down: product-api
To continue with the tutorial, exit the ssh session to return to the bastion host.
$ exit
logout
Connection to hashicups-nginx closed.
admin@bastion:~$
Security considerations
When allowing connections towards an HTTP-based service, you also allow access to all endpoints exposed by that service. While some of these endpoints might be necessary for the correct behavior of the service, or might be needed by the service's downstreams, some of them might be unnecessary or might also expose sensitive information to external users.
In this case, allowing general connections towards the HashiCups API service allows access to all paths, including the / and /health paths, which are not required for the correct behavior of HashiCups. Extraneous application paths such as these could represent a security risk.
For cases like this, where a simple yes/no configuration is not enough to guarantee security, Consul allows to specify a different, more powerful, kind of intentions.
They are usually referred as L7 intentions, or application-aware intentions, because they operate at the application layer of he ISO/OSI model and permit a fine grained filtering of the different requests.
In the rest of the tutorial, you will learn how to apply application-aware intentions to your Consul datacenter to fine tune allowed requests for the API service.
Define and apply L7 intentions
The use of L7 intentions is only available for destination services using an HTTP-based protocol.
In this case the destination service is hashicups-api and you can verify the configuration for it using the consul config command.
$ consul config read -kind service-defaults -name hashicups-api
Error reading config entry service-defaults/hashicups-api: Unexpected response code: 404 (Config entry not found for "service-defaults" / "hashicups-api")
The 404 message indicates that there is no configuration present for the hashicups-api service. The service can be regarded as a TCP-based service.
Tip
Service protocol can also be specified using proxy-defaults and, in that case, they will apply to all services in the service mesh. The protocol specified for individual service instances in the service-defaults configuration entry takes precedence over the globally-configured value set in the proxy-defaults.
Define service protocol for hashicups-api
To apply L7 intentions to hashicups-api you will have to define it as an HTTP-based service first.
Define the output folder for the configuration file to be created.
$ export OUTPUT_FOLDER=assets/scenario/conf/
Then, create the configuration file that defines the Protocol value for the service.
$ tee ${OUTPUT_FOLDER}config-global-default-hashicups-api.hcl > /dev/null << EOF
Kind = "service-defaults"
Name = "hashicups-api"
Protocol = "http"
EOF
Finally, apply the configuration to your Consul datacenter.
$ consul config write ${OUTPUT_FOLDER}config-global-default-hashicups-api.hcl
Config entry written: service-defaults/hashicups-api | jq
Once the configuration is applied, verify the hashicups-api service now gets a configuration applied that defines it as an HTTP-based service.
$ consul config read -kind service-defaults -name hashicups-api
The command will return the service-defaults configuration applied to the hashicups-api service. Notice that the output is not empty this time and that the Protocol value is set to http.
{
"Kind": "service-defaults",
"Name": "hashicups-api",
"Protocol": "http",
"TransparentProxy": {},
"MeshGateway": {},
"Expose": {},
"CreateIndex": 1039,
"ModifyIndex": 1039
}
Define L7 intentions for hashicups-api
Once the API service is configured to be HTTP-based, it is possible to apply the L7 intention to it.
First, create the configuration file to describe the intention.
$ tee ${OUTPUT_FOLDER}config-intention-hashicups-api.hcl > /dev/null << EOF
Kind = "service-intentions"
Name = "hashicups-api"
Sources = [
{
Name = "hashicups-nginx"
Permissions = [
{
Action = "allow"
HTTP {
PathExact = "/api"
}
}
]
},
{
Name = "*"
Permissions = [
{
Action = "deny"
HTTP {
PathExact = "/health"
Methods = ["GET"]
}
},
{
Action = "deny"
HTTP {
PathExact = "/"
Methods = ["GET"]
}
}
]
},
# NOTE: a default catch-all based on the default ACL policy will apply to
# unmatched connections and requests. Typically this will be DENY.
]
EOF
Then, apply the configuration using the consul config command.
$ consul config write ${OUTPUT_FOLDER}config-intention-hashicups-api.hcl
Config entry written: service-intentions/hashicups-api
Note
Notice that the intention definition file also defines the intention between hashicups-nginx and hashicups-api even if an intention already exists for that connection in Consul. This is because when writing a configuration into Consul, the Name parameter is used to describe the configuration and, in case an existing configuration with that name is already in place, it will get overwritten by the new one.
Check intentions in your Consul datacenter
After applying the desired intention, check the new intentions configuration for your Consul datacenter.
Use the consul intention command to check existing intentions.
$ consul intention list
ID Source Action Destination Precedence
hashicups-api allow hashicups-db 9
hashicups-nginx hashicups-api 9
* hashicups-api 8
Note
The intentions shown by the command depend on the ACL token present in the CONSUL_HTTP_TOKEN variable. If using a limited token, that only gives permissions over the hashicups-api service, only the intentions relative to that service will be returned by the command.
To get more details over the specific intentions in place for hashicups-api, use the consul config command.
$ consul config read -kind service-intentions -name hashicups-api
This will provide you with the full details of all service intentions regarding the hashicups-api service.
{
"Kind": "service-intentions",
"Name": "hashicups-api",
"Sources": [
{
"Name": "hashicups-nginx",
"Permissions": [
{
"Action": "allow",
"HTTP": {
"PathExact": "/api"
}
}
],
"Precedence": 9,
"Type": "consul"
},
{
"Name": "*",
"Permissions": [
{
"Action": "deny",
"HTTP": {
"PathExact": "/health",
"Methods": [
"GET"
]
}
},
{
"Action": "deny",
"HTTP": {
"PathExact": "/",
"Methods": [
"GET"
]
}
}
],
"Precedence": 8,
"Type": "consul"
}
],
"CreateIndex": 2293,
"ModifyIndex": 2976
}
Verify connections are now secured
Now that all intentions are in place, it is time to verify that the service works as expected and that the undesired endpoints are now unavailable from other nodes.
Check HashiCups UI
The first check to perform is to make sure that the HashiCups application is still functional.
First, retrieve the API Gateway address.
For this scenario, you can get the API Gateway public IP directly from the bastion host.
$ echo "https://`cat /etc/hosts | grep gateway-api-public | awk '{print $1}'`:8443"
The output will be similar to the following:
https://35.87.44.101:8443
Then, open the address in a browser.
Confirm that HashiCups still works with the new intentions in place.
Test service to service connection
Now, after confirming that the application is working as expected, you can verify that only the /api endpoint is reachable from the NGINX node.
Login to hashicups-nginx.
$ ssh -i certs/id_rsa hashicups-nginx
#..
admin@hashicups-nginx:~$
Verify the connection to the API service from the NGINX node.
$ curl --silent 'http://localhost:8081/api' \
-H 'Accept-Encoding: gzip, deflate, br' \
-H 'Content-Type: application/json' \
-H 'Accept: application/json' \
-H 'Connection: keep-alive' \
-H 'DNT: 1' \
-H 'Origin: http://localhost:8081' \
--data-binary '{"query":"mutation{ pay(details:{ name: \"nic\", type: \"mastercard\", number: \"1234123-0123123\", expiry:\"10/02\", cv2: 1231, amount: 12.23 }){id, card_plaintext, card_ciphertext, message } }"}' --compressed | jq
If the intentions are being enforced properly, the command will return an output similar to the following:
{
"data": {
"pay": {
"id": "b7912087-80c8-4520-99bb-79a89f1b611e",
"card_plaintext": "1234123-0123123",
"card_ciphertext": "Encryption Disabled",
"message": "Payment processed successfully, card details returned for demo purposes, not for production"
}
}
}
Next, you can test if the / endpoint is reachable.
$ curl --silent 'http://localhost:8081/'
If the intentions are being enforced properly, the command will return an output similar to the following:
RBAC: access denied
This means that the request is being blocked by the Role Based Access Control (RBAC) enforced by Consul.
Finally, you can test if the /health endpoint is reachable.
$ curl --silent 'http://localhost:8081/health'
If the intentions are being enforced properly, the command will return an output similar to the following:
RBAC: access denied
This means that the request is being blocked by the Role Based Access Control (RBAC) enforced by Consul.
Next steps
In this tutorial you learned how to use L7 intentions in Consul service mesh to control service requests at the application level.
More specifically, you learned to:
- Verify the existing intentions in a Consul datacenter
- Modify the service default protocol from TCP to HTTP
- Create an L7 intention definition
- Apply the intention to Consul datacenter using API endpoints or Consul CLI commands
For more information about the topics covered in this tutorial, refer to the following resources: