Consul
Observe Consul service mesh traffic
In this tutorial, you will configure and use Consul to observe traffic within your service mesh. This enables you to quickly understand how services interact with each other and effectively debug your services' traffic.
To enable Consul's observability features, you will:
- Enable additional Consul features using Helm or the Consul K8S CLI
- Deploy monitoring suite (Grafana, Loki, Prometheus)
- Generate traffic in the HashiCups UI
- Explore the Consul UI metrics visualization
- Explore the monitoring suite
Prerequisites
The tutorial assumes that you have completed the previous tutorials in this getting started collection.
For this tutorial, you will need:
Enable Consul telemetry features
You will now enable additional Consul features in your Kubernetes cluster using the official Consul Helm chart or the consul-k8s CLI.
Review the additional configuration details. These configuration items enable metrics features for Consul agents, the Consul UI, and the Envoy proxy sidecars.
helm/values-v3.yaml
# Contains values that affect multiple components of the chart.
global:
## …
metrics:
enabled: true
# Enables Consul servers and clients metrics.
enableAgentMetrics: true
# Configures the retention time for metrics in Consul servers and clients.
agentMetricsRetentionTime: "1m"
## …
ui:
metrics:
enabled: true
# The metrics provider specification.
provider: "prometheus"
# The URL of the prometheus metrics server.
baseURL: http://prometheus-server.default.svc.cluster.local
## ...
connectInject:
metrics:
defaultEnabled: true
# Enables service metrics and sidecar proxy metrics merging into one endpoint.
defaultEnableMerging: true
## …
Update Consul in your Kubernetes cluster with Consul K8S CLI. Confirm the run by entering y.
$ consul-k8s upgrade -config-file=helm/values-v3.yaml
Refer to the Consul K8S CLI documentation to learn more about additional settings.
In addition to updating your Consul environment, you will also configure the Envoy proxy sidecars in your service mesh to expose a Prometheus metrics endpoint. This allows Prometheus to scrape metrics from the Envoy sidecar proxies.
The ProxyDefaults configuration entry lets you configure global defaults across all services for Consul service mesh proxy configurations. The proxy/proxy-defaults.yaml file configures envoy_prometheus_bind_addr to expose a Prometheus metrics endpoint.
proxy/proxy-defaults.yaml
apiVersion: consul.hashicorp.com/v1alpha1
kind: ProxyDefaults
metadata:
name: global
spec:
config:
envoy_prometheus_bind_addr: '0.0.0.0:20200'
Modify the proxy defaults.
$ kubectl apply -f proxy/proxy-defaults.yaml
proxydefaults.consul.hashicorp.com/global created
Review the Consul Proxy Defaults documentation to learn more.
Restart sidecar proxies
You need to restart your sidecar proxies to retrieve the updated proxy defaults configuration. To do so, redeploy your HashiCups services.
First, delete your existing HashiCups services.
$ kubectl delete --filename hashicups/v1/
Then, redeploy the HashiCups application.
$ kubectl apply --filename hashicups/v1/
Confirm that your proxy defaults updated your Envoy proxy’s configuration. You can view your Envoy sidecar proxy’s configuration by going to port 19000 from a service that has a sidecar proxy.
$ kubectl port-forward deploy/frontend 19000:19000
Open http://localhost:19000/config_dump in your browser to find the Envoy configuration. Search for 20200, the value you defined in the ProxyDefaults. You should find two different stanzas that reference this port. One of them is included next for reference.
{
"name": "envoy_prometheus_metrics_listener",
"address": {
"socket_address": {
"address": "0.0.0.0",
"port_value": 20200
}
}
This confirms that Consul has configured Envoy to publish Prometheus metrics.
Deploy observability suite
The monitoring suite you deploy in this tutorial uses Grafana for visualization, Prometheus for metrics, and Loki for logs.
Deploy the observability suite. This adds and installs the respective Helm charts for Grafana, Prometheus, and Loki.
$ ./install-observability-suite.sh
Tip
The observability suite deployment will take about 1-2 minutes to complete.
Check the pods to make sure the monitoring suite is up and running successfully. Your complete output will include Grafana, Prometheus services, Loki services, and HashiCups microservices.
$ kubectl get pods --namespace default
NAME READY STATUS RESTARTS AGE
frontend-5cd9997867-2kn2k 2/2 Running 0 4m53s
grafana-679c5c7854-wdpsm 1/1 Running 0 50s
loki-0 1/1 Running 0 2m13s
loki-promtail-597t7 1/1 Running 0 2m13s
nginx-64f569ff6f-fkpln 2/2 Running 0 4m53s
payments-6888957c45-k28jm 2/2 Running 0 4m52s
product-api-55c547f984-gfssk 2/2 Running 0 4m52s
product-api-db-58cbc7f8bb-cv4cm 2/2 Running 0 4m53s
prometheus-kube-state-metrics-644f869f97-wv8g4 1/1 Running 0 2m37s
prometheus-node-exporter-hq7bt 1/1 Running 0 2m37s
prometheus-pushgateway-67cf8576b7-k88c8 1/1 Running 0 2m37s
prometheus-server-84dfcc8695-nhtlp 2/2 Running 0 2m37s
public-api-bd744c57b-jnkbf 2/2 Running 0 4m52s
Generate traffic in the demo application
Since you deployed HashiCups, start generating some traffic so that you will have some metrics to look at in the Consul UI and Grafana.
Output the Consul API Gateway value to your terminal and paste it in your browser to access the HashiCups UI through the API Gateway. Click through some components within the HashiCups UI to generate traffic for visualization in the next steps.
$ echo $CONSUL_APIGW_ADDR
http://a46cc0cc9dc6841a4b1a0fa1771973ca-1032165333.us-west-2.elb.amazonaws.com:8080
Explore the Consul UI metrics visualization
Consul collects metrics and provides a built-in UI to help visualize topology between interdependent services. This can be helpful when configuring new services or troubleshooting existing connections.
Output the Consul URL value to your terminal and paste it in your browser.
$ echo $CONSUL_HTTP_ADDR
https://my-loadbalancer-1234567890.us-east-1.elb.amazonaws.com
Output the token value to your terminal and copy the value to your clipboard. You will use this ACL token to authenticate in the Consul UI.
$ echo $CONSUL_HTTP_TOKEN
fe0dd5c3-f2e1-81e8-cde8-49d26cee5efc
On the left navigation pane, click the Services tab to review your deployed services. Click the nginx service to view the metrics details.
Explore the monitoring suite
Consul service mesh lets you expose metrics from the Envoy proxies and your services for third-party monitoring tools like Prometheus and Grafana to provide visualization for layer 7 application traffic.
Open a separate terminal window and expose the Grafana UI with kubectl port-forward using the grafana service name as the target.
$ kubectl port-forward svc/grafana --namespace default 3000:3000
Navigate to the custom HashiCups dashboard in Grafana at http://localhost:3000/d/hashicups/hashicups. Notice this dashboard includes both the Envoy proxy's metrics for each HashiCups services and the individual HashiCups service logs.
Consul integrates with a wide variety of providers to enhance your complete service mesh observability experience. Check out this page to learn more about Consul and observability.
Clean up
Destroy the Terraform resources to clean up your environment. Confirm the destroy operation by inputting yes.
$ terraform destroy
Note: Due to race conditions with the various cloud resources created in this tutorial, it may be necessary to run the destroy operation twice to ensure all resources have been properly removed.
Next steps
In this tutorial, you enabled Consul monitoring features and deployed a monitoring suite into your Kubernetes cluster. After deploying the monitoring suite, you used the monitoring suite to explore the metrics and logs of the demo application, HashiCups, in your service mesh.
You have now completed the getting started collection!
Check out the following resources to learn more about running Consul on Kubernetes: