Resolve Consul DNS Requests in Kubernetes

One of the primary query interfaces to Consul is the DNS interface. You can configure Consul DNS in Kubernetes using a stub-domain configuration if using KubeDNS or a proxy configuration if using CoreDNS.

Once configured, DNS requests in the form <consul-service-name>.service.consul will resolve for services in Consul. This will work from all Kubernetes namespaces.

Consul DNS Cluster IP

To configure KubeDNS or CoreDNS you'll first need the ClusterIP of the Consul DNS service created by the Helm chart.

The default name of the Consul DNS service will be consul-dns. Use that name to get the ClusterIP:

$ kubectl get svc consul-dns --output jsonpath='{.spec.clusterIP}'
10.35.240.78%

For this installation the ClusterIP is 10.35.240.78.

KubeDNS

If using KubeDNS, you need to create a ConfigMap that tells KubeDNS to use the Consul DNS service to resolve all domains ending with .consul:

Export the Consul DNS IP as an environment variable:

export CONSUL_DNS_IP=10.35.240.78

And create the ConfigMap:

$ cat <<EOF | kubectl apply --filename -
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
  name: kube-dns
  namespace: kube-system
data:
  stubDomains: |
    {"consul": ["$CONSUL_DNS_IP"]}
EOF
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
configmap/kube-dns configured

Ensure that the ConfigMap was created successfully:

$ kubectl get configmap kube-dns --namespace kube-system --output yaml
apiVersion: v1
data:
  stubDomains: |
    {"consul": ["10.35.240.78"]}
kind: ConfigMap
...

Now skip ahead to the Verifying DNS Works section.

CoreDNS Configuration

If using CoreDNS instead of KubeDNS in your Kubernetes cluster, you will need to update your existing coredns ConfigMap in the kube-system namespace to include a forward definition for consul that points to the cluster IP of the Consul DNS service.

Edit the ConfigMap:

$ kubectl edit configmap coredns --namespace kube-system

And add the consul block below the default .:53 block and replace <consul-dns-service-cluster-ip> with the DNS Service's IP address you found previously.

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    addonmanager.kubernetes.io/mode: EnsureExists
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        <Existing CoreDNS definition>
    }
+   consul {
+     errors
+     cache 30
+     forward . <consul-dns-service-cluster-ip>
+   }

OpenShift DNS Operator

You can use DNS forwarding to override the default forwarding configuration in the /etc/resolv.conf file by specifying the consul-dns service for the consul subdomain (zone).

Find consul-dns service clusterIP:

$ oc get svc consul-dns --namespace consul --output jsonpath='{.spec.clusterIP}'
172.30.186.254

Edit the default DNS Operator:

$ oc edit edit dns.operator/default

Append the following servers section entry to the spec section of the DNS Operator configuration:

spec:
  servers:
  - name: consul-server
    zones:
    - consul
    forwardPlugin:
      policy: Random
      upstreams:
      - 172.30.186.254 # Set to clusterIP of consul-dns service

Save the configuration changes and verify the dns-default configmap has been updated:

$ oc get configmap/dns-default -n openshift-dns -o yaml

Example output with updated consul forwarding zone:

...
data:
  Corefile: |
    # consul-server
    consul:5353 {
        prometheus 127.0.0.1:9153
        forward . 172.30.186.254 {
            policy random
        }
        errors
        log . {
            class error
        }
        bufsize 1232
        cache 900 {
            denial 9984 30
        }
    }
...

Verifying DNS Works

To verify DNS works, run a simple job to query DNS. Save the following job to the file job.yaml and run it:

apiVersion: batch/v1
kind: Job
metadata:
  name: dns
spec:
  template:
    spec:
      containers:
        - name: dns
          image: anubhavmishra/tiny-tools
          command: ['dig', 'consul.service.consul']
      restartPolicy: Never
  backoffLimit: 4

$ kubectl apply --filename job.yaml

Then query the pod name for the job and check the logs. You should see output similar to the following showing a successful DNS query. If you see any errors, then DNS is not configured properly.

$ kubectl get pods --show-all | grep dns
dns-lkgzl         0/1       Completed   0          6m

$ kubectl logs dns-lkgzl
; <<>> DiG 9.11.2-P1 <<>> consul.service.consul
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4489
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 4

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;consul.service.consul.     IN  A

;; ANSWER SECTION:
consul.service.consul.  0   IN  A   10.36.2.23
consul.service.consul.  0   IN  A   10.36.4.12
consul.service.consul.  0   IN  A   10.36.0.11

;; ADDITIONAL SECTION:
consul.service.consul.  0   IN  TXT "consul-network-segment="
consul.service.consul.  0   IN  TXT "consul-network-segment="
consul.service.consul.  0   IN  TXT "consul-network-segment="

;; Query time: 5 msec
;; SERVER: 10.39.240.10#53(10.39.240.10)
;; WHEN: Wed Sep 12 02:12:30 UTC 2018
;; MSG SIZE  rcvd: 206