Application leader election

This topic describes the process for building client-side leader elections for service instances using Consul's session mechanism for building distributed locks and the Consul key/value store, which is Consul's key/value datastore.

This topic is not related to Consul's leader election. For more information about the Raft leader election used internally by Consul, refer to consensus protocol documentation.

Background

Some distributed applications, like HDFS or ActiveMQ, require setting up one instance as a leader to ensure application data is current and stable.

Consul's support for sessions and watches allows you to build a client-side leader election process where clients use a lock on a key in the KV datastore to ensure mutual exclusion and to gracefully handle failures.

All service instances that are participating should coordinate on a key format. We recommend the following pattern:

service/<service name>/leader

Requirements

A running Consul server
A path in the Consul KV datastore to acquire locks and to store information about the leader. The instructions on this page use the following key: service/leader.
If ACLs are enabled, a token with the following permissions:
- session:write permissions over the service session name
- key:write permissions over the key
- The curl command

Expose the token using the CONSUL_HTTP_TOKEN environment variable.

Client-side leader election procedure

The workflow for building a client-side leader election process has the following steps:

For each client trying to acquire the lock:
1. Create a session associated with the client node.
2. Acquire the lock on the designated key in the KV store using the acquire parameter.
3. Watch the KV key to verify if the lock was released. If no lock is present, try to acquire a lock.
For the client that acquires the lock:
1. Periodically, renew the session to avoid expiration.
2. Optionally, release the lock.
For other services:
1. Watch the KV key to verify there is at least one process holding the lock.
2. Use the values written under the KV path to identify the leader and update configurations accordingly.

Create a new session

Create a configuration for the session. The minimum viable configuration requires that you specify the session name. The following example demonstrates this configuration.

{
  "Name": "session_name"
}

Create a session using the /session Consul HTTP API endpoint. In the following example, the node's hostname is the session name.

$ curl --silent \
  --header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
  --data '{"Name": "'`hostname`'"}' \
  --request PUT \
  http://127.0.0.1:8500/v1/session/create | jq

The command returns a JSON object containing the ID of the newly created session.

{
  "ID": "d21d60ad-c2d2-b32a-7432-ca4048a4a7d6"
}

Verify session

Use the /v1/session/list endpoint to retrieve existing sessions.

$ curl --silent \
  --header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
  --request GET \
  http://127.0.0.1:8500/v1/session/list | jq

The command returns a JSON array containing all available sessions in the system.

[
  {
    "ID": "d21d60ad-c2d2-b32a-7432-ca4048a4a7d6",
    "Name": "hashicups-db-0",
    "Node": "hashicups-db-0",
    "LockDelay": 15000000000,
    "Behavior": "release",
    "TTL": "",
    "NodeChecks": [
      "serfHealth"
    ],
    "ServiceChecks": null,
    "CreateIndex": 11956,
    "ModifyIndex": 11956
  }
]

You can verify from the output that the session is associated with the hashicups-db-0 node, which is the client agent where the API request was made.

With the exception of the Name, all parameters are set to their default values. The session is created without a TTL value, which means that it never expires and requires you to delete it explicitly.

Depending on your needs you can create sessions specifying more parameters such as:

TTL - If provided, the session is invalidated and deleted if it is not renewed before the TTL expires.
ServiceChecks - Specifies a list of service checks to monitor. The session is invalidated if the checks return a critical state.

By setting these extra parameters, you can create a client-side leader election workflow that automatically releases the lock after a specified amount of time since the last renew, or that automatically releases locks when the service holding them fails.

For a full list of parameters available refer to the /session/create endpoint documentation.

Acquire the lock

Create the data object to associate to the lock request.

The data of the request should be a JSON object representing the local instance. This value is opaque to Consul, but it should contain whatever information clients require to communicate with your application. For example, it could be a JSON object that contains the node's name and the application's port.

{
  "Node": "node-name",
  "Port": "8080"
}

Acquire a lock for a given key using the PUT method on a KV entry with the ?acquire=<session> query parameter.

$ curl --silent \
  --header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
  --data '{"Node": "'`hostname`'"}' \
  --request PUT \
  http://localhost:8500/v1/kv/service/leader?acquire=d21d60ad-c2d2-b32a-7432-ca4048a4a7d6 | jq

This request returns either true or false. If true, the lock was acquired and the local service instance is now the leader. If false, a different node acquired the lock.

$ consul kv put -acquire -session=d21d60ad-c2d2-b32a-7432-ca4048a4a7d6 /service/leader '{"Node": "'`hostname`'"}'

In case of success, the command exits with exit code 0 and outputs the following message.

Success! Lock acquired on: service/leader

If the lock was already acquired by another node, the command exits with exit code 1 and outputs the following message.

Error! Did not acquire lock

This example used the node's hostname as the key data. This data can be used by the other services to create configuration files.

Be aware that this locking system has no enforcement mechanism that requires clients to acquire a lock before they perform an operation. Any client can read, write, and delete a key without owning the corresponding lock.

Watch the KV key for locks

Existing locks need to be monitored by all nodes involved in the client-side leader elections, as well as by the other nodes that need to know the identity of the leader.

Lock holders need to monitor the lock because the session might get invalidated by an operator.
Other services that want to acquire the lock need to monitor it to check if the lock is released so they can try acquire the lock.
Other nodes need to monitor the lock to see if the value of the key changed and update their configuration accordingly.

Monitor the lock using the GET method on a KV entry with the blocking query enabled.

First, verify the latest index for the current value.

$ curl --silent \
  --header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
  --request GET \
  http://127.0.0.1:8500/v1/kv/service/leader?index=1 | jq

The command outputs the key data, including the ModifyIndex for the object.

[
  {
    "LockIndex": 0,
    "Key": "service/leader",
    "Flags": 0,
    "Value": "eyJOb2RlIjogImhhc2hpY3Vwcy1kYi0wIn0=",
    "Session": "d21d60ad-c2d2-b32a-7432-ca4048a4a7d6",
    "CreateIndex": 12399,
    "ModifyIndex": 13061
  }
]

Using the value of the ModifyIndex, run a blocking query against the lock.

$ curl --silent \
  --header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
  --request GET \
  http://127.0.0.1:8500/v1/kv/service/leader?index=13061 | jq

The command hangs until a change is made on the KV path and after that the path data prints on the console.

[
  {
    "LockIndex": 0,
    "Key": "service/leader",
    "Flags": 0,
    "Value": "eyJOb2RlIjogImhhc2hpY3Vwcy1kYi0xIn0=",
    "Session": "d21d60ad-c2d2-b32a-7432-ca4048a4a7d6",
    "CreateIndex": 12399,
    "ModifyIndex": 13329
  }
]

For automation purposes, add logic to the blocking query mechanism to trigger a command every time a change is returned. A better approach is to use the CLI command consul watch.

Monitor the lock using the consul watch command.

$ consul watch -type=key -key=service/leader cat | jq

In this example, the command output prints to the shell. However, it is possible to pass more complex option to the command as well as a script that contains more complex logic to react to the lock data change.

An example output for the command is:

{
  "Key": "service/leader",
  "CreateIndex": 12399,
  "ModifyIndex": 13061,
  "LockIndex": 0,
  "Flags": 0,
  "Value": "eyJOb2RlIjogImhhc2hpY3Vwcy1kYi0wIn0=",
  "Session": "d21d60ad-c2d2-b32a-7432-ca4048a4a7d6"
}

The consul watch command polls the KV path for changes and runs the specified command on the output when a change is made.

From the output, notice that once the lock is acquired, the Session parameter contains the ID of the session that holds the lock.

Renew a session

If a session is created with a TTL value set, you need to renew the session before the TTL expires.

Use the /v1/session/renew endpoint to renew existing sessions.

$ curl --silent \
  --header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
  --request PUT \
  http://127.0.0.1:8500/v1/session/renew/f027470f-2759-6b53-542d-066ae4185e67 | jq

If the command succeeds, the session information in JSON format is printed.

[
  {
    "ID": "f027470f-2759-6b53-542d-066ae4185e67",
    "Name": "test",
    "Node": "consul-server-0",
    "LockDelay": 15000000000,
    "Behavior": "release",
    "TTL": "30s",
    "NodeChecks": [
      "serfHealth"
    ],
    "ServiceChecks": null,
    "CreateIndex": 11842,
    "ModifyIndex": 11842
  }
]

Release a lock

A lock associated with a session with no TTL value set might never be released, even when the service holding it fails.

In such cases, you need to manually release the lock.

$ curl --silent \
  --header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
  --data '{"Node": "'`hostname`'"}' \
  --request PUT \
  http://localhost:8500/v1/kv/service/leader?release=d21d60ad-c2d2-b32a-7432-ca4048a4a7d6 | jq

The command prints true on success.

$ consul kv put -release -session=d21d60ad-c2d2-b32a-7432-ca4048a4a7d6 service/leader '{"Node": "'`hostname`'"}'

On success, the command outputs a success message.

Success! Lock released on: service/leader

After a lock is released, the key data do not show a value for Session in the results. Other clients can use this as a way to coordinate their lock requests.