HashiConf 2024 Now streaming live from Boston! Attend for free
Consul
Consul Home

Prepared Query HTTP Endpoint

The /query endpoints create, update, destroy, and execute prepared queries.

Prepared queries allow you to register a complex service query and then execute it later by specifying the query ID or name. Consul returns a set of healthy nodes that provide a given service. Refer to Enable Dynamic DNS Queries for additional information.

Check the Geo Failover tutorial for details and examples for using prepared queries to implement geo failover for services.

Check the prepared query rules section of the agent ACL documentation for more details about how prepared queries work with Consul's ACL system.

Prepared Query Templates

Consul 0.6.4 and later support prepared query templates. These are created similar to static queries, except with some additional fields and features. Here is an example prepared query template:

{
  "Template": {
    "Type": "name_prefix_match",
    "Regexp": "^geo-db-(.*?)-([^\\-]+?)$",
    "RemoveEmptyTags": false
  }
}

The Template structure configures a prepared query as a template instead of a static query. It has two fields:

  • Type is the query type, which must be name_prefix_match. This means that the template will apply to any query lookup with a name whose prefix matches the Name field of the template. In this example, any query for geo-db will match this query. Query templates are resolved using a longest prefix match, so it's possible to have high-level templates that are overridden for specific services. Static queries are always resolved first, so they can also override templates.

  • Regexp is an optional regular expression which is used to extract fields from the entire name, once this template is selected. In this example, the regular expression takes the first item after the "-" as the database name and everything else after as a tag. See the RE2 reference for syntax of this regular expression.

  • RemoveEmptyTags is optional, and if set to true, will cause the Tags list inside the Service structure to be stripped of any empty strings. This defaults to false, meaning that empty strings will remain in the list. This is useful when interpolating into tags in a way where the tag is optional, and where searching for an empty tag would yield no results from the query.

All other fields of the query have the same meanings as for a static query, except that several interpolation variables are available to dynamically populate the query before it is executed. All of the string fields inside the Service structure are interpolated, with the following variables available:

  • ${name.full} has the entire name that was queried. For example, a DNS lookup for geo-db-customer-primary.query.consul in the example above would set this variable to geo-db-customer-primary.

  • ${name.prefix} has the prefix that matched. This would always be geo-db for the example above.

  • ${name.suffix} has the suffix after the prefix. For example, a DNS lookup for geo-db-customer-primary.query.consul in the example above would set this variable to -customer-primary.

  • ${match(N)} returns the regular expression match at the given index N. The 0 index will have the entire match, and >0 will have the results of each match group. For example, a DNS lookup for geo-db-customer-primary.query.consul in the example above with a Regexp field set to ^geo-db-(.*?)-([^\-]+?)$ would return geo-db-customer-primary for ${match(0)}, customer for ${match(1)}, and primary for ${match(2)}. If the regular expression doesn't match, or an invalid index is given, then ${match(N)} will return an empty string.

  • ${agent.segment} Enterprise - the network segment of the agent that initiated the query. This variable can be used with the NodeMeta field to limit the results of a query to service instances within its own network segment:

    {
  "Name": "",
  "Template": {
    "Type": "name_prefix_match"
  },
  "Service": {
    "Service": "${name.full}",
    "NodeMeta": { "consul-network-segment": "${agent.segment}" }
  }
}

    This will map all names of the form <service>.query.consul over DNS to a query that will select an instance of the service in the agent's own network segment.

Using templates, it is possible to apply prepared query behaviors to many services with a single template. Here's an example template that matches any query and applies a failover policy to it:

{
  "Name": "",
  "Template": {
    "Type": "name_prefix_match"
  },
  "Service": {
    "Service": "${name.full}",
    "Failover": {
      "NearestN": 3
    }
  }
}

This will match any lookup for *.query.consul and will attempt to find the service locally, and otherwise attempt to find that service in the next three closest datacenters. If ACLs are enabled, a catch-all template like this with an empty Name requires an ACL token that can write to any query prefix. Also, only a single catch-all template can be registered at any time.

Create Prepared Query

This endpoint creates a new prepared query and returns its ID if it is created successfully.

MethodPathProduces
POST/queryapplication/json

The table below shows this endpoint's support for blocking queries, consistency modes, agent caching, and required ACLs.

Blocking QueriesConsistency ModesAgent CachingACL Required
NOnonenonequery:write

Query Parameters

  • dc (string: "") - Specifies the datacenter to query. This will default to the datacenter of the agent being queried.

JSON Request Body Schema

  • Name (string: "") - Specifies an optional friendly name that can be used to execute a query instead of using its ID.

  • Session (string: "") - Specifies the ID of an existing session. This provides a way to automatically remove a prepared query when the given session is invalidated. If not given the prepared query must be manually removed when no longer needed.

  • Token (string: "") - Specifies the ACL token to use each time the query is executed. This allows queries to be executed by clients with lesser or even no ACL Token, so this should be used with care. The token itself can only be seen by clients with a management token. If the Token field is left blank or omitted, the client's ACL Token will be used to determine if they have access to the service being queried. If the client does not supply an ACL Token, the anonymous token will be used.

  • Service (Service: <required>) - Specifies the structure to define the query's behavior.

    • Service (string: <required>) - Specifies the name of the service to query.

    • SamenessGroup (string: "") Enterprise - Specifies a Sameness group to forward the query to. The SamenessGroup will forward to its members in the order defined, returning on the first healthy query. SamenessGroup is mutually exclusive with Failover as SamenessGroup members will be used in place of a defined list of failovers.

    • Namespace (string: "") Enterprise - Specifies the Consul namespace to query. If not provided the query will use Consul default namespace for resolution. When combined with SamenessGroup this will specify the namespaces in which the SamenessGroup will resolve all members listed.

    • Partition (string: "") Enterprise - Specifies the Consul partition to query. If not provided the query will use Consul's default partition for resolution. When combined with SamenessGroup, this will specify the partition where the SamenessGroup exists.

    • Failover Determines what happens if no healthy nodes are available in the local datacenter when the query is executed. It allows the use of nodes in other datacenters with very little configuration. This field is mutually exclusive with SamenessGroup.

      • NearestN (int: 0) - Specifies that the query will be forwarded to up to NearestN other datacenters based on their estimated network round trip time using Network Coordinates from the WAN gossip pool. The median round trip time from the server handling the query to the servers in the remote datacenter is used to determine the priority.

      • Datacenters (array<string>: nil) - Specifies a fixed list of WAN federated datacenters to forward the query to when there are no healthy nodes in the local datacenter. Datacenters are queried in the order given in the list. If this option is combined with NearestN, then the NearestN queries will be performed first, followed by the list given by Datacenters. A given datacenter will only be queried one time during a failover, even if it is selected by both NearestN and is listed in Datacenters. Use Targets to failover to cluster peers.

      • Targets (array<Target>: nil) - Specifies a sequential list of remote datacenters and cluster peers to failover to if there are no healthy service instances in the local datacenter. This option cannot be used with NearestN or Datacenters.

        • Peer (string: "") - Specifies a cluster peer to use for failover.

        • Datacenter (string: "") - Specifies a WAN federated datacenter to forward the query to.

        • Partition (string: "") Enterprise - Specifies a Partition to forward the query to.

        • Namespace (string: "") Enterprise - Specifies a Namespace to forward the query to.

    • IgnoreCheckIDs (array<string>: nil) - Specifies a list of check IDs that should be ignored when filtering unhealthy instances. This is mostly useful in an emergency or as a temporary measure when a health check is found to be unreliable. Being able to ignore it in centrally-defined queries can be simpler than deregistering the check as an interim solution until the check can be fixed.

    • OnlyPassing (bool: false) - Specifies the behavior of the query's health check filtering. If this is set to false, the results will include nodes with checks in the passing as well as the warning states. If this is set to true, only nodes with checks in the passing state will be returned.

    • Near (string: "") - Specifies a node to sort near based on distance sorting using Network Coordinates. The nearest instance to the specified node will be returned first, and subsequent nodes in the response will be sorted in ascending order of estimated round-trip times. If the node given does not exist, the nodes in the response will be shuffled. If unspecified, the response will be shuffled by default.

      • _agent - Returns results nearest the agent servicing the request.
      • _ip - Returns results nearest to the node associated with the source IP where the query was executed from. For HTTP the source IP is the remote peer's IP address or the value of the X-Forwarded-For header with the header taking precedence. For DNS the source IP is the remote peer's IP address or the value of the EDNS client IP with the EDNS client IP taking precedence.

  • Tags (array<string>: nil) - Specifies a list of service tags to filter the query results. For a service to pass the tag filter it must have all of the required tags, and none of the excluded tags (prefixed with !).

  • NodeMeta (map<string|string>: nil) - Specifies a list of user-defined key/value pairs that will be used for filtering the query results to nodes with the given metadata values present.

  • ServiceMeta (map<string|string>: nil) - Specifies a list of user-defined key/value pairs that will be used for filtering the query results to services with the given metadata values present.

  • Connect (bool: false) - If true, only mesh-capable services for the specified service name will be returned. This includes both natively integrated services and proxies. For proxies, the proxy name may not match Service, because the proxy destination will. Any constraints beyond the service name such as Near, Tags, and NodeMeta are applied to mesh-capable service.

  • DNS (DNS: nil) - Specifies DNS configuration

    • TTL (string: "") - Specifies the TTL duration when query results are served over DNS. If this is specified, it will take precedence over any Consul agent-specific configuration.

Sample Payload

{
  "Name": "my-query",
  "Session": "adf4238a-882b-9ddc-4a9d-5b6758e4159e",
  "Token": "",
  "Service": {
    "Service": "redis",
    "Failover": {
      "NearestN": 3,
      "Datacenters": ["dc1", "dc2"]
    },
    "Near": "node1",
    "OnlyPassing": false,
    "Tags": ["primary", "!experimental"],
    "NodeMeta": { "instance_type": "m3.large" },
    "ServiceMeta": { "environment": "production" }
  },
  "DNS": {
    "TTL": "10s"
  }
}

Sample Request

$ curl \
    --request POST \
    --data @payload.json \
    http://127.0.0.1:8500/v1/query

Sample Response

{
  "ID": "8f246b77-f3e1-ff88-5b48-8ec93abf3e05"
}

List Prepared Queries

This endpoint returns a list of all prepared queries.

The HTTP response includes the X-Consul-Results-Filtered-By-ACLs: true header if the response array excludes results due to ACL policy configuration. Refer to the HTTP API documentation for more information.

MethodPathProduces
GET/queryapplication/json

The table below shows this endpoint's support for blocking queries, consistency modes, agent caching, and required ACLs.

Blocking QueriesConsistency ModesAgent CachingACL Required
NOallnonequery:read

Query Parameters

  • dc (string: "") - Specifies the datacenter to query. This will default to the datacenter of the agent being queried.

Sample Request

$ curl \
    http://127.0.0.1:8500/v1/query

Sample Response

[
  {
    "ID": "8f246b77-f3e1-ff88-5b48-8ec93abf3e05",
    "Name": "my-query",
    "Session": "adf4238a-882b-9ddc-4a9d-5b6758e4159e",
    "Token": "<hidden>",
    "Service": {
      "Service": "redis",
      "Failover": {
        "NearestN": 3,
        "Datacenters": ["dc1", "dc2"]
      },
      "OnlyPassing": false,
      "Tags": ["primary", "!experimental"],
      "NodeMeta": { "instance_type": "m3.large" },
      "ServiceMeta": { "environment": "production" }
    },
    "DNS": {
      "TTL": "10s"
    },
    "RaftIndex": {
      "CreateIndex": 23,
      "ModifyIndex": 42
    }
  }
]

Update Prepared Query

This endpoint updates an existing prepared query. If no query exists by the given ID, an error is returned.

MethodPathProduces
PUT/query/:uuidapplication/json

The table below shows this endpoint's support for blocking queries, consistency modes, agent caching, and required ACLs.

Blocking QueriesConsistency ModesAgent CachingACL Required
NOnonenonequery:write

Path Parameters

  • uuid (string: <required>) - Specifies the UUID of the query to update.

Query Parameters

  • dc (string: "") - Specifies the datacenter to query. This will default to the datacenter of the agent being queried.

The body is the same as is used to create a prepared query. Please see above for more information.

Sample Request

$ curl \
    --request PUT \
    --data @payload.json \
    http://127.0.0.1:8500/v1/query/8f246b77-f3e1-ff88-5b48-8ec93abf3e05

Read Prepared Query

This endpoint reads an existing prepared query. If no query exists by the given ID, an error is returned.

MethodPathProduces
GET/query/:uuidapplication/json

The table below shows this endpoint's support for blocking queries, consistency modes, agent caching, and required ACLs.

Blocking QueriesConsistency ModesAgent CachingACL Required
NOallnonequery:read

Path Parameters

  • uuid (string: <required>) - Specifies the UUID of the query to read.

Query Parameters

  • dc (string: "") - Specifies the datacenter to query. This will default to the datacenter of the agent being queried.

Sample Request

$ curl \
  http://127.0.0.1:8500/v1/query/8f246b77-f3e1-ff88-5b48-8ec93abf3e05

Sample Response

The returned response is the same as the list of prepared queries above, only with a single item present.

Delete Prepared Query

This endpoint deletes an existing prepared query. If no query exists by the given ID, an error is returned.

MethodPathProduces
DELETE/query/:uuidapplication/json

The table below shows this endpoint's support for blocking queries, consistency modes, agent caching, and required ACLs.

Blocking QueriesConsistency ModesAgent CachingACL Required
NOnonenonequery:write

Path Parameters

  • uuid (string: <required>) - Specifies the UUID of the query to delete.

Query Parameters

  • dc (string: "") - Specifies the datacenter to query. This will default to the datacenter of the agent being queried.

Sample Request

$ curl \
    --request DELETE \
    http://127.0.0.1:8500/v1/query/8f246b77-f3e1-ff88-5b48-8ec93abf3e05

Execute Prepared Query

This endpoint executes an existing prepared query. If no query exists by the given ID, an error is returned.

The HTTP response includes the X-Consul-Results-Filtered-By-ACLs: true header if the Nodes response array excludes results due to ACL policy configuration. Refer to the HTTP API documentation for more information.

MethodPathProduces
GET/query/:uuid/executeapplication/json

The table below shows this endpoint's support for blocking queries, consistency modes, agent caching, and required ACLs.

Blocking QueriesConsistency ModesAgent CachingACL Required
NOallsimpledepends1
1 If an ACL Token was bound to the query when it was defined then it will be used when executing the request. Otherwise, the client's supplied ACL Token will be used.

Path Parameters

  • uuid (string: <required>) - Specifies the UUID of the query to execute. This parameter can also be the name of an existing prepared query, or a name that matches a prefix name for a prepared query template.

Query Parameters

  • dc (string: "") - Specifies the datacenter to query. This will default to the datacenter of the agent being queried.

  • near (string: "") - Specifies to sort the resulting list in ascending order based on the estimated round trip time from that node. Passing ?near=_agent will use the agent's node for the sort. Passing ?near=_ip will use the source IP of the request or the value of the X-Forwarded-For header to lookup the node to use for the sort. If this is not present, the default behavior will shuffle the nodes randomly each time the query is executed.

  • limit (int: 0) - Limit the size of the list to the given number of nodes. This is applied after any sorting or shuffling.

  • connect (bool: false) - If true, limit results to nodes that are mesh-capable only. This parameter can also be specified directly on the template itself to force all executions of a query to be mesh-only. See the template documentation for more information.

Sample Request

$ curl \
    http://127.0.0.1:8500/v1/query/8f246b77-f3e1-ff88-5b48-8ec93abf3e05/execute?near=_agent

Sample Response

{
  "Service": "redis",
  "Nodes": [
    {
      "Node": {
        "ID": "40e4a748-2192-161a-0510-9bf59fe950b5",
        "Node": "foobar",
        "Address": "10.1.10.12",
        "Datacenter": "dc1",
        "TaggedAddresses": {
          "lan": "10.1.10.12",
          "wan": "10.1.10.12"
        },
        "NodeMeta": { "instance_type": "m3.large" }
      },
      "Service": {
        "ID": "redis",
        "Service": "redis",
        "Tags": null,
        "Meta": { "redis_version": "4.0" },
        "Port": 8000
      },
      "Checks": [
        {
          "Node": "foobar",
          "CheckID": "service:redis",
          "Name": "Service 'redis' check",
          "Status": "passing",
          "Notes": "",
          "Output": "",
          "ServiceID": "redis",
          "ServiceName": "redis"
        },
        {
          "Node": "foobar",
          "CheckID": "serfHealth",
          "Name": "Serf Health Status",
          "Status": "passing",
          "Notes": "",
          "Output": "",
          "ServiceID": "",
          "ServiceName": ""
        }
      ],
      "DNS": {
        "TTL": "10s"
      },
      "Datacenter": "dc3",
      "Failovers": 2
    }
  ]
}

  • Nodes contains the list of healthy nodes providing the given service, as specified by the constraints of the prepared query.

  • Service has the service name that the query was selecting. This is useful for context in case an empty list of nodes is returned.

  • DNS has information used when serving the results over DNS. This is just a copy of the structure given when the prepared query was created.

  • Datacenter has the datacenter that ultimately provided the list of nodes and Failovers has the number of remote datacenters that were queried while executing the query. This provides some insight into where the data came from. This will be zero during non-failover operations where there were healthy nodes found in the local datacenter.

Explain Prepared Query

This endpoint generates a fully-rendered query for a given name, post interpolation.

MethodPathProduces
GET/query/:uuid/explainapplication/json

The table below shows this endpoint's support for blocking queries, consistency modes, agent caching, and required ACLs.

Blocking QueriesConsistency ModesAgent CachingACL Required
NOallnonequery:read

Path Parameters

  • uuid (string: <required>) - Specifies the UUID of the query to explain. This parameter can also be the name of an existing prepared query, or a name that matches a prefix name for a prepared query template.

Query Parameters

  • dc (string: "") - Specifies the datacenter to query. This will default to the datacenter of the agent being queried.

Sample Request

$ curl \
    http://127.0.0.1:8500/v1/query/8f246b77-f3e1-ff88-5b48-8ec93abf3e05/explain

Sample Response

{
  "Query": {
    "ID": "8f246b77-f3e1-ff88-5b48-8ec93abf3e05",
    "Name": "my-query",
    "Session": "adf4238a-882b-9ddc-4a9d-5b6758e4159e",
    "Token": "<hidden>",
    "Name": "geo-db",
    "Template": {
      "Type": "name_prefix_match",
      "Regexp": "^geo-db-(.*?)-([^\\-]+?)$"
    },
    "Service": {
      "Service": "mysql-customer",
      "Failover": {
        "NearestN": 3,
        "Datacenters": ["dc1", "dc2"]
      },
      "OnlyPassing": true,
      "Tags": ["primary"],
      "Meta": { "mysql_version": "5.7.20" },
      "NodeMeta": { "instance_type": "m3.large" }
    }
  }
}
Edit this page on GitHub