Audit devices

Audit devices are the components in Vault that collectively keep a detailed log of all requests to Vault, and their responses. Because every operation with Vault is an API request/response, when using a single audit device, the audit log contains every interaction with the Vault API, including errors - except for a few paths which do not go via the audit system.

The non-audited paths are:

sys/init
sys/seal-status
sys/seal
sys/step-down
sys/unseal
sys/leader
sys/health
sys/rekey/init
sys/rekey/update
sys/rekey/verify
sys/rekey-recovery-key/init
sys/rekey-recovery-key/update
sys/rekey-recovery-key/verify
sys/storage/raft/bootstrap
sys/storage/raft/join
sys/internal/ui/feature-flags

and also, if the relevant listener configuration settings allow unauthenticated access:

sys/metrics
sys/pprof/*
sys/in-flight-req

Enabling multiple devices

When multiple audit devices are enabled, Vault will attempt to send the audit logs to all of them. This allows you to not only have redundant copies, but also a way to check for data tampering in the logs themselves.

Vault considers a request to be successful if it can log to at least one configured audit device (see: Blocked Audit Devices section below). Therefore in order to build a complete picture of all audited actions, use the aggregate/union of the logs from each audit device.

Note: It is highly recommended that you configure Vault to use multiple audit devices. Audit failures can prevent Vault from servicing requests, so it is important to provide at least one other device.

Format

Each line in the audit log is a JSON object. The type field specifies what type of object it is. Currently, only two types exist: request and response. The line contains all of the information for any given request and response. By default, all the sensitive information is first hashed before logging in the audit logs.

Sensitive information

The audit logs contain the full request and response objects for every interaction with Vault. The request and response can be matched utilizing a unique identifier assigned to each request.

Most strings contained within requests and responses are hashed with a salt using HMAC-SHA256. The purpose of the hash is so that secrets aren't in plaintext within your audit logs. However, you're still able to check the value of secrets by generating HMACs yourself; this can be done with the audit device's hash function and salt by using the /sys/audit-hash API endpoint (see the documentation for more details).

Currently, only strings that come from JSON or returned in JSON are HMAC'd. Other data types, like integers, booleans, and so on, are passed through in plaintext. We recommend that all sensitive data be provided as string values inside all JSON sent to Vault (i.e., that integer values are provided in quotes).

While most strings are hashed, Vault can be configured to make some exceptions. For example in auth methods and secrets engines, users can enable additional exceptions using the secrets enable command, and then tune it afterward.

see also:

auth enable

auth tune

Audit request headers

Use the Vault API to configure request headers to monitor and audit headers in incoming client request.

By default, Vault does not encrypt request header values with HMAC if you create an exception to allow request headers in the audit log. To encrypt the header values, you must configure the relevant headers individually.

Default headers

To help correlate requests across distributed systems, Vault automatically records the following headers in the audit log:

To ensure Vault uses HMAC on the header values during logging, set the hmac value to true for the config/auditing/request-headers API call.

For example, to enable HMAC for correlation-id

curl \
    --header "X-Vault-Token: ..." \
    http://127.0.0.1:8200/v1/sys/config/auditing/request-headers/correlation-id \
    --data '{ "hmac": true }'

Another way to identify the source of a request is through the User-Agent request header. Vault will automatically record this value as user-agent within the headers of a request entry within the audit log.

Enabling/Disabling audit devices

When a Vault server is first initialized, no auditing is enabled. Audit devices must be enabled by a root user using vault audit enable.

When enabling an audit device, options can be passed to it to configure it. For example, the command below enables the file audit device:

$ vault audit enable file file_path=/var/log/vault_audit.log

In the command above, we passed the "file_path" parameter to specify the path where the audit log will be written to. Each audit device has its own set of parameters. See the documentation to the left for more details.

Note: Audit device configuration is replicated to all nodes within a cluster by default, and to performance/DR secondaries for Vault Enterprise clusters. Before enabling an audit device, ensure that all nodes within the cluster(s) will be able to successfully log to the audit device to avoid Vault being blocked from serving requests. An audit device can be limited to only within the node's cluster with the local parameter.

When an audit device is disabled, it will stop receiving logs immediately. The existing logs that it did store are untouched.

Note: Once an audit device is disabled, you will no longer be able to HMAC values for comparison with entries in the audit logs. This is true even if you re-enable the audit device at the same path, as a new salt will be created for hashing.

Blocked audit devices

Audit device logs are critically important and ignoring auditing failures opens an avenue for attack. Vault will not respond to requests when no enabled audit devices can record them.

Vault can distinguish between two types of audit device failures.

A blocking failure is one where an attempt to write to the audit device never completes. This is unlikely with a local disk device, but could occure with a network-based audit device.
When multiple audit devices are enabled, if any of them fail in a non-blocking fashion, Vault requests can still complete successfully provided at least one audit device successfully writes the audit record. If any of the audit devices fail in a blocking fashion however, Vault requests will hang until the blocking is resolved.

In other words, Vault will not complete any requests until the blocked audit device can write.

Tutorial

Refer to Blocked Audit Devices for a step-by-step tutorial.

API

Audit devices also have a full HTTP API. Please see the Audit device API docs for more details.

Common configuration options

elide_list_responses (bool: false) - See Eliding list response bodies.
exclude (string: "") - Enterprise Defines a set of rules such that, when the condition (optional) is matched, Vault removes the specified fields from the audit entry before writing to the audit log. Refer to the exclusion section of the auditing overview for more information.
fallback (bool: false) - Enterprise Indicates whether the audit device is the fallback for filtering purposes. Vault only supports one fallback audit device at a time.
filter (string: "") - Enterprise Sets an optional string used to filter the audit entries logged by the audit device. See the filtering section of the auditing overview for more information.
format (string: "json") - Allows selecting the output format. Valid values are "json" and "jsonx", which formats the normal log entries as XML.
hmac_accessor (bool: true) - If enabled, enables the hashing of token accessor.
log_raw (bool: false) - If enabled, logs the security sensitive information without hashing, in the raw format.
prefix (string: "") - A customizable string prefix to write before the actual log line.

Eliding list response bodies

Some Vault responses can be very large. Primarily, this affects list operations - as Vault lacks pagination in its APIs, listing a very large collection can result in a response that is tens of megabytes long. Some audit backends are unable to process individual audit records of larger sizes.

The contents of the response for a list operation is often not very interesting; most contain only a "keys" field, containing a list of IDs. Select API endpoints additionally return a "key_info" field, a map from ID to some additional information about the list entry - identity/entity/id/ is an example of this. Even in this case, the response to a list operation is usually less-confidential or public information, for which having the full response in the audit logs is of lesser importance.

The elide_list_responses audit option provides the flexibility to not write the full list response data from the audit log, to mitigate the creation of very long individual audit records.

When enabled, it affects only audit records of type=response and request.operation=list. The values of response.data.keys and response.data.key_info will be replaced with a simple integer, recording how many entries were contained in the list (keys) or map (key_info) - therefore even with this feature enabled, it is still possible to see how many items were returned by a list operation.

This extra processing only affects the response data fields keys and key_info, and only when they have the expected data types - in the event a list response contains data outside of the usual conventions that apply to Vault list responses, it will be left as is by this feature.

Here is an example of an audit record that has been processed by this feature (formatted with extra whitespace, and with fields not relevant to the example omitted):

{
  "type": "response",
  "request": {
    "operation": "list"
  },
  "response": {
    "data": {
      "key_info": 4,
      "keys": 4
    }
  }
}