Write a policy using audit logs

8min
|
Vault

Every Vault operation performed through the command-line interface (CLI), API, or web UI require that the authenticated client is granted access; access defined through policies. Everything in Vault is stored at different paths, like a filesystem, and every action in Vault has a corresponding path and capability. Policies provide a declarative way to grant or forbid access to paths and the capabilities at each path.

In this tutorial, you will learn Vault's policy language and how to query an audit log to define policies.

Prerequisites

Vault installed.
jq for consuming Vault JSON formatted output and capturing specific fields.

Start Vault

Start a Vault dev server with root as the root token.

$ vault server -dev -dev-root-token-id root

The Vault dev server defaults to running at 127.0.0.1:8200. The server is initialized and unsealed.

Insecure operation

Do not run a Vault dev server in production. This approach starts a Vault server with an in-memory database and runs in an insecure way.

In another terminal, export an environment variable for the vault CLI to address the Vault server.

$ export VAULT_ADDR=http://127.0.0.1:8200

Export an environment variable for the vault CLI to authenticate with the Vault server.

$ export VAULT_TOKEN=root

The Vault server is ready.

Enable audit logs

Audit devices keep a detailed log of all requests and response to Vault. The file audit device appends these events to a file.

Audit device filters

Starting in Vault 1.16.0, you can enable audit devices with a filter option that Vault uses to evaluate audit entries to determine whether it writes them to the log. You should determine if your own audit devices are filtered and make necessary changes to expose the log fields which you need to monitor for your use case.

You can familiarize yourself with Vault filtering concepts and filtering audit entries and how to enable audit filters in the documentation.

Enable the file audit device at a file named vault-audit.log in the current directory.

$ vault audit enable file file_path=$PWD/vault-audit.log
Success! Enabled the file audit device at: file/

The file audit device is enabled and immediately appends a test message to the audit log file.

Display the contents of the audit log file.

$ cat vault-audit.log | jq

The output displays two entries similar to these examples.

{
  "time": "2023-06-21T16:53:09.915468Z",
  "type": "request",
  "auth": {
    "token_type": "default"
  },
  "request": {
    "id": "3f11991f-98d6-d88e-94db-f3e1dfab0c0d",
    "operation": "update",
    "namespace": {
      "id": "root"
    },
    "path": "sys/audit/test"
  }
}
{
  "time": "2023-06-21T16:53:09.925391Z",
  "type": "response",
  "auth": {
    "client_token": "hmac-sha256:2538d1fb96fe45b811af0c6db2c2cbe78e4f10d1d3ad0c8eb6f7dd36828587bb",
    "accessor": "hmac-sha256:defd1a0c008c6438b48d733578ca7f50f978c2db2750a1e508b4d1d8674d670d",
    "display_name": "token",
    "policies": [
      "root"
    ],
    "token_policies": [
      "root"
    ],
    "policy_results": {
      "allowed": true,
      "granting_policies": [
        {
          "name": "root",
          "namespace_id": "root",
          "type": "acl"
        }
      ]
    },
    "token_type": "service",
    "token_issue_time": "2023-06-21T12:52:53-04:00"
  },
  "request": {
    "id": "e11d675e-6e83-1ce1-48a4-8f82869ccb9b",
    "client_id": "0DHqvq2D77kL2/JTPSZkTMJbkFVmUu0TzMi0jiXcFy8=",
    "operation": "update",
    "mount_point": "sys/",
    "mount_type": "system",
    "mount_accessor": "system_18f60a0c",
    "mount_running_version": "v1.14.0+builtin.vault",
    "mount_class": "secret",
    "client_token": "hmac-sha256:2538d1fb96fe45b811af0c6db2c2cbe78e4f10d1d3ad0c8eb6f7dd36828587bb",
    "client_token_accessor": "hmac-sha256:defd1a0c008c6438b48d733578ca7f50f978c2db2750a1e508b4d1d8674d670d",
    "namespace": {
      "id": "root"
    },
    "path": "sys/audit/file",
    "data": {
      "description": "hmac-sha256:5f9da19df55a1a3f82d640bbf552847a11da9bfd962ef63cbc4848e78bb36ad4",
      "local": false,
      "options": {
        "file_path": "hmac-sha256:b4a151e47019f98aab7b38f0af27bd967cad01d863cb83c75ddb07817029bb02"
      },
      "type": "hmac-sha256:7967d1e46655340a7dfce10305eebea1a80d70cef13e9b064c84b9e4b415b6a6"
    },
    "remote_address": "127.0.0.1",
    "remote_port": 51132
  },
  "response": {
    "mount_point": "sys/",
    "mount_type": "system",
    "mount_accessor": "system_18f60a0c",
    "mount_running_plugin_version": "v1.14.0+builtin.vault",
    "mount_class": "secret"
  }
}

The audit log appends JSON objects to the log file. The first object describes an audit test message. The second object describes the operation that enabled the audit log.

Enable transit secrets engine

The transit secrets engine handles cryptographic functions on data in-transit through keys that Vault manages.

Enable the transit secrets engine at the default path.

$ vault secrets enable transit
Success! Enabled the transit secrets engine at: transit/

The transit secrets engine is enabled and the operation that performed it was written to the audit log.

View only the last entry in the audit log file.

$ cat vault-audit.log | jq -s ".[-1]"

The output displays an entry similar to this example.

{
  "time": "2023-06-21T16:54:04.026976Z",
  "type": "response",
  "auth": {
    "client_token": "hmac-sha256:2538d1fb96fe45b811af0c6db2c2cbe78e4f10d1d3ad0c8eb6f7dd36828587bb",
    "accessor": "hmac-sha256:defd1a0c008c6438b48d733578ca7f50f978c2db2750a1e508b4d1d8674d670d",
    "display_name": "token",
    "policies": [
      "root"
    ],
    "token_policies": [
      "root"
    ],
    "policy_results": {
      "allowed": true,
      "granting_policies": [
        {
          "name": "root",
          "namespace_id": "root",
          "type": "acl"
        }
      ]
    },
    "token_type": "service",
    "token_issue_time": "2023-06-21T12:52:53-04:00"
  },
  "request": {
    "id": "8804bf89-b0ab-8368-dc06-52a5f3fbb702",
    "client_id": "0DHqvq2D77kL2/JTPSZkTMJbkFVmUu0TzMi0jiXcFy8=",
    "operation": "update",
    "mount_point": "sys/",
    "mount_type": "system",
    "mount_accessor": "system_18f60a0c",
    "mount_running_version": "v1.14.0+builtin.vault",
    "mount_class": "secret",
    "client_token": "hmac-sha256:2538d1fb96fe45b811af0c6db2c2cbe78e4f10d1d3ad0c8eb6f7dd36828587bb",
    "client_token_accessor": "hmac-sha256:defd1a0c008c6438b48d733578ca7f50f978c2db2750a1e508b4d1d8674d670d",
    "namespace": {
      "id": "root"
    },
    "path": "sys/mounts/transit",
    "data": {
      "config": {
        "default_lease_ttl": "hmac-sha256:8ed0513cb3a7070d1f2a5ed643c1b0f455e887b66b82204085ca86c3661b26b4",
        "force_no_cache": false,
        "max_lease_ttl": "hmac-sha256:8ed0513cb3a7070d1f2a5ed643c1b0f455e887b66b82204085ca86c3661b26b4",
        "options": null
      },
      "description": "hmac-sha256:5f9da19df55a1a3f82d640bbf552847a11da9bfd962ef63cbc4848e78bb36ad4",
      "external_entropy_access": false,
      "local": false,
      "options": null,
      "seal_wrap": false,
      "type": "hmac-sha256:c27c9dc8f2e0a07cad1e12e722503879100baba3f859ec123ce22755a049ebf1"
    },
    "remote_address": "127.0.0.1",
    "remote_port": 51188
  },
  "response": {
    "mount_point": "sys/",
    "mount_type": "system",
    "mount_accessor": "system_18f60a0c",
    "mount_running_plugin_version": "v1.14.0+builtin.vault",
    "mount_class": "secret"
  }
}

The object describes the time, the authorized token, the request, and the response.

Display the request of the last logged object.

$ cat vault-audit.log | jq -s ".[-1].request"

The output displays an entry similar to this example.

{
  "id": "8804bf89-b0ab-8368-dc06-52a5f3fbb702",
  "client_id": "0DHqvq2D77kL2/JTPSZkTMJbkFVmUu0TzMi0jiXcFy8=",
  "operation": "update",
  "mount_point": "sys/",
  "mount_type": "system",
  "mount_accessor": "system_18f60a0c",
  "mount_running_version": "v1.14.0+builtin.vault",
  "mount_class": "secret",
  "client_token": "hmac-sha256:2538d1fb96fe45b811af0c6db2c2cbe78e4f10d1d3ad0c8eb6f7dd36828587bb",
  "client_token_accessor": "hmac-sha256:defd1a0c008c6438b48d733578ca7f50f978c2db2750a1e508b4d1d8674d670d",
  "namespace": {
    "id": "root"
  },
  "path": "sys/mounts/transit",
  "data": {
    "config": {
      "default_lease_ttl": "hmac-sha256:8ed0513cb3a7070d1f2a5ed643c1b0f455e887b66b82204085ca86c3661b26b4",
      "force_no_cache": false,
      "max_lease_ttl": "hmac-sha256:8ed0513cb3a7070d1f2a5ed643c1b0f455e887b66b82204085ca86c3661b26b4",
      "options": null
    },
    "description": "hmac-sha256:5f9da19df55a1a3f82d640bbf552847a11da9bfd962ef63cbc4848e78bb36ad4",
    "external_entropy_access": false,
    "local": false,
    "options": null,
    "seal_wrap": false,
    "type": "hmac-sha256:c27c9dc8f2e0a07cad1e12e722503879100baba3f859ec123ce22755a049ebf1"
  },
  "remote_address": "127.0.0.1",
  "remote_port": 51188
}

The request describes the operation that was performed on the path.

Display the request's path and the request's operation.

$ cat vault-audit.log | jq -s ".[-1].request.path,.[-1].request.operation"
"sys/mounts/transit"
"update"

The output displays that this operation performed an update on the path sys/mounts/transit.

To learn more about querying the Audit Log, read the Querying Audit Device Logs tutorial.

ACL Policy workflow

You can use the information you gained from the audit device log to define a policy that grants the capability to update at the path sys/mounts/transit. That policy is expressed in the following example.

path "sys/mounts/transit" {
  capabilities = ["update"]
}

You can define your policy as files and place them under version control as part of a Vault configuration codification strategy.

Write the above policy example to the file transit-updates.hcl.

$ cat > transit-updates.hcl << EOF
path "sys/mounts/transit" {
  capabilities = ["update"]
}
EOF

Write the policy into your Vault server from the file, and name it transit-updates.

$ vault policy write transit-updates transit-updates.hcl

Note

You can also manage and create ACL policies from files in the Vault web UI. Refer to the Vault Policies tutorial for detailed examples,

As part of the policy workflow, you can update existing ACL policies by changing their file content, and simply writing the policy again with the same name. The existing policy will be overwritten, but existing tokens with the policy attached will use the previous policy until a new token is requested.

Additional ACL policy challenges

Use the techniques you've learned here to complete these optional challenges.

Create an encryption key

Create a encryption key named user-data.

$ vault write -f transit/keys/user-data
Success! Data written to: transit/keys/user-data

Display the request path and the request operation of the last logged object.

$ cat vault-audit.log | jq -s ".[-1].request.path,.[-1].request.operation"
"transit/keys/user-data"
"sys/mounts/transit"
"update"

Challenge

Define a policy that grants the capability to update at path transit/keys/user-data.

👁‍🗨 Show policy

path "transit/keys/user-data" {
  capabilities = ["update"]
}

Encrypt plaintext with the key

The user-data key is capable of encrypting base64 encoded plaintext and returns the ciphertext.

Encrypt user sensitive data with the user-data key.

$ vault write transit/encrypt/user-data plaintext=$(base64 <<< "sensitive user data")
Key            Value
---            -----
ciphertext     vault:v1:k1nJTMMmYM2nsYitgHRml6ExwWySGtkqgaOz3JI6kV+IT9995O26qOY5Uec/dn+l
key_version    1

Display the request path and the request operation of the last logged object.

$ cat vault-audit.log | jq -s ".[-1].request.path,.[-1].request.operation"
"transit/encrypt/user-data"
"transit/encrypt/user-data"
"create"

Challenge

Define a policy that grants the capability to update at path transit/encrypt/user-data.

👁‍🗨 Show policy

path "transit/encrypt/user-data" {
  capabilities = ["update"]
}

Decrypt ciphertext with the key

The same key is used to decrypt the ciphertext and return the base64 encoded plaintext.

Create a variable named USER_SECRET_DATA that stores the cipher text generated.

$ USER_SECRET_DATA=$(vault write -format=json \
   transit/encrypt/user-data plaintext=$(base64 <<< "sensitive user data") \
   | jq -r ".data.ciphertext")

Decrypt the user sensitive data with the user-data key.

$ vault write transit/decrypt/user-data ciphertext=$USER_SECRET_DATA
Key          Value
---          -----
plaintext    c2Vuc2l0aXZlIHVzZXIgZGF0YQo=

Display the request path and the request operation of the last logged object.

$ cat vault-audit.log | jq -s ".[-1].request.path,.[-1].request.operation"
"transit/decrypt/user-data"
"update"

Challenge

Define a policy that grants the capability to update at path transit/decrypt/user-data.

👁‍🗨 Show policy

path "transit/decrypt/user-data" {
  capabilities = ["update"]
}

Enable a key/value secrets engine

For the next challenges, enable an instance of the Key/Value Secrets Engine version 2 at the path kv-v2.

$ vault secrets enable -version=2 -path=kv-v2 kv

Display the request path and the request operation of the last logged object.

$ cat vault-audit.log | jq -s ".[-1].request.path,.[-1].request.operation"
"sys/mounts/kv-v2"
"update"

Write an example secret in the newly enabled secrets engine.

$ vault kv put kv-v2/my-secret vault=awesome
==== Secret Path ====
kv-v2/data/my-secret

======= Metadata =======
Key                Value
---                -----
created_time       2022-09-01T14:43:41.273398Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

Display the request path and the request operation of the last logged object.

$ cat vault-audit.log | jq -s ".[-1].request.path,.[-1].request.operation"
"kv-v2/data/my-secret"
"create"

Challenge

Write a policy to Vault that grants the capability to list and read at the paths kv-v2/data/my-secret and kv-v2/metadata/my-secret.

👁‍🗨 Show policy

vault policy write my-secret-read-list - <<EOF
path "kv-v2/+/my-secret" {
  capabilities = ["read", "list"]
}
EOF

Note

The policy uses the + character for path matching so that it can match the sub-paths kv-v2/data and kv-v2/metadata required by the Key/Value Secrets Engine version 2. You can learn more in the KV Secrets Engine - Version 2 ACL rules documentation.

Create a token with the policy attached and set its value to the environment variable MY_SECRET_TOKEN.

$ export MY_SECRET_TOKEN="$(vault token create -field=token -policy=my-secret-read-list)"

Read the secret using the new token with attached my-secret-read-list policy.

$ VAULT_TOKEN=$MY_SECRET_TOKEN vault kv get kv-v2/my-secret
==== Secret Path ====
kv-v2/data/my-secret

======= Metadata =======
Key                Value
---                -----
created_time       2022-09-01T14:43:41.273398Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

==== Data ====
Key      Value
---      -----
vault    awesome

Clean up

Here are the steps you need to completely clean up the scenario content from your local environment.

Stop the Vault server process.
```
$ pkill vault
```
Remove the example policy file.
```
$ rm -f transit-updates.hcl
```
Remove the audit device log file.
```
$ rm -f vault-audit.log
```

Unset environment variables.

$ unset VAULT_ADDR VAULT_TOKEN MY_SECRET_TOKEN

Next steps

In this tutorial, you performed operations and then queried the audit device log to discover the path and action involved in the operations. Learn more about querying the Audit Log, read the Querying Audit Device Logs tutorial.

To define policies requires understanding the paths and capabilities of each authentication method and secrets engine. Learn additional approaches in the Write a Policy from API documentation tutorial.

Help and reference

Write policies using API documentation

Simplify management of policies