Vault
Sentinel HTTP import
Enterprise Only
Sentinel requires Vault Enterprise.
Sentinel is a language framework for policy built to embed in Vault Enterprise, and enable fine-grained, logic-based policy decisions which cannot be fully handled by the ACL policies.
If you are not yet familiar with using Sentinel policies in Vault, review Sentinel Policies.
Challenge
Sentinel Endpoint Governing Policies (EGP) are flexible and offer rich capabilities and rules to enable uses like Multi Factor Authentication requirements or per path policy delegation. What if your use case requires policy rules which get information from the output of an external HTTP API?
Solution
The Sentinel http import enables the use of HTTP-accessible data from outside the runtime in Sentinel policy rules. At a basic level, the import uses an HTTP GET request and by default, any request response that is not a successful "200 OK" HTTP response causes a policy error. This behavior can be further customized to your particular use case.
In this tutorial you will write a Sentinel EGP that uses the HTTP import to query the Vault server API for information about enabled secrets engines. The Sentinel EGP will either allow or deny certain operations based on the specific criteria defined in the policy.
Personas
Sentinel policies provide a declarative way to grant or forbid access to certain paths and operations in Vault. Therefore, the steps described in this tutorial are performed by Vault admins or security operations.
Prerequisites
To perform the tasks described in this tutorial, you need to have:
- Vault Enterprise binary and license key.
- jq binary installed in your system PATH.
- Git binary installed in your system PATH.
You should have familiarity with Vault and Sentinel, and be comfortable authoring and deploying a Sentinel EGP for Vault to follow the example in this tutorial.
Policy requirements
This tutorial requires two Vault tokens - one for the Vault admin who will be creating and testing the Sentinel EGP, and one to use to for the EGP.
You need to authenticate to Vault with a token that has the required capabilities to follow the example workflow.
Warning
Do not use the initial root token for any purpose other than creating the required tokens. A root token is not subject to Sentinel policy enforcement, and will cause issues with policy development and testing.
Vault admin token policy
Review the Vault ACL policy which the admin user's token needs to create the EGP.
# To list policies
path "sys/policies/*"
{
capabilities = ["list"]
}
# To manage ACLs
path "sys/policies/acl/*"
{
capabilities = ["create", "read", "update", "delete", "list"]
}
# To manage EGPs
path "sys/policies/egp/*"
{
capabilities = ["create", "read", "update", "delete", "list"]
}
# To manage secrets engines
path "sys/mounts/*"
{
capabilities = ["create", "read", "update", "delete", "list"]
}
Going forward, the tutorial refers to this token as the admin token.
Sentinel EGP token policy
The Sentinel EGP example in this tutorial makes a call to the /sys/mounts API to learn about the enabled secrets engines.
This is an authenticated API endpoint, so the EGP requires a Vault token with the minimum required ACL policy capability
to read from /sys/mounts
.
That policy is as follows.
path "/sys/mounts" {
capabilities = ["read"]
}
Going forward, the tutorial refers to this token as the EGP token.
Lab setup
Before you can use the Sentinel HTTP import module, you need to configure Vault and define additional enabled modules. You can do this with the additional_enabled_modules parameter.
For this tutorial, you will use a Vault server in dev mode and pass just the Sentinel specific configuration to it at runtime from the configuration file,
vault-sentinel-configuration.hcl
.Write the configuration file to enable the HTTP module.
$ cat > vault-sentinel-configuration.hcl << EOF sentinel { additional_enabled_modules = ["http"] } EOF
Export an environment variable for the Vault Enterprise license .
$ export VAULT_LICENSE=actual-license-key
In the same terminal session where you created the configuration file and
VAULT_LICENSE
environment variable, start the Vault dev mode server and pass the configuration file to the-config
flag.$ vault server -dev \ -dev-root-token-id root \ -config vault-sentinel-configuration.hcl
The Vault dev server defaults to running at
127.0.0.1:8200
. The server is automatically initialized, and unsealed.Insecure operation
Do not run a Vault dev server in production. This tutorial uses a dev mode server to simplify the unsealing process for the hands on lab.
Open a new terminal and export an environment variable for the
vault
CLI to address the Vault server.$ export VAULT_ADDR=http://127.0.0.1:8200
Export an environment variable for the
vault
CLI to authenticate with the Vault server.$ export VAULT_TOKEN=root
Note
For these tasks, you can use Vault's root token. Keep in mind though, that the best practice is to use root tokens just for initial setup or in emergencies.
The Vault server is ready.
Write ACL policies
Create two ACL policies and generate the admin token and EGP token.
Create the admin ACL policy.
$ vault policy write admin - << EOF # To list policies path "sys/policies/*" { capabilities = ["list"] } # To manage ACLs path "sys/policies/acl/*" { capabilities = ["create", "read", "update", "delete", "list"] } # To manage EGPs path "sys/policies/egp/*" { capabilities = ["create", "read", "update", "delete", "list"] } # To manage secrets engines path "sys/mounts/*" { capabilities = ["create", "read", "update", "delete", "list"] } EOF
Example output:
Success! Uploaded policy: admin
Create the EGP ACL policy.
$ vault policy write sentinel-egp-token - << EOF path "/sys/mounts" { capabilities = ["read"] } EOF
Example output:
Success! Uploaded policy: sentinel-egp-token
Capture the admin user token value to the
VAULT_ADMIN_TOKEN
environment variable.$ export VAULT_ADMIN_TOKEN="$(vault token create -policy=admin -field=token)"
Review the
VAULT_ADMIN_TOKEN
token.$ echo $VAULT_ADMIN_TOKEN
Capture the EGP token value to the
VAULT_EGP_TOKEN
environment variable.$ export VAULT_EGP_TOKEN="$(vault token create -policy=sentinel-egp-token -field=token)"
Review the
VAULT_EGP_TOKEN
token.$ echo $VAULT_EGP_TOKEN
Unset the
VAULT_TOKEN
environment variable.$ unset VAULT_TOKEN
Login with the admin token.
$ vault login -no-print $VAULT_ADMIN_TOKEN
Review Sentinel EGP
With the initial Vault configuration in place, you will now create the Sentinel EGP.
In this example, you will limit the number of enabled Transit secrets engines instances to just a single one.
You do so by leveraging Sentinel with the HTTP import to query the Vault API, and check the number of enabled Transit secrets engines.
The Sentinel policy can then further choose to allow or deny enabling of new secrets engines based on the current count of enabled transit secrets engines.
Example policy
Review the details in the Sentinel policy.
# Validate that a transit secrets engine is not already enabled before
# allowing the enabling of a transit secrets engine.
import "http"
import "json"
# Set parameters for Vault address and EGP token values
param vault_addr default "http://127.0.0.1:8200"
param vault_token default "$VAULT_EGP_TOKEN"
# Print some information about the request.
# Note that these messages are printed only when the policy is violated.
print("Request path:", request.path)
print("Request data:", request.data)
print("Request operation:", request.operation)
validate_transit_not_present = func() {
# Start with validated set to false
validated = false
# Make request to the Vault API
req = http.request(vault_addr + "/v1/sys/mounts").
with_header("X-Vault-Token", vault_token)
resp = http.accept_status_codes([200, 404]).get(req)
body = json.unmarshal(resp.body)
print("Body Keys:", keys(body))
# if transit type secrets engine is found, return validated with false value
if "data" in keys(body) {
for values(body.data) as secrets_engine {
if secrets_engine.type is "transit" {
validated = false
break
} else {
validated = true
}
}
}
return validated
}
# Main rule
main = rule when request.path matches "sys/mounts/*" and
request.data.type in ["transit"] and
request.operation in ["update"] {
validate_transit_not_present()
}
- Lines 3-4 declare the http and json imports needed for making HTTP requests and processing JSON.
- Lines 7-8 define parameter values for the Vault address and Vault token for the EGP itself. You should replace http://127.0.0.1:8200 with the address of your Vault server in the same URL format with port (ex. http://localhost:8200) and replace \$VAULT_EGP_TOKEN with the actual token value for your EGP token that you created in step 3.
- Lines 12-14 print statements which are useful for debugging. Vault prints this information whenever a request violates the policy and fails.
- Lines 16-37 includes the EGP logic in a single function,
validate_transit_not_present()
. It makes the HTTP request to the Vault /sys/mounts API endpoint to query the state of enabled secrets engines by examining each enabled engine. If one is the type "transit", then the validated variable value returns as false, causing the main rule evaluation to fail; permission to enable a transit secrets engine for a non-root token users gets denied. - Lines 40-44 includes the main rule and sets conditions for its evaluation based on request parameters; the request must be for the /sys/mounts path, it must be an update operation, and the data type for the operation needs to be transit. If the request matches these conditions, then
validate_transit_not_present
checks for an enabled transit secrets engine.
Create and test Sentinel EGP
Write the
transit-check.sentinel
Sentinel EGP.$ cat > transit-check.sentinel << EOF # Validate that a transit secrets engine is not already enabled before # allowing the enabling of a transit secrets engine. import "http" import "json" # Set parameters for Vault address and EGP token values param vault_addr default "$VAULT_ADDR" param vault_token default "$VAULT_EGP_TOKEN" # Print some information about the request. # Note that these messages are printed only when the policy is violated. print("Request path:", request.path) print("Request data:", request.data) print("Request operation:", request.operation) validate_transit_not_present = func() { # Start with validated set to false validated = false # Make request to the Vault API req = http.request(vault_addr + "/v1/sys/mounts"). with_header("X-Vault-Token", vault_token) resp = http.accept_status_codes([200, 404]).get(req) body = json.unmarshal(resp.body) print("Body Keys:", keys(body)) # if transit type secrets engine is found, return validated with false value if "data" in keys(body) { for values(body.data) as secrets_engine { if secrets_engine.type is "transit" { validated = false break } else { validated = true } } } return validated } # Main rule main = rule when request.path matches "sys/mounts/*" and request.data.type in ["transit"] and request.operation in ["update"] { validate_transit_not_present() } EOF
Warning
If you deploy the Sentinel EGP with the HTTP API or Web UI, you need to replace the
$VAULT_EGP_TOKEN
value with the actual EGP token value before deploying the policy.Test the Sentinel policy before deployment to check syntax and to document expected behavior by downloading the Sentinel simulator.
$ curl https://releases.hashicorp.com/sentinel/0.19.5/sentinel_0.19.5_darwin_amd64.zip --output sentinel_0.19.5_darwin_amd64.zip
Unzip the downloaded file.
$ unzip sentinel_0.19.5_darwin_amd64.zip -d /usr/local/bin
Create the
test/transit-check
directory structure.$ mkdir -p test/transit-check
Create the tests under the
test/<policy_name>
folder.Write a passing test case for the
transit-check
policy,test/transit-check/success.json
.$ cat > test/transit-check/success.json << EOF { "global": { "request": { "operation": "update", "path": "sys/mounts/transit", "data": { "type": "transit" } }, "body": { "sys/": { "accessor": "system_ab0f1127", "config": { "default_lease_ttl": 0, "force_no_cache": false, "max_lease_ttl": 0, "passthrough_request_headers": [ "Accept" ] }, "description": "system endpoints used for control, policy and debugging", "external_entropy_access": false, "local": false, "options": null, "seal_wrap": false, "type": "system", "uuid": "5e4101dd-ea91-5d26-9f09-85b675a72a95" }, "identity/": { "accessor": "identity_9228cd88", "config": { "default_lease_ttl": 0, "force_no_cache": false, "max_lease_ttl": 0 }, "description": "identity store", "external_entropy_access": false, "local": false, "options": null, "seal_wrap": false, "type": "identity", "uuid": "c584e265-8417-4671-7fec-225393fcf307" }, "cubbyhole/": { "accessor": "cubbyhole_da4d2e11", "config": { "default_lease_ttl": 0, "force_no_cache": false, "max_lease_ttl": 0 }, "description": "per-token private secret storage", "external_entropy_access": false, "local": true, "options": null, "seal_wrap": false, "type": "cubbyhole", "uuid": "59b388df-c992-e9b9-43ec-7830d9eed35d" }, "secret/": { "accessor": "kv_086ea056", "config": { "default_lease_ttl": 0, "force_no_cache": false, "max_lease_ttl": 0 }, "description": "key/value secret storage", "external_entropy_access": false, "local": false, "options": { "version": "2" }, "seal_wrap": false, "type": "kv", "uuid": "25bbe652-5f01-664d-a4ac-64a1bb45c478" }, "request_id": "aac465cc-3101-cfa8-67d0-cc2bce8c38b1", "lease_id": "", "renewable": false, "lease_duration": 0, "data": { "cubbyhole/": { "accessor": "cubbyhole_da4d2e11", "config": { "default_lease_ttl": 0, "force_no_cache": false, "max_lease_ttl": 0 }, "description": "per-token private secret storage", "external_entropy_access": false, "local": true, "options": null, "seal_wrap": false, "type": "cubbyhole", "uuid": "59b388df-c992-e9b9-43ec-7830d9eed35d" }, "identity/": { "accessor": "identity_9228cd88", "config": { "default_lease_ttl": 0, "force_no_cache": false, "max_lease_ttl": 0 }, "description": "identity store", "external_entropy_access": false, "local": false, "options": null, "seal_wrap": false, "type": "identity", "uuid": "c584e265-8417-4671-7fec-225393fcf307" }, "secret/": { "accessor": "kv_086ea056", "config": { "default_lease_ttl": 0, "force_no_cache": false, "max_lease_ttl": 0 }, "description": "key/value secret storage", "external_entropy_access": false, "local": false, "options": { "version": "2" }, "seal_wrap": false, "type": "kv", "uuid": "25bbe652-5f01-664d-a4ac-64a1bb45c478" }, "sys/": { "accessor": "system_ab0f1127", "config": { "default_lease_ttl": 0, "force_no_cache": false, "max_lease_ttl": 0, "passthrough_request_headers": [ "Accept" ] }, "description": "system endpoints used for control, policy and debugging", "external_entropy_access": false, "local": false, "options": null, "seal_wrap": false, "type": "system", "uuid": "5e4101dd-ea91-5d26-9f09-85b675a72a95" } }, "wrap_info": null, "warnings": null, "auth": null }, "test": { "main": true, "validate_transit_not_present": true } } } EOF
This test will pass because it does not include an enabled transit secrets engine.
The optional
test
definition adds more context to why the test should pass. The expected behavior is that the test passes becausevalidate_transit_not_present
returnstrue
andmain
returnstrue
.Write a failing test for the
transit-check
policy,test/transit-check/fail.json
.$ cat > test/transit-check/fail.json << EOF { "global": { "request": { "operation": "update", "path": "sys/mounts/transit", "data": { "type": "transit" } }, "body": { "sys/": { "accessor": "system_ab0f1127", "config": { "default_lease_ttl": 0, "force_no_cache": false, "max_lease_ttl": 0, "passthrough_request_headers": [ "Accept" ] }, "description": "system endpoints used for control, policy and debugging", "external_entropy_access": false, "local": false, "options": null, "seal_wrap": false, "type": "system", "uuid": "5e4101dd-ea91-5d26-9f09-85b675a72a95" }, "identity/": { "accessor": "identity_9228cd88", "config": { "default_lease_ttl": 0, "force_no_cache": false, "max_lease_ttl": 0 }, "description": "identity store", "external_entropy_access": false, "local": false, "options": null, "seal_wrap": false, "type": "identity", "uuid": "c584e265-8417-4671-7fec-225393fcf307" }, "cubbyhole/": { "accessor": "cubbyhole_da4d2e11", "config": { "default_lease_ttl": 0, "force_no_cache": false, "max_lease_ttl": 0 }, "description": "per-token private secret storage", "external_entropy_access": false, "local": true, "options": null, "seal_wrap": false, "type": "cubbyhole", "uuid": "59b388df-c992-e9b9-43ec-7830d9eed35d" }, "secret/": { "accessor": "kv_086ea056", "config": { "default_lease_ttl": 0, "force_no_cache": false, "max_lease_ttl": 0 }, "description": "key/value secret storage", "external_entropy_access": false, "local": false, "options": { "version": "2" }, "seal_wrap": false, "type": "kv", "uuid": "25bbe652-5f01-664d-a4ac-64a1bb45c478" }, "transit/": { "accessor": "transit_1f715ad9", "config": { "default_lease_ttl": 0, "force_no_cache": false, "max_lease_ttl": 0 }, "description": "", "external_entropy_access": false, "local": false, "options": null, "seal_wrap": false, "type": "transit", "uuid": "dcb11046-caec-5b9b-e902-9f1281c520de" }, "request_id": "d73a6a7a-a537-dd97-79a9-9e1b3654cc73", "lease_id": "", "renewable": false, "lease_duration": 0, "data": { "cubbyhole/": { "accessor": "cubbyhole_da4d2e11", "config": { "default_lease_ttl": 0, "force_no_cache": false, "max_lease_ttl": 0 }, "description": "per-token private secret storage", "external_entropy_access": false, "local": true, "options": null, "seal_wrap": false, "type": "cubbyhole", "uuid": "59b388df-c992-e9b9-43ec-7830d9eed35d" }, "identity/": { "accessor": "identity_9228cd88", "config": { "default_lease_ttl": 0, "force_no_cache": false, "max_lease_ttl": 0 }, "description": "identity store", "external_entropy_access": false, "local": false, "options": null, "seal_wrap": false, "type": "identity", "uuid": "c584e265-8417-4671-7fec-225393fcf307" }, "secret/": { "accessor": "kv_086ea056", "config": { "default_lease_ttl": 0, "force_no_cache": false, "max_lease_ttl": 0 }, "description": "key/value secret storage", "external_entropy_access": false, "local": false, "options": { "version": "2" }, "seal_wrap": false, "type": "kv", "uuid": "25bbe652-5f01-664d-a4ac-64a1bb45c478" }, "sys/": { "accessor": "system_ab0f1127", "config": { "default_lease_ttl": 0, "force_no_cache": false, "max_lease_ttl": 0, "passthrough_request_headers": [ "Accept" ] }, "description": "system endpoints used for control, policy and debugging", "external_entropy_access": false, "local": false, "options": null, "seal_wrap": false, "type": "system", "uuid": "5e4101dd-ea91-5d26-9f09-85b675a72a95" }, "transit/": { "accessor": "transit_1f715ad9", "config": { "default_lease_ttl": 0, "force_no_cache": false, "max_lease_ttl": 0 }, "description": "", "external_entropy_access": false, "local": false, "options": null, "seal_wrap": false, "type": "transit", "uuid": "dcb11046-caec-5b9b-e902-9f1281c520de" } }, "wrap_info": null, "warnings": null, "auth": null }, "test": { "main": false, "validate_transit_not_present": false } } } EOF
This test will fail because it includes an enabled transit secrets engine.
The optional
test
definition adds more context to why the test should fail. The expected behavior is that the test fails becausevalidate_transit_not_present
returnsfalse
andmain
returnsfalse
.Verify success and failure tests creation.
├── transit-check.sentinel └── test │ └── transit-check ├── fail.json └── success.json
Execute the Sentinel test.
$ sentinel test
Example output:
PASS - transit-check.sentinel PASS - test/transit-check/fail.json PASS - test/transit-check/success.json
Note
If you want to see the tracing and log output for tests, run the command with a
-verbose
flag.
Deploy Sentinel EGP
Sentinel policies have three enforcement levels:
Level | Description |
---|---|
advisory | The policy permitted to fail. Useful as a tool to educate new users. |
soft-mandatory | The policy must pass unless user specifies override. |
hard-mandatory | The policy must pass no matter what! |
Deploy the Sentinel policy with an enforcement level of hard-mandatory.
Login with the admin token.
$ vault login -no-print $VAULT_ADMIN_TOKEN
Store the Base64 encoded
transit-check.sentinel
policy in an environment variable namedPOLICY
.$ POLICY=$(base64 -i transit-check.sentinel)
Create a policy
transit-check
with enforcement level of hard-mandatory if there is more than one transit secrets engine enabled.$ vault write sys/policies/egp/transit-check \ policy="${POLICY}" \ paths="sys/mounts/*" \ enforcement_level="hard-mandatory"
Example output:
Success! Data written to: sys/policies/egp/transit-check
Read the
transit-check
policy.$ vault read sys/policies/egp/transit-check
Example output:
Key Value --- ----- enforcement_level hard-mandatory name transit-check paths [sys/mounts/*] policy # Validate that a transit secrets engine is not already enabled before # allowing the enabling of a transit secrets engine. import "http" import "json" # Set parameters for Vault address and EGP token values param vault_addr default "http://127.0.0.1:8200" param vault_token default "s.7j0ybDH9pklYXctdPbhAYoNI" # Print some information about the request. # Note that these messages are printed only when the policy is violated. print("Request path:", request.path) print("Request data:", request.data) print("Request operation:", request.operation) validate_transit_not_present = func() { # Start with validated set to false validated = false # Make request to the Vault API req = http.request(vault_addr + "/v1/sys/mounts"). with_header("X-Vault-Token", vault_token) resp = http.accept_status_codes([200, 404]).get(req) body = json.unmarshal(resp.body) print("Body Keys:", keys(body)) # if transit type secrets engine is found, return validated with false value if "data" in keys(body) { for values(body.data) as secrets_engine { if secrets_engine.type is "transit" { validated = false break } else { validated = true } } } return validated } # Main rule main = rule when request.path matches "sys/mounts/*" and request.data.type in ["transit"] and request.operation in ["update"] { validate_transit_not_present() }
Verification
Once you've deployed the policy, Vault will deny permission for all operations which fail the policy rule assertions when the server learns there is already a transit secrets engine enabled.
Login with the admin token.
$ vault login -no-print $VAULT_ADMIN_TOKEN
Enable one instance of the transit secrets engine.
$ vault secrets enable transit Success! Enabled the transit secrets engine at: transit/
Try to enable another instance of the transit secrets engine at the path another-transit-secrets-engine.
$ vault secrets enable -path=another-transit-secrets-engine transit
Example output:
Error enabling: Error making API request. URL: POST http://127.0.0.1:8200/v1/sys/mounts/another-transit-secrets-engine Code: 403. Errors: * 2 errors occurred: * egp standard policy "root/transit-check" evaluation resulted in denial. The specific error was: <nil> A trace of the execution for policy "root/transit-check" is available: Result: false Description: Main rule print() output: Request path: sys/mounts/another-transit-secrets-engine Request data: {"config": {"default_lease_ttl": "0s", "force_no_cache": false, "max_lease_ttl": "0s", "options": null}, "description": "", "external_entropy_access": false, "local": false, "options": null, "seal_wrap": false, "type": "transit"} Request operation: update Body Keys: ["sys/" "request_id" "renewable" "lease_duration" "secret/" "warnings" "identity/" "transit/" "auth" "lease_id" "cubbyhole/" "wrap_info" "data"] Rule "main" (byte offset 1256) = false * permission denied
A policy violation results in the inclusion of debug
print()
statement output.Warning
As with ACL policies,
root
tokens are NOT subject to Sentinel policy checks. Be sure to use a non-root token for the verification test.
Going even further with Vault Enterprise Namespaces
If you want to try a more advanced example involving the Vault Enterprise Namespaces feature, you can follow the guidance in Restrict Members of a Group with Sentinel. You will learn how to restrict the subgroups and entities of a group by requiring all subgroups and entities of a group to belong to the same namespace.
Cleanup
To clean up the example files and stop your Vault dev server follow these steps.
Remove the files.
$ rm -f transit-check-payload.json \ transit-check.sentinel\ vault-sentinel-configuration.hcl
Remove the Sentinel tests.
$ rm -rf test
In the terminal session where you started the Vault dev server, press CTRL+C to stop Vault.
Troubleshooting
If you have an error such as this example while testing the EGP.
Error enabling: Error making API request.
URL: POST http://127.0.0.1:8200/v1/sys/mounts/example
Code: 403. Errors:
* 2 errors occurred:
* egp standard policy "root/transit-example" evaluation resulted in denial.
The specific error was:
root/transit-example:4:1: Import "http" is not available
A trace of the execution for policy "root/transit-example" is available:
Result: false
Error: false
Description: Validate that a Transit secrets engine is not already enabled
when an operation to enable a Transit secrets engine is attempted
* permission denied
Note the Import "http" is not available part of the error. This indicates that Vault is not configured to allow the HTTP import. Check the instructions in the Lab setup section and make sure you have configured and restarted your Vault.