Manage tenants with Vault namespaces

19min
|
HCP
Vault

HashiCorp Vault provides several options for providing applications, teams, or even separate lines of business access to dedicated resources in Vault. For example, you could enable multiple kv (key/value) secret engines using different paths, or use policies to restrict access to specific prefixes within a single secret engine.

An additional method for providing dedicated resources in a Vault deployment are namespaces. All instances of Vault start with the root namespace. With Vault Enterprise and HCP Vault Dedicated, customers can create additional namespaces to allow great isolation of resources. This allows Vault administrators to implement a Vault-as-a-Service model allowing each group using Vault to manage their own secret engines, auth methods, and policies. Most importantly, each group is restricted to work only with the namespace they have been granted access to.

Note

With Vault Community Edition, all plugins are enabled in the root namespace. Additional namespaces cannot be created.

In this tutorial, you will create multiple namespaces, enable plugins, and create policies to demonstrate the isolation capabilities of namespaces.

Prerequisites

An HCP account with a dev tier or higher cluster.
Vault installed on your local machine.
jq installed.

Overview

Namespaces are isolated environments that functionally create "Vaults within a Vault." They have separate login paths, and support creating and managing data isolated to their namespace. This functionality enables you to provide Vault as a service to tenants, allowing each tenant to manage resources within their own namespace.

A tenant, relative to namespaces, is any logical group within your organization that needs to utilize Vault Dedicated. Your Vault design will help dictate how you define a tenant. For example, a tenant may be a specific team such as your front end, or back end development teams. Additionally, you could design Vault around departments, separate lines-of-business, or specific applications where a tenant is your inventory application or marketing automation application.

HCP Vault Dedicated admin namespace

When working with Vault Dedicated, you will start your configuration in the admin namespace. This namespace is created during the deployment of your Vault Dedicated cluster and provides a similar experience to Vault Enterprise clusters that utilize nested namespaces.

Note

For Vault Dedicated clusters, the root namespace is reserved for platform operations and is not customer accessible. Visit HCP Vault Dedicated Constraints and Known Issues page for more information.

diagram-hcp-vault-namespace

Because you are operating from the admin namespace, you must pass the namespace parameter through one of the supported methods such as the -namespace CLI flag, X-Vault-Namespace API header, VAULT_NAMESPACE environment variable, or the vault_namespace resource in Terraform.

Your Vault Dedicated cluster will start with the token auth method enabled. This allows you to log in and perform the initial configuration of Vault Dedicated.

A good practice is to limit the use of the admin token and configure your preferred auth method(s) for regular use. The auth methods you choose to enable will be based on the who (person) or what (machine) will access your Vault Dedicated cluster. It is not uncommon to have multiple auth methods enabled.

For example, you will likely have a team responsible for administering your Vault Dedicated cluster, and applications retrieving secrets. To support human authentication you may choose to enable the Okta, GitHub, or user pass auth method. When an application needs to authenticate with Vault Dedicated, you may choose the AppRole auth method, or a platform specific auth method such as the AWS auth method.

When planning your Vault Dedicated deployment, it is important to consider who or what will need to access Vault, and select one or more auth methods that can support your organizations use case(s).

Reference

You can manage the Vault clients' identity by creating entities and groups. To learn more about Vault identity, refer to the Identity: Entities and Groups tutorial.

Vault mount points

Everything in Vault is path-based. Each path corresponds to an operation or secret in Vault, and the Vault API endpoints map to these paths; therefore, writing policies configures the permitted operations to specific secret paths is impacted by the namespace it is created in.

Auth methods and secrets engines can be categorized into two types:

Dedicated: Auth methods and secrets engines that can be managed and mapped directly to a specific organizational unit. For example, the team that manages app-1 can utilize their own AppRole and/or KV mounts without the ability to impact other teams' mounts.

Shared: An organization level resources such as Kubernetes auth method and the Active Directory (AD) secrets engine that are shared and managed at the company or admin level; therefore, mounted at the company or admin level namespace.

diagram-hcp-vault-auth-methods-compare

You want to find the right balance of granularity between the various mounts needed and the roles defined within the mounts.

Sharing mounts between teams has benefits and risk. It is up to you to find the right balance of granularity between the various mounts needed and the roles defined within the mounts. Below are a couple use cases with their benefits and risks.

Single shared namespace with sub-paths

Benefit: Reduces management overhead from managing multiple namespaces.

Risk: A mount is accidentally deleted causing all users of that secret engine to be impacted.

diagram-hcp-vault-shared-namespaces

Namespace per tenant

Benefit: Provide sub-paths for different teams or applications, and limit the blast-radius of an errant change to a single mount.

Risk: Unique mounts per namespace becomes inefficient from a management perspective at scale.

diagram-hcp-vault-nested-namespaces

Lab setup

Note

If you do not have access to an HCP Vault Dedicated cluster, visit the Create a Vault Cluster on HCP tutorial.

Launch the HCP Portal and login.
Click Vault in the left navigation pane.
In the Vault clusters pane, click vault-cluster.
Under Cluster URLs, click Public Cluster URL.
In a terminal, set the VAULT_ADDR environment variable to the copied address.
```
$ export VAULT_ADDR=<Public_Cluster_URL>
```
Return to the Overview page and click Generate token.
Within a few moments, a new token will be generated.
Copy the Admin Token.
Return to the terminal and set the VAULT_TOKEN environment variable.
```
$ export VAULT_TOKEN=<token>
```
Set the VAULT_NAMESPACE environment variable to admin.
```
$ export VAULT_NAMESPACE=admin
```
The admin namespace is the top-level namespace automatically created by HCP Vault. All CLI operations default to use the namespace defined in this environment variable.

The Vault Dedicated server is ready.

Personas

This tutorial involves two personas:

Vault admin (super-user) - responsible for managing the cluster & admin namespace. Uses the Vault Dedicated generated admin token to set up the namespaces, etc.
Tenant admin - responsible for managing the tenant namespace (e.g. /admin/namespace). Uses an auth method configured by the Vault admin to log in.

Create namespaces

(Persona: Vault admin)

Interactions with Vault Dedicated start in the admin namespace. You can, however, nest namespaces based on your Vault design and organizational needs. In this section you will create two namespaces - namespaceA and namespaceB.

Note

Remember that namespaceA and namespaceB could be any logical group of resources within your organization that requires access to Vault Dedicated.

Create namespaceA.
```
$ vault namespace create namespaceA

Key     Value
---     -----
id      rXVUH
path    admin/namespaceA/
```
Notice the path is admin/namespaceA. This is because Vault Dedicated starts with the admin namespace.

Create namespaceB.

$ vault namespace create namespaceB

Key     Value
---     -----
id      rLMNOP
path    admin/namespaceB/

List all the available namespaces.
```
$ vault namespace list

Keys
----
namespaceA/
namespaceB/
```
The results display the two namespaces. The partial path is displayed because the admin/ portion of the path was provided to the command by the environment variable VAULT_NAMESPACE.

Enable auth methods

(Persona: Vault admin)

Another design decision to consider when implementing Vault Dedicated is where to enable auth methods. You can enable auth methods in the admin namespace, or you can enable auth methods in a sub-namespace.

The best practice is to enable auth methods in each child namespace and let the namespace-level admin to configure and manage them. This reduces the involvement of the cluster-level admin, and make the child namespace independent from its parent namespace.

Note

Despite the best practice, it may not be desirable to enable an auth method such as Active Directory or LDAP (a company-level identity provider) on every child namespace. Refer to the Leveraging identity for auth methods with external groups discussion.

You will now enable the userpass auth method in the admin namespace, and namespaceA.

Enable the userpass auth method in the admin namespace.

$ vault auth enable userpass

Success! Enabled userpass auth method at: userpass/

Using the -namespace flag, enable the userpass auth method in namespaceA.
```
$ vault auth enable -namespace=admin/namespaceA userpass

Success! Enabled userpass auth method at: userpass/
```
By using the -namespace flag, you have enabled the auth method at admin/namespaceA/userpass.

Create ACL policies

Now that you have created namespaces, you need to create a policy to define what each user can access, and create the individual user accounts.

List and review the policies in the admin namespace.
```
$ vault policy list

default
hcp-root
```
The hcp-root policy is only available in the admin namespace. This policy provides full access to the admin namespace and any additional namespaces, secret engines, and auth methods you create in Vault Dedicated. You should limit the use of the hcp-root policy. Once you have performed some initial configuration, a good practice would be to create a more limited policy that grants explicit access to the resources needed by your Vault admin and/or security team.
The default policy is, by default, attached to all authenticated resources, in this case the users you create in Vault Dedicated. This policy provides some basic access to allow the resource to perform token lookups, or access the cubbyhole secrets engine for any individual resource.
List and review the policies in the adminA namespace.
```
$ vault policy list -namespace=admin/namespaceA

default
```
Each additional namespace you create will also have a default policy available.
Create a policy file that allows full access to all paths.
```
$ tee vault-admin-policy.hcl <<EOF
# Admin policy
path "*" {
        capabilities = ["sudo","read","create","update","delete","list","patch"]
}
EOF
```
Observe the path is set to "*" meaning any path can be read by the assigned resource.
What do you think would happen if you assign this policy to a user in the admin namespace?
What about to a user in namespaceA or namespaceB?
Would the users in each namespace be able to access all other namespaces?

Test the policy in the admin namespace

Create a policy named vault-admin-policy in the admin namespace.

$ vault policy write vault-admin-policy vault-admin-policy.hcl

Success! Uploaded policy: vault-admin-policy

Create a user named vault-admin in the admin namespace and assign the vault-admin-policy policy.

$ vault write auth/userpass/users/vault-admin \
   password=be3xcellent! \
   policies=vault-admin-policy \
   token_ttl=8h

Return to the HCP Portal and click the Public link under Cluster URLs.
Click the Method pulldown menu and select Username.
Enter vault-admin in the Username field, be3xcellent! in the Password field, and then click Sign In.
You are now signed into the Vault UI and placed in admin.
Based on the policy assigned, should you have full access to the namespace?
Click admin and select Manage namespaces.
You have access to both namespaceA and namespaceB because they are under the scope of the admin namespace, and the policy provides access to all all paths (*).
Click the user icon in the top right of the page and select Sign out.

Test the policy in the namespaceA namespace

Switch back to the terminal, create a user named vault-admina in the adminA namespace and assign the vault-admin-policy policy.

$ vault write -namespace=admin/namespaceA auth/userpass/users/vault-admina \
   password=be3xcellent! \
   policies=vault-admin-policy

Read the user details to confirm the user was created.

$ vault read -namespace=admin/namespaceA auth/userpass/users/vault-admina

Key                        Value
---                        -----
policies                   [vault-admin-policy]
token_bound_cidrs          []
token_explicit_max_ttl     0s
token_max_ttl              0s
token_no_default_policy    false
token_num_uses             0
token_period               0s
token_policies             [vault-admin-policy]
token_ttl                  0s
token_type                 default

Return to the HCP Portal and click the Public link in the Access web UI card.
Click the Method pulldown menu and select Username.
In the Namespace textbox enter /namespaceA.
Enter vault-admina in the Username field, be3xcellent! in the Password field, and then click Sign In.
You are now signed into the Vault UI and placed in namespaceA.
Based on the policy assigned, should you have full access to the namespace?
Should you have access to the namespaceB or admin namespace like the vault-admin user has?
Click namespaceA and select Manage namespaces.
You receive a Not authorized message because the policy you created exists in the admin namespace, but the user exists in the namespaceA namespace. The only effective policy is Default.
Policies must exist in the same namespace as the auth method for the authenticated resource. You cannot assign a policy from namespaceA to a resource in namespaceB.
Click the user icon in the top right of the page and select Sign out.

Return to the terminal and create a policy in the namespaceA namespace.

$ vault policy write -namespace=admin/namespaceA vault-admina-policy vault-admin-policy.hcl

Success! Uploaded policy: vault-admina-policy

Update the vault-admina user with the new policy.

$ vault write -namespace=admin/namespaceA auth/userpass/users/vault-admina \
   password=be3xcellent! \
   policies=vault-admina-policy

Return the Vault UI and enter vault-admina in the Username field, be3xcellent! in the Password field, and then click Sign In.
Click namespaceA and select Manage namespaces.
You now have the ability to create additional namespaces under /admin/namespaceA because the vault-admina-policy policy was created in the same namespace as the auth method (userpass) and authenticated resource (vault-admina).
The policy is relative to the namespace it is created in. The path * allows access to any path within the namespaceA namespace but not the admin or namespaceB namespace.
Sign out of the Vault UI.

Test the policy in the namespaceB namespace

Thus far you have created and configured the admin and namespaceA namespace with an auth method, user, and policy. You have logged into each namespace to demonstrate how a policy works relative to the namespace it is created in.

In the admin namespace, using a path of "*" in the policy you were able to access all other namespaces. When you created the a policy in the namespaceA namespace using the same policy file, you were able to only access that namespace and not the admin or namespaceB namespace.

Now, you will simulate creating an admin for namespaceB, but you will create the user in the admin namespace and see how this changes how you may have to write policies.

Create a policy file that allows full access to all paths in namespaceB.
```
$ tee vault-adminb-policy.hcl <<EOF
# Admin policy
path "namespaceB/*" {
        capabilities = ["sudo","read","create","update","delete","list","patch"]
}
EOF
```
Because the vault-adminb user is being created in the admin namespace, you will need to grant access to the specific mount point for the namespaceB namespace. If you were to reuse the existing vault-admin-policy.hcl file, the user would have access to all namespaces instead of just namespaceB.
Centralizing all users in the admin namespace is one model that an organization may choose to adopt. It is used to consolidate auth methods to a single namespace and reduce the number of auth methods enabled across all namespaces. A trade off of this mode is that will likely increases the number of policies that are needed because they need to grant or deny permission to specific path.

Create a policy named vault-adminb-policy in the admin namespace.

$ vault policy write vault-adminb-policy vault-adminb-policy.hcl

Success! Uploaded policy: vault-adminb-policy

Create user named vault-adminb in the admin namespace and assign the vault-adminb-policy

$ vault write auth/userpass/users/vault-adminb \
   password=be3xcellent! \
   policies=vault-adminb-policy

Return to the Vault UI.
If it is not already selected, click the Method pulldown menu and select Username.
Delete any previous namespace text from Namespace text box.
Enter vault-adminb in the Username field, be3xcellent! in the Password field, and then click Sign In.
You are now signed into the Vault UI and placed in the admin namespace.
Click admin and select Manage namespaces.
You receive a Not authorized message because the policy you created only permits access to namespaceB/*.
Click admin and select namespaceB, you are now in the namespaceB namespace.
Navigate back to Manage namespaces. Instead of the Not authorized message, you are presented with the Namespaces page where you can create nested namespaces under namespaceB.
Sign out of the Vault UI and close the page.

You have successfully created multiple namespaces in Vault Dedicated, enabled the userpass auth method in the admin and namespaceA namespace, and created policies to show how namespaces impact access to resources authenticated to Vault Dedicated.

Using Namespace parameters

Now that you have a working Vault Dedicated environment with multiple namespaces, you can explore the supported options for including the namespace using the Vault CLI and API.

To target a specific namespace, do one of the following:

Use the VAULT_NAMESPACE environment variable
Use the -namespace (or -ns for short-hand) flag

In the Lab setup section, you created several environment variables to enable CLI access to your Vault Dedicated environment. One of these environment variables is VAULT_NAMESPACE. This environment variable is one of the supported methods for declaring the namespace.

List all namespaces created using the Vault CLI.

$ vault namespace list

Keys
----
namespaceA/
namespaceB/

Because you configured the environment variables in the lab setup section, you can run Vault commands without any additional flags.

You can overwrite the current VAULT_NAMESPACE value without resetting it.

$ VAULT_NAMESPACE=admin/namespaceA vault auth list

Path         Type        Accessor                  Description                Version
----         ----        --------                  -----------                -------
token/       ns_token    auth_ns_token_82b2e82c    token based credentials    n/a
userpass/    userpass    auth_userpass_7fcc15dd    n/a                        n/a

Another option is to pass the -namespace (or -ns) command line flag. The command line flag takes precedence over the environment variable, if set. As a Vault admin, this precedence order is useful because you can configure your environment for the admin or primary namespace you are responsible for but still can access sub-namespaces without changing your local environment.
List the auth methods in the namespaceA namespace using the -namespace flag.
```
$ vault auth list -namespace=admin/namespaceA

Path         Type        Accessor                  Description                Version
----         ----        --------                  -----------                -------
token/       ns_token    auth_ns_token_82b2e82c    token based credentials    n/a
userpass/    userpass    auth_userpass_7fcc15dd    n/a                        n/a
```
Without having to unset the VAULT_NAMESPACE environment variable, you are able to interact with other namespaces using the -namespace command line flag.

To target a specific namespace, do one of the following:

Set the X-Vault-Namespace HTTP header
Use the fully qualified path ($VAULT_ADDR/v1/<namespace>/<endpoint_path>)

List all namespaces created using the Vault API by using cURL with HTTP headers.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
    --request LIST \
    $VAULT_ADDR/v1/sys/namespaces | jq -r ".data"

Example output:

{
  "key_info": {
    "namespaceA/": {
      "custom_metadata": {},
      "id": "3hfi7",
      "path": "admin/namespaceA/"
    },
    "namespaceB/": {
      "custom_metadata": {},
      "id": "UUsuB",
      "path": "admin/namespaceB/"
    }
  },
  "keys": [
    "namespaceA/",
    "namespaceB/"
  ]
}

The X-Vault-Namespace header is provided by the value from the VAULT_NAMESPACE environment variable you created in the Lab setup section.

Another option is to directly reference the namespace mount point in the API call without providing the HTTP header.

List all namespaces created using the Vault API by using cURL with the full mount point path.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --request LIST \
    $VAULT_ADDR/v1/admin/sys/namespaces | jq -r ".data"

Example output:

{
  "key_info": {
    "namespaceA/": {
      "custom_metadata": {},
      "id": "3hfi7",
      "path": "admin/namespaceA/"
    },
    "namespaceB/": {
      "custom_metadata": {},
      "id": "UUsuB",
      "path": "admin/namespaceB/"
    }
  },
  "keys": [
    "namespaceA/",
    "namespaceB/"
  ]
}

Troubleshooting

Omitting the namespace from any Vault command or API call results in common error messages.

Unset the VAULT_NAMESPACE environment variable.
```
$ unset VAULT_NAMESPACE
```

Verify there is no VAULT_NAMESPACE environment variable.

$ printenv | grep VAULT_

VAULT_ADDR=https://vault-cluster-public-vault-b0iz2.m3n4l1fe.n1.hashicorp.cloud:8200
VAULT_TOKEN=hvs.CAESIBxule_a2c4e6-qhfMZB5TDg8WI8pd8gN0ol0Bhb_5iGicKImh2cy5FcFpCZFJiMUpyeHRkR3hHZTlFenRMR2YudDFsVGQQuAE

When the VAULT_NAMESPACE environment variable is unset, click the common errors below to produce the example error messages.

There are two common causes of the 403 error code - the namespace value was not provided, or the policy attached to the token does not permit access to the specified resource.

In this example, the namespace value is omitted.

$ vault namespace list
Error listing namespaces: Error making API request.

URL: GET https://vault-cluster-public-vault-b0iz2.m3n4l1fe.n1.hashicorp.cloud:8200/v1/sys/namespaces?list=true
Code: 403. Errors:

* 1 error occurred:
        * permission denied

Because the environment variable was unset, the effective path the command is trying list resources from is the root namespace. Your VAULT_TOKEN environment variable was created from the token auth method in the admin namespace, with the hcp-root policy attached. The hcp-root policy provides access to the admin namespace.

A null response will be observed if the namespace is not provided by an HTTP header or included in the path.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --request LIST \
    $VAULT_ADDR/v1/sys/namespaces | jq -r ".data"

Example output:

null

Cleanup

To avoid unnecessary charges, you should delete any Vault Dedicated clusters you created for this tutorial.

Help and reference

Manage HCP Vault with Terraform

OIDC authentication with Okta