Google Cloud Platform (GCP) auth method

15min
|
Vault

Before a client can interact with Vault, it must authenticate with an auth method to acquire a token. This token has policies attached so that the behavior of the client can be governed.

NOTE: To learn the basics of Vault tokens, go through the Tokens tutorial.

Auth methods perform authentication to verify the user or machine-supplied information. Some of the supported auth methods are targeted towards users while others are targeted toward machines or apps.

Challenge

Vault supports a number of auth methods for users or system to prove their identity so that a token with appropriate policies can be obtained.

Google Cloud Platform customers require a solution that supports multiple services for both users and compute services.

Solution

The gcp auth method allows Google Cloud Platform entities to authenticate to Vault. Vault treats Google Cloud as a trusted third party and verifies authenticating entities against the Google Cloud APIs.

Prerequisites

This tutorial assumes the following:

You have a Google Cloud Platform account with permission to:
- Create IAM service accounts, policies
- Enable GCP APIs
- Access the GCP Cloud Shell
- Create VMs
Vault installed on your local machine
ngrok installed and configured with an auth token (required for Vault)

Lab setup

Workstation setup

Create a temporary directory to store the files used for this tutorial.
```
$ mkdir ~/TUTORIAL_TEMP
```
Retrieve the directory path and store it in the TUTORIAL_TEMP environment variable.
```
$ TUTORIAL_TEMP=$(echo ~/TUTORIAL_TEMP)
```
Go to the TUTORIAL_TEMP directory.
```
$ cd $TUTORIAL_TEMP
```

Vault setup

Note

If you do not have access to an HCP Vault Dedicated cluster, visit the Create a Vault Cluster on HCP tutorial.

Launch the HCP Portal and login.
Click Vault in the left navigation pane.
In the Vault clusters pane, click vault-cluster.
Under Cluster URLs, click Public Cluster URL.
In the same terminal you set the TUTORIAL_TEMP environment variable, set the VAULT_ADDR environment variable to the copied address.
```
$ export VAULT_ADDR=<Public_Cluster_URL>
```
Return to the Overview page and click Generate token.
Within a few moments, a new token will be generated.
Copy the Admin Token.
Return to the terminal and set the VAULT_TOKEN environment variable.
```
$ export VAULT_TOKEN=<token>
```
Set the VAULT_NAMESPACE environment variable to admin.
```
$ export VAULT_NAMESPACE=admin
```
The admin namespace is the top-level namespace automatically created by HCP Vault. All CLI operations default to use the namespace defined in this environment variable.
Note
For these tasks, you can use HCP Vault Dedicated's admin token. However, it is recommended that admin tokens are only used for enough initial setup or in emergencies. As a best practice, use an authentication method or token that meets the policy requirements.

Enable the KV secrets engine.

$ vault secrets enable -version=2 -path=secret kv

Success! Enabled the kv secrets engine at: secret/

The Vault Dedicated server is ready.

Open a new terminal window and start a Vault dev server with root as the root token.
```
$ vault server -dev -dev-root-token-id root
```
The Vault dev server defaults to running at 127.0.0.1:8200. The server is initialized and unsealed.
Insecure operation
Do not run a Vault dev server in production. This approach starts a Vault server with an in-memory database and runs in an insecure way.
Return to the terminal where you created the TUTORIAL_TEMP directory.
Export an environment variable for the vault CLI to address the Vault server.
```
$ export VAULT_ADDR=http://127.0.0.1:8200
```
Export an environment variable for the vault CLI to authenticate with the Vault server.
```
$ export VAULT_TOKEN=root
```
Note
For these tasks, you can use Vault's root token. However, it is recommended that root tokens are only used for enough initial setup or in emergencies. As a best practice, use an authentication method or token that meets the policy requirements.
The Vault server is ready.

Configure GCP services for Vault

(Persona: admin)

Warning

Resources will be provisioned during this tutorial that may result in charges to your Google Cloud Platform account.

Before Vault can authenticate GCP IAM service accounts and GCE instances, you need to configure the necessary resources in GCP. This includes enabling the required GCP APIs, creating a IAM service account and IAM policy for Vault, and creating a key for the service account that Vault will use to authenticate with GCP.

Open a browser and sign into the GCP console.
Click the terminal icon to launch the GCP Cloud Shell.
A new pane will open at the bottom of your window and provision a Cloud Shell instance.
In the Cloud Shell terminal, list the GCP project ID.
Note
If this is the first time using the Cloud Shell terminal, you will be prompted to authorize permission to use your credentials.
Click Authorize when prompted.
```
$ gcloud projects list

PROJECT_ID: abc-123defde860044c59e853ee2819
NAME: my-gcp-account-name
PROJECT_NUMBER: 85527753462
```
The project ID is used when configuring both GCP and Vault.

Verify the cloud resource manager and IAM API's are enabled.

$ gcloud services list --enabled | grep 'resource\|iam\|compute'

NAME: cloudresourcemanager.googleapis.com
NAME: compute.googleapis.com
NAME: iam.googleapis.com
NAME: iamcredentials.googleapis.com

If cloudresourcemanager.googleapis.com, compute.googleapis.com, or iam.googleapis.com are not listed, expand the menu below and enable the required API(s).

Enable GCP APIs

Enable the iam.googleapis.com API.

$ gcloud services enable iam.googleapis.com

Enable the compute.googleapis.com API.

$ gcloud services enable compute.googleapis.com

Enable the cloudresourcemanager.googleapis.com API.

$ gcloud services enable cloudresourcemanager.googleapis.com

Create a service account for Vault to authenticate with GCP.

$ gcloud iam service-accounts create \
    VaultServiceAccount \
    --display-name="VaultServiceAccount"

Create a GCP IAM role for Vault and assign the required permissions.
Note
The permissions assigned in this tutorial are for both IAM and GCE.
If you only require one of these services, remove the unused permissions.
```
$ gcloud iam roles create VaultServiceRole \
    --project=$DEVSHELL_PROJECT_ID \
    --title=VaultServiceRole \
    --stage=GA \
    --permissions=iam.serviceAccounts.get,iam.serviceAccountKeys.get,compute.instances.get,compute.instanceGroups.list,iam.serviceAccounts.signJwt
```
Note
The iam.serviceAccounts.signJwt permission is required for any service account used to authenticate with Vault using the IAM method. If you are not using the IAM method, you can remove this permission.

Retrieve the role name and store it in the ROLE_NAME environment variable.

$ ROLE_NAME=$(gcloud iam roles list --project=$DEVSHELL_PROJECT_ID --format=json --filter="vault" | jq -r .[].name)

Retrieve the service account email and store it in the SERVICE_ACCOUNT environment variable.

$ SERVICE_ACCOUNT=$(gcloud iam service-accounts list --project=$DEVSHELL_PROJECT_ID --filter=vault --format=json | jq -r .[].email)

Add the Vault service account to the GCP project and bind the role to the service account.

$ gcloud projects add-iam-policy-binding $DEVSHELL_PROJECT_ID \
    --member="serviceAccount:$SERVICE_ACCOUNT" \
    --role="$ROLE_NAME"

Create a service account key credential file.

$ gcloud iam service-accounts keys create VaultServiceAccountKey.json \
    --iam-account=$SERVICE_ACCOUNT \
    --project=$DEVSHELL_PROJECT_ID

Click the More button (represented by the vertical ellipsis) and select Download.
Click the folder icon, expand the directory, select the VaultServiceAccountKey.json file, and click Download.
Store the VaultServiceAccountKey.json file in TUTORIAL_TEMP directory you created in the lab setup section.
Tip
Depending on your browser settings, the file may automatically download to a default directory. To complete this tutorial, make sure the VaultServiceAccountKey.json file is located in the $TUTORIAL_TEMP directory.
Example OSX command:
$ mv ~/Downloads/VaultServiceAccountKey.json $TUTORIAL_TEMP/
Retrieve and copy the GCP project ID from the Cloud Shell terminal.
```
$ echo $DEVSHELL_PROJECT_ID
```
Return to the terminal where you set the TUTORIAL_TEMP environment variable and set the GCP_PROJECT_ID environment variable to the value from the previous step.
```
$ export GCP_PROJECT_ID=<actual-project-id>
```
You have completed the necessary configuration in GCP to support the Vault GCP secrets engine.

Configure Vault

(Persona: admin)

Note

The GCP auth method focuses on identities specific to Google Cloud and does not support authenticating arbitrary Google or Google Workspace users or generic OAuth against Google.

In the terminal where you set the TUTORIAL_TEMP environment variable, enable the GCP secrets engine.
```
$ vault auth enable gcp

Success! Enabled gcp auth method at: gcp/
```

Configure the GCP auth method to use the VaultServiceAccountKey.json credentials.

$ vault write auth/gcp/config \
    credentials=@$TUTORIAL_TEMP/VaultServiceAccountKey.json

Example output:

Success! Data written to: auth/gcp/config

Create a policy file named dev.hcl.

$ tee dev.hcl <<EOF
  # Read permission on the k/v secrets
  path "/secret/*" {
      capabilities = ["read", "list"]
  }
EOF

Create a policy named dev with the policy defined in dev.hcl.

$ vault policy write dev dev.hcl

Success! Uploaded policy: dev

Create a role for GCE instances.
A role includes several parameters.
- policies: One or more comma separated Vault policies that will be applied.
- bound_projects: The name of your GCP project.
- bound_zones: One or more comma separated GCP zones permitted to authenticate with Vault.
```
$ vault write auth/gcp/role/vault-gce-auth-role \
   type="gce" \
   policies="dev" \
   bound_projects="$GCP_PROJECT_ID" \
   bound_zones="us-east1-b" \
```
Review the GCP auth method API documentation for a full list of supported parameters.
The Vault configuration is complete

Return to the GCP console. Retrieve and copy the service account email used for Vault from the Cloud Shell terminal.

$ gcloud iam service-accounts list --project=$DEVSHELL_PROJECT_ID --filter=vault --format=json | jq -r .[].email

Return to the terminal where you set the TUTORIAL_TEMP environment variable and set the GCP_SERVICE_EMAIL environment variable to the value from the previous step.
```
$ export GCP_SERVICE_EMAIL=<actual-email-id>
```
In the terminal where you set the TUTORIAL_TEMP environment variable, enable the GCP secrets engine.
```
$ vault auth enable gcp

Success! Enabled gcp auth method at: gcp/
```

Configure the GCP auth method to use the VaultServiceAccountKey.json credentials.

$ vault write auth/gcp/config \
    credentials=@$TUTORIAL_TEMP/VaultServiceAccountKey.json

Example output:

Success! Data written to: auth/gcp/config

Create a policy file named dev.hcl.

$ tee dev.hcl <<EOF
  # Read permission on the k/v secrets
  path "/secret/*" {
      capabilities = ["read", "list"]
  }
EOF

Create a policy named dev with the policy defined in dev.hcl.

$ vault policy write dev dev.hcl

Success! Uploaded policy: dev

Create a role for IAM service account.
A role includes several parameters.
- policies: One or more comma separated Vault policies that will be applied.
- bound_service_accounts: The service account for which you wish to authenticate with Vault.
Note
The service account created in this tutorial is used for simplicity.
```
$ vault write auth/gcp/role/vault-iam-auth-role \
   type="iam" \
   policies="dev" \
   bound_service_accounts="$GCP_SERVICE_EMAIL"
```
Review the GCP auth method API documentation for a full list of supported parameters.
The Vault configuration is complete

Authenticate with Vault from GCP

(Persona: app)

To complete authentication with Vault, a signed JSON Web Token must be passed to Vault. The Vault binary simplifies this process. Refer to the generating JWTs documentation for steps on retrieving the JWT manually.

Deploy GCE VM

Return to the GCP console.
Expand the hamburger menu and navigate to Compute Engine >> VM instances.
Click CREATE INSTANCE with the following details:
- Name: vault-auth-test
- Region: us-east1
- Zone: us-east1-b
Keep all other options set to their default values and click Create.
When the instance becomes available, click SSH. A new window will open and log you into the VM.

Setup Vault CLI

The Vault binary will require access to your Vault Dedicated cluster. Ensure that you followed the Lab setup section and enabled public access. In production environments, you can connect your GCP account via an AWS transit gateway and use the Vault Dedicated private address.

Download the HashiCorp GPG key.

$ curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg

Add the HashiCorp repo.

$ echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list

Install Vault.

$ sudo apt update && sudo apt install vault

Return to the HCP Portal and login if necessary.
Click Vault in the left navigation pane.
In the Vault clusters pane, click vault-cluster.
Under Cluster URLs, click Public Cluster URL.
Return to the SSH session for the GCP instance and set the VAULT_ADDR environment variable to the copied address.
```
$ export VAULT_ADDR=<Public_Cluster_URL>
```
Set the VAULT_NAMESPACE environment variable to admin.
```
$ export VAULT_NAMESPACE=admin
```

Authenticate with Vault using the vault-gce-auth-role role.

$ vault login -method=gcp role="vault-gce-auth-role"

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                                       Value
---                                       -----
token                                     hvs.CAESIJr5Wfq_LO0TOVb4CKqkHq1bJGeSkEX5dvdm8f6QM8MBGicKImh2cy4zTTlNaE1iZFV0T0pVZFNreG43TUhZWjAuQ0xHemgQsQY
token_accessor                            V0eS8zohjaTBnLDQEucMsJKM.CLGzh
token_duration                            1h
token_renewable                           true
token_policies                            ["default" "dev"]
identity_policies                         []
policies                                  ["default" "dev"]
token_meta_project_number                 1060482098095
token_meta_zone                           us-east1-b
token_meta_instance_creation_timestamp    1678121533
token_meta_instance_id                    1617931539497854164
token_meta_project_id                     hc-c0af9a5c6c9a42328ef9b7ce806
token_meta_service_account_id             115056664069215052875
token_meta_instance_name                  instance-1
token_meta_role                           vault-gce-auth-role
token_meta_service_account_email          1060482098095-compute@developer.gserviceaccount.com

You have successfully authenticated with Vault using the GCP auth method from a GCE instance.

The dev policy was assigned through the vault-gce-auth-role.

The Vault binary will require access to your Vault cluster. To support this tutorial, you will create a tunnel using ngrok and connect to the Vault dev mode server you started in the Lab setup section.

In another terminal on your local workstation, start ngrok and connect to Vault.

$ ngrok http http://127.0.0.1:8200

Example output:

ngrok                                                                                                                                                              (Ctrl+C to quit)

Session Status                online
Account                       username (Plan: Free)
Update                        update available (version 3.0.5, Ctrl-U to update)
Version                       3.0.3
Region                        United States (us)
Latency                       32.791235ms
Web Interface                 http://127.0.0.1:4040
Forwarding                    https://d12b-34-567-89-10.ngrok.io -> https://127.0.0.1:8200

Connections                   ttl     opn     rt1     rt5     p50     p90
                              0       0       0.00    0.00    0.00    0.00

Copy the full ngrok address including https://.
Return to the GCE SSH session and set an environment variable for the Vault ngrok address.
```
$ export VAULT_ADDR=<actual-address-from-ngrok>
```

Download the HashiCorp GPG key.

$ curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg

Add the HashiCorp repo.

$ echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list

Install Vault.

$ sudo apt update && sudo apt install vault

Authenticate with Vault using the vault-gce-auth-role role.

$ vault login -method=gcp role="vault-gce-auth-role"

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                                       Value
---                                       -----
token                                     hvs.CAESIJr5Wfq_LO0TOVb4CKqkHq1bJGeSkEX5dvdm8f6QM8MBGicKImh2cy4zTTlNaE1iZFV0T0pVZFNreG43TUhZWjAuQ0xHemgQsQY
token_accessor                            V0eS8zohjaTBnLDQEucMsJKM.CLGzh
token_duration                            1h
token_renewable                           true
token_policies                            ["default" "dev"]
identity_policies                         []
policies                                  ["default" "dev"]
token_meta_project_number                 1060482098095
token_meta_zone                           us-east1-b
token_meta_instance_creation_timestamp    1678121533
token_meta_instance_id                    1617931539497854164
token_meta_project_id                     hc-c0af9a5c6c9a42328ef9b7ce806
token_meta_service_account_id             115056664069215052875
token_meta_instance_name                  instance-1
token_meta_role                           vault-gce-auth-role
token_meta_service_account_email          1060482098095-compute@developer.gserviceaccount.com

You have successfully authenticated with Vault using the GCP auth method from a GCE instance.

The dev policy was assigned through the vault-gce-auth-role.

To authenticate with Vault using a service account, you must have access to the JSON formatted service account keys. The VaultServiceAccountKey.json file has already been downloaded to your workstation in the Configure GCP services for Vault section.

In terminal where you set the TUTORIAL_TEMP environment variable, authenticate with Vault using the vault-iam-auth-role role.

$ vault login -method=gcp \
    role="vault-iam-auth-role" \
    service_account="$GCP_SERVICE_EMAIL" \
    jwt_exp="15m" \
    credentials=@$TUTORIAL_TEMP/VaultServiceAccountKey.json

Example output:

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                                 Value
---                                 -----
token                               hvs.CAESIGtGaiDK8GxaFX6x94zyXWB1FOqlaYuVG7OQt8NjrmsEGicKImh2cy5LUVdrMktsUHJKbnNOaGZOaGpmZkI2R1IuQ0xHemgQ4Ak
token_accessor                      d28u62fraUlKkjuqJB4stBpy.CLGzh
token_duration                      1h
token_renewable                     true
token_policies                      ["default" "dev"]
identity_policies                   []
policies                            ["default" "dev"]
token_meta_project_id               hc-c0af9a5c6c9a42328ef9b7ce806
token_meta_role                     vault-iam-auth-role
token_meta_service_account_email    VaultServiceAccount@hc-c0af9a5c6c9a42328ef9b7ce806.iam.gserviceaccount.com
token_meta_service_account_id       113395201674290626828

You have successfully authenticated with Vault using the GCP auth method for a IAM service account.

The dev policy was assigned through the vault-iam-auth-role.

Cleanup

To avoid unnecessary charges and having unused security resources in your GCP account, HCP account, or local workstation, remove the following:

Cleanup Vault

Return to the HCP Portal.
Delete any Vault clusters created for this tutorial.

Cleanup GCP

Return to the GCP console and close the Cloud Shell terminal.
Expand the hamburger menu and navigate to Compute Engine >> VM instances. Delete any GCE instances created for this tutorial.
Expand the hamburger menu and navigate to IAM & Admin >> Service accounts. Delete the VaultServiceAccount service account.
From the IAM & Admin console, click Roles and delete the VaultServiceRole role.

Cleanup local workstation

On your local workstation, switch to the terminal used to configure Vault and create an array of the local environment variables.
```
$ learnVariables=( \
GCP_PROJECT_ID \
GCP_SERVICE_EMAIL \
VAULT_ADDR \
VAULT_TOKEN \
VAULT_NAMESPACE
)
```

Unset the environment variables.

for v in ${learnVariables[@]}; do
unset $v
done

Remove the TUTORIAL_TEMP directory.

$ cd ~ && rm -rf $TUTORIAL_TEMP && unset TUTORIAL_TEMP

Help and reference

Google Cloud auth method documentation
Google Cloud API documentation
Code examples to generate JWTs
GCP auth method repository

OIDC authentication with Google Workspace

SAML authentication