HashiTalks 2025 Learn about unique use cases, homelab setups, and best practices at scale at our 24-hour virtual knowledge sharing event. Register
Vault
Certificate management

PKI Unified CRL and OCSP with cross cluster revocation

  • 25min
  • |
  • Enterprise
  • Vault

Requires Vault Enterprise

The PKI secrets engine unified CRL and OCSP, and cross cluster revocation features demonstrated here require a Vault Enterprise license.

A PKI secrets engine in Vault servers at versions before 1.13.0 writes certificate revocation data to the cluster that issues a certificate. This means revocations which require information from the issued certificate must also occur on that same local cluster.

The challenge for end-users in Performance Replication environments becomes tracking which cluster issued a given leaf certificate to verify validity or revoke the certificate by its serial number.

Solution

Vault Enterprise version 1.13.0 introduces new opt-in features which help resolve this challenge:

  • Unified CRL and OCSP functionality across clusters. Enabling this feature results in an eventually consistent, globally unified CRL based on the revocation entries of all clusters. With this feature enabled, Vault writes revocation entries from each cluster locally to the cluster that issued the certificate. It then also writes a revocation entry to the primary cluster that replicates across all Vault clusters.

  • Asynchronous cross cluster revocation. When you opt in to this feature, you gain the ability to revoke certificates from clusters other than the issuing cluster.

Scenario introduction

Diagram of scenario featuring two Vault clusters in Performance Replication with secrets engines and auth method

In this scenario, you will configure and start 2 Vault Enterprise servers, which will act as the Performance Replication primary and secondary clusters.

You will use the Vault CLI to enable and configure 4 PKI Secrets Engine instances along with a username and password auth method.

You will also use the Vault CLI to enable Performance Replication between the 2 servers and issue leaf certificates from each of the 4 secrets engine roles.

To wrap up, you will query and revoke the certificates to learn more about the unified CRL and OCSP, and cross cluster revocation features.

Personas

The steps described in this tutorial are typically performed by a security engineer.

Prerequisites

To perform the tasks described in this tutorial, you need:

  • A Vault Enterprise binary v1.13.0 or later installed and in your system PATH.
    • Vault token with policies to manage Performance Replication and secrets engines (this tutorial uses the initial root token).
  • jq binary installed and in system PATH for parsing JSON in terminal sessions.
  • A text editor that you can use in terminal sessions to edit scripts.
  • openssl binary installed and in system PATH for validating certificates. If your OS distribution provides an openssl binary, you can use it for this tutorial.
  • Familiarity with the Vault Enterprise Performance Replication feature; if you are not familiar, you can learn more in Setting up Performance Replication.
  • Familiarity with the Vault PKI Secrets Engine; if you are not familiar, you can learn more in Build Your Own Certificate Authority (CA).

Lab setup

The lab for this scenario is a simplified Performance Replication setup made up of 2 Vault Enterprise servers, which represent the primary and secondary clusters.

You will prepare the environment by making directories and writing configuration files.

You will then initialize, and unseal the Vault server for each cluster.

For the sake of simplicity in this lab, you will use the initial root token for all steps in this scenario.

Create lab home

You can create a temporary directory to hold all the content needed for this scenario, and then assign its path to an environment variable for later reference.

Open a terminal, and create the directory /tmp/learn-vault-pki.

$ mkdir /tmp/learn-vault-pki

Export the directory path as the value to the HC_LEARN_LAB environment variable.

$ export HC_LEARN_LAB=/tmp/learn-vault-pki

Set up primary cluster server

The primary cluster server uses Integrated Storage and listens on 127.0.0.1:8200 for client communications.

  1. Export your Vault Enterprise license string as the value of the VAULT_LICENSE environment variable.

    $ export VAULT_LICENSE=02MV4UU43BK5....

  2. Create the primary server configuration and data directories.

    $ mkdir -p $HC_LEARN_LAB/primary-cluster/{config,data}

  3. Write the primary server configuration file.

    $ cat > $HC_LEARN_LAB/primary-cluster/config/server.hcl << EOF
disable_mlock = true
ui            = true
api_addr      = "http://127.0.0.1:8200"
cluster_addr  = "http://127.0.0.1:8201"

listener "tcp" {
  address            = "0.0.0.0:8200"
  tls_disable        = "true"
}

storage "raft" {
  path    = "$HC_LEARN_LAB/primary-cluster/data"
  node_id = "primary-cluster"
}
EOF

  4. Start the primary cluster Vault server.

    $ vault server -config $HC_LEARN_LAB/primary-cluster/config/server.hcl

  5. Open another terminal session, and use it to export an environment variable for the vault CLI to address the primary server.

    $ export VAULT_ADDR=http://127.0.0.1:8200

  6. For the purposes of this scenario, you can initialize the server with just 1 key share and a key threshold of 1. Redirect the initialization information to the file $HC_LEARN_LAB/primary-cluster/.init.

    $ vault operator init -key-shares=1 -key-threshold=1 \
    > $HC_LEARN_LAB/primary-cluster/.init

  7. Export the primary cluster server unseal key as the value to the PRIMARY_UNSEAL_KEY environment variable.

    $ export PRIMARY_UNSEAL_KEY="$(grep 'Unseal Key 1' $HC_LEARN_LAB/primary-cluster/.init | awk '{print $NF}')"

  8. Unseal the primary cluster server.

    $ vault operator unseal $PRIMARY_UNSEAL_KEY

    Example expected output:

    Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            1
Threshold               1
Version                 1.13.0+ent
Build Date              2023-02-14T14:25:23Z
Storage Type            raft
Cluster Name            vault-cluster-ab95b64a
Cluster ID              d91ceaa5-859d-f053-a9ac-12f13e89c644
HA Enabled              true
HA Cluster              n/a
HA Mode                 standby
Active Node Address     <none>
Raft Committed Index    64
Raft Applied Index      64

    The primary cluster server is ready.

Set up secondary cluster server

The secondary cluster server uses Integrated Storage and listens on 127.0.0.1:8210 for client communications.

  1. Export your Vault Enterprise license string as the value of the VAULT_LICENSE environment variable.

    $ export VAULT_LICENSE=02MV4UU43BK5....

  2. Create the secondary server configuration and data directories.

    $ mkdir -p $HC_LEARN_LAB/secondary-cluster/{config,data}

  3. Write the secondary server configuration file.

    $ cat > $HC_LEARN_LAB/secondary-cluster/config/server.hcl << EOF
disable_mlock = true
ui            = true
api_addr      = "http://127.0.0.1:8210"
cluster_addr  = "http://127.0.0.1:8211"

listener "tcp" {
  address            = "0.0.0.0:8210"
  tls_disable        = "true"
}

storage "raft" {
  path    = "$HC_LEARN_LAB/secondary-cluster/data"
  node_id = "secondary-cluster"
}
EOF

  4. Start the secondary cluster server.

    $ vault server -config $HC_LEARN_LAB/secondary-cluster/config/server.hcl

  5. Open another terminal session, and in that one, export the directory path as the value to the HC_LEARN_LAB environment variable.

    $ export HC_LEARN_LAB=/tmp/learn-vault-pki

  6. Change into the lab directory.

    $ cd $HC_LEARN_LAB

  7. Export an environment variable for the vault CLI to address the secondary cluster server.

    $ export VAULT_ADDR=http://127.0.0.1:8210

  8. For the purposes of this scenario, you can initialize the server with just 1 key share and a key threshold of 1. Redirect the initialization information to the file $HC_LEARN_LAB/secondary-cluster/.init.

    $ vault operator init -key-shares=1 -key-threshold=1 \
    > $HC_LEARN_LAB/secondary-cluster/.init

  9. Export the secondary cluster server unseal key as the value to the SECONDARY_UNSEAL_KEY environment variable.

    $ export SECONDARY_UNSEAL_KEY="$(grep 'Unseal Key 1' $HC_LEARN_LAB/secondary-cluster/.init | awk '{print $NF}')"

  10. Unseal the secondary cluster server.

    $ vault operator unseal $SECONDARY_UNSEAL_KEY

    Example expected output:

    Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            1
Threshold               1
Version                 1.13.0+ent
Build Date              2023-02-14T14:25:23Z
Storage Type            raft
Cluster Name            vault-cluster-c272fe50
Cluster ID              62b78af3-9fad-eaac-17d4-3d3f0b0a64ca
HA Enabled              true
HA Cluster              n/a
HA Mode                 standby
Active Node Address     <none>
Raft Committed Index    64
Raft Applied Index      64

    The secondary cluster server is ready.

Enable PKI secrets engines on primary cluster server

Diagram shows enabling PKI secrets engines

The goal of this section is for you to enable and configure 4 instances of the PKI secrets engine, each with a different set of features available.

This way, you can learn about revocation behaviors in each case.

You will use a script to enable the following named secrets engine engines on the primary server:

  • pki-local: No unified revocation features enabled.
  • pki-unified: You will enable the unified CRL feature.
  • pki-cross: You will enable both unified CRL and cross cluster revocation features.

  1. Export an environment variable for the vault CLI to address the primary server.

    $ export VAULT_ADDR=http://127.0.0.1:8200

  2. Export the primary cluster server initial root token as the value to the VAULT_TOKEN environment variable.

    $ export VAULT_TOKEN="$(grep 'Initial Root Token' $HC_LEARN_LAB/primary-cluster/.init | awk '{print $NF}')"

  3. To save time and typing, you will use a script to enable and configure secrets engines based on steps like those in Build Your Own Certificate Authority (CA).

    Use a text editor to write and save the example shell script content to the file enable_engines.sh.

    enable_engines.sh

    #!/bin/sh
for engine in local unified cross both
do
   vault secrets enable -path "pki-$engine" pki
done
for engine in local unified cross both
do
   vault secrets tune -max-lease-ttl=87600h "pki-$engine"
done
for engine in local unified cross both
do
   vault write -field=certificate "pki-$engine/root/generate/internal" \
      common_name="$engine.example.com" \
      issuer_name="$engine-root-2023" \
      ttl=87600h > "$engine-local-root-2023-ca.crt"
   done
for engine in local unified cross both
do
   vault write "pki-$engine/roles/$engine-2023-servers" allow_any_name=true no_store=false
done
for engine in local unified cross both
do
   vault write pki-$engine/config/urls \
      issuing_certificates="$VAULT_ADDR/v1/pki-$engine/ca" \
      crl_distribution_points="$VAULT_ADDR/v1/pki-$engine/crl"
done
for engine in local unified cross both
do
   vault secrets enable -path=pki-int-$engine pki
done
for engine in local unified cross both
do
   vault secrets tune -max-lease-ttl=43800h "pki-int-$engine"
done
for engine in local unified cross both
do
   vault write -field=csr "pki-int-$engine/intermediate/generate/internal" \
      common_name="$engine.example.com Intermediate Authority" \
      issuer_name="$engine-example-dot-com-intermediate" \
      > "pki-int-$engine.csr"
done
for engine in local unified cross both
do
   vault write -field=certificate pki-$engine/root/sign-intermediate \
      issuer_ref="$engine-root-2023" \
      csr=@pki-int-$engine.csr \
      format=pem_bundle ttl="43800h" \
      > "pki-int-$engine.cert.pem"
done
for engine in local unified cross both
do
   vault write "pki-int-$engine/intermediate/set-signed" \
      certificate=@"pki-int-$engine.cert.pem"
done
for engine in local unified cross both
do
   vault write pki-int-$engine/roles/$engine-example-dot-com \
      issuer_ref="$(vault read -field=default pki-int-$engine/config/issuers)" \
      allowed_domains="$engine.example.com" \
      allow_subdomains=true \
      max_ttl="720h" \
      no_store=false
done

  4. Make the script executable.

    $ chmod +x enable_engines.sh

  5. Execute the script to enable and configure the secrets engines.

    $ ./enable_engines.sh

    Abbreviated expected output:

    Success! Enabled the pki secrets engine at: pki-local/
Success! Enabled the pki secrets engine at: pki-unified/
Success! Enabled the pki secrets engine at: pki-cross/
Success! Enabled the pki secrets engine at: pki-both/
Success! Tuned the secrets engine at: pki-local/
Success! Tuned the secrets engine at: pki-unified/
Success! Tuned the secrets engine at: pki-cross/
Success! Tuned the secrets engine at: pki-both/
...snip...
signature_bits                        256
street_address                        []
ttl                                   0s
use_csr_common_name                   true
use_csr_sans                          true
use_pss                               false

  6. List enabled secrets engines.

    $ vault secrets list

    Example expected output:

    Path                Type         Accessor              Description
----                ----         --------              -----------
cubbyhole/          cubbyhole    cubbyhole_f84fa2f4    per-token private secret storage
identity/           identity     identity_67f57adf     identity store
pki-both/           pki          pki_c8c83cd0          n/a
pki-cross/          pki          pki_a9cf7255          n/a
pki-int-both/       pki          pki_99e7f751          n/a
pki-int-cross/      pki          pki_18b227b7          n/a
pki-int-local/      pki          pki_187f3e0d          n/a
pki-int-unified/    pki          pki_db7cd9d8          n/a
pki-local/          pki          pki_c49dd19e          n/a
pki-unified/        pki          pki_2a596159          n/a
sys/                system       system_8f6f7690       system endpoints used for control, policy and debugging

Configure PKI secrets engines

Diagram shows configuring PKI secrets engines

The unified CRL and OCSP, and cross cluster revocation features are opt-in.

When you wrote and ran the shell script that enables and configures the PKI secrets engines, it did not configure the secrets engines with the new features. This means that 3 of the 4 PKI secrets engines require configuration updates. You can update them through the Set Revocation Configuration API endpoint.

The goal of this section is to learn about these configuration parameters, and apply them to the 3 secrets engines which require configuration updates.

Tip

If you need further details about these parameters, refer to their API documentation:

  1. Check the current CRL configuration for the pki-int-unified secrets engine.

    $ vault read pki-int-unified/config/crl

    Example expected output:

    Key                              Value
---                              -----
auto_rebuild                     false
auto_rebuild_grace_period        12h
cross_cluster_revocation         false
delta_rebuild_interval           15m
disable                          false
enable_delta                     false
expiry                           72h
ocsp_disable                     false
ocsp_expiry                      12h
unified_crl                      false
unified_crl_on_existing_paths    false

    Recall that the features covered in this tutorial are opt-in, so unified_crl is false as expected.

  2. The pki-int-unified secrets engine needs configuration to use just the unified CRL and OCSP feature and allow it for existing paths. Enabling this feature requires that you also set the auto_rebuild parameter to true, as unified CRLs cannot be rebuilt on every revocation.

    $ vault write pki-int-unified/config/crl \
    auto_rebuild=true \
    unified_crl=true \
    unified_crl_on_existing_paths=true

    Example expected output:

    Key                              Value
---                              -----
auto_rebuild                     true
auto_rebuild_grace_period        12h
cross_cluster_revocation         false
delta_rebuild_interval           15m
disable                          false
enable_delta                     false
expiry                           72h
ocsp_disable                     false
ocsp_expiry                      12h
unified_crl                      true
unified_crl_on_existing_paths    true

  3. The pki-int-cross secrets engine needs configuration to use just the cross cluster revocation feature. Enabling this feature requires that you also set the auto_rebuild parameter to true, as the local cluster may not have the certificate entry when auto_rebuild is not enabled.

    $ vault write pki-int-cross/config/crl \
       auto_rebuild=true \
       cross_cluster_revocation=true

    Example expected output:

    Key                              Value
---                              -----
auto_rebuild                     true
auto_rebuild_grace_period        12h
cross_cluster_revocation         true
delta_rebuild_interval           15m
disable                          false
enable_delta                     false
expiry                           72h
ocsp_disable                     false
ocsp_expiry                      12h
unified_crl                      false
unified_crl_on_existing_paths    false

  4. The pki-int-both secrets engine needs configuration to use both the unified CRL and OCSP, and cross cluster revocation features. Since this secrets engine combines both features, you must also set the auto_rebuild parameter to true.

    $ vault write pki-int-both/config/crl \
    auto_rebuild=true \
    unified_crl=true \
    unified_crl_on_existing_paths=true \
    cross_cluster_revocation=true

    Example expected output:

    Key                              Value
---                              -----
auto_rebuild                     true
auto_rebuild_grace_period        12h
cross_cluster_revocation         true
delta_rebuild_interval           15m
disable                          false
enable_delta                     false
expiry                           72h
ocsp_disable                     false
ocsp_expiry                      12h
unified_crl                      true
unified_crl_on_existing_paths    true

You have completed the PKI secrets engine configuration, and the PKI secrets engines are ready for certificate issuance.

Enable Performance Replication

Diagram showing enable Performance Replication workflow

The goal of this section is for you to enable a userpass auth method instance, a user and password, and Performance Replication between the primary and secondary cluster servers.

Primary server

Before enabling performance replication, create a superuser policy and enable userpass auth method with tester user on the primary cluster.

  1. Create a superuser ACL policy.

    $ vault policy write superuser - <<EOF
path "*" {
capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}
EOF

  2. Enable the userpass auth method.

    $ vault auth enable userpass

  3. Create a new user named tester in userpass with the password changeme, and attach the superuser policy.

    $ vault write auth/userpass/users/tester password="changeme" policies="superuser"

Later, you can authenticate with Vault using the userpass auth method.

Use the CLI to enable Performance Replication on the primary and secondary servers.

  1. Enable replication on the primary cluster server.

    $ vault write -f sys/replication/performance/primary/enable

    Example expected output:

    WARNING! The following warnings were returned from Vault:

* This cluster is being enabled as a primary for replication. Vault will be
unavailable for a brief period and will resume service shortly.

  2. Generate a secondary activation token and assign its value to the SECONDARY_ACTIVATION_TOKEN environment variable.

    $ export SECONDARY_ACTIVATION_TOKEN="$(vault write -field wrapping_token sys/replication/performance/primary/secondary-token id=secondary)"

    This command is not expected to produce output, but you can verify the environment variable if you like.

    $ echo $SECONDARY_ACTIVATION_TOKEN

    Expected example output:

    eyJhbGciOiJFUzUxMiIsInR5cCI6IkpXVCJ9.eyJhY2Nlc3NvciI6IiIsImFkZHIiOiJodHRwOi8vMTI3LjAuMC4xOjgyMDAiLCJleHAiOjE2NzY5OTgyNTYsImlhdCI6MTY3Njk5NjQ1NiwianRpIjoiaHZzLlJMYmVvbFlsY0Y0MUQwc0hJQjdZc2ZhciIsIm5iZiI6MTY3Njk5NjQ1MiwidHlwZSI6IndyYXBwaW5nIn0.AfnDspZjJ6MRHACJ6_lpAqoPajSJTtpr5421a2pA6x_C96Fj2TPfLrVmojgyiMkJuV-XEyT_yj02qUW3bIKy4WQiAHXXM8FkpvFD0uqKYcsGQuwrzJNcD1X6caDKZRApZGstsuUb3yws87Qjcd9R1dNWOsmTekDV-rPU2cnQqAAX2JmA

Secondary server

  1. Export an environment variable for the vault CLI to address the secondary server.

    $ export VAULT_ADDR=http://127.0.0.1:8210

  2. Export the secondary cluster server initial root token as the value to the VAULT_TOKEN environment variable.

    $ export VAULT_TOKEN="$(grep 'Initial Root Token' $HC_LEARN_LAB/secondary-cluster/.init | awk '{print $NF}')"

  3. Enable replication on the secondary server with the secondary activation token.

    Warning

    This will clear all data in the secondary cluster!

    $ vault write sys/replication/performance/secondary/enable \
     token=$SECONDARY_ACTIVATION_TOKEN

    Expected example output:

    WARNING! The following warnings were returned from Vault:

 * Vault has successfully found secondary information; it may take a while to
 perform setup tasks. Vault will be unavailable until these tasks and initial
 sync complete.

Issue certificates

Diagram showing PKI certificate issuance workflow

The goal of this section is for you to issue a leaf certificate from each of the PKI secrets engine roles on the secondary cluster server.

You can then try revoking the certificates and observing the different behaviors across each PKI secrets engine instance.

  1. Unset the VAULT_TOKEN environment variable.

    $ unset VAULT_TOKEN

  2. Before issuing the certificates, you must first use the userpass auth method to authenticate with the secondary cluster server.

    $ vault login -no-print -method userpass username=tester password=changeme

    This command produces no output, but you can confirm that the token helper cached the token value by displaying part of it.

    $ cat ~/.vault-token | cut -d '.' -f1

    Example expected output:

    hvs

    Tip

    If you meet with an error message containing * local node not active but active cluster node not found, when you try to login with the userpass auth method this means that the replication hasn't finished syncing. Wait a moment and try the command again.

  3. Issue a certificate from the pki-int-local PKI secrets engine local-example-dot-com role.

    $ vault write \
    pki-int-local/issue/local-example-dot-com \
    common_name="test.local.example.com" \
    ttl="1h" -format=json > test.local.example.com.json

    Store the serial number in a test.local.example.com.serial.txt file.

    $ cat test.local.example.com.json | jq -r '.data.serial_number' > test.local.example.com.serial.txt

    Store the certificate in the test.local.example.com.crt file.

    $ cat test.local.example.com.json | jq -r '.data.certificate' > test.local.example.com.crt

  4. Issue a certificate from the pki-int-unified PKI secrets engine unified-example-dot-com role.

    $ vault write \
    pki-int-unified/issue/unified-example-dot-com \
    common_name="test.unified.example.com" \
    ttl="1h" -format=json > test.unified.example.com.json

    Store its serial number in the test.unified.example.com.serial.txt file.

    $ cat test.unified.example.com.json | jq -r '.data.serial_number' > test.unified.example.com.serial.txt

    Store the certificate in the test.unified.example.com.crt file.

    $ cat test.unified.example.com.json | jq -r '.data.certificate' > test.unified.example.com.crt'

  5. Issue a certificate from the pki-int-cross PKI secrets engine cross-example-dot-com role.

    $ vault write \
    pki-int-cross/issue/cross-example-dot-com \
    common_name="test.cross.example.com" \
    ttl="1h" -format=json > test.cross.example.com.json

    Store its serial number in the test.cross.example.com.serial.txt file.

    $ cat test.cross.example.com.json | jq -r '.data.serial_number' > test.cross.example.com.serial.txt

    Store the certificate in the test.cross.example.com.crt file.

    $ cat test.cross.example.com.json | jq -r '.data.certificate' > test.cross.example.com.crt

  6. Issue a certificate from the pki-int-both PKI secrets engine both-example-dot-com role.

    $ vault write \
    pki-int-both/issue/both-example-dot-com \
    common_name="test.both.example.com" \
    ttl="1h" -format=json > test.both.example.com.json

    Store its serial number in the test.both.example.com.serial.txt file.

    $ cat test.both.example.com.json | jq -r '.data.serial_number' > test.both.example.com.serial.txt

    Store the certificate in the test.both.example.com.crt file.

    $ cat test.both.example.com.json | jq -r '.data.certificate' > test.both.example.com.crt

Explore behavior without features

Diagram showing revocation scenario for certificate issued from pki-local

The goal of this section is for you to query and revoke the leaf certificate from the pki-int-local PKI secrets engine. In doing so, you learn more about the PKI secrets engine behavior without the unified CRL and OCSP feature.

  1. Use openssl to query the secondary cluster server OCSP about the certificate that you issued from the pki-int-local PKI secrets engine.

    $ openssl ocsp \
    -noverify \
    -no_nonce \
    -issuer pki-int-local.cert.pem \
    -cert test.local.example.com.crt \
    -url http://127.0.0.1:8210/v1/pki-int-local/ocsp

    Example expected output:

    test.local.example.com.crt: good
    This Update: Feb 26 17:16:09 2023 GMT
    Next Update: Feb 27 05:16:09 2023 GMT

    According to the secondary cluster OCSP, the certificate is valid.

  2. Try revoking the certificate by its serial number through the primary cluster server. For convenience, you can set the environment variables for the primary cluster server and token as part of the command.

    $ VAULT_ADDR=http://127.0.0.1:8200 \
  VAULT_TOKEN="$(grep 'Initial Root Token' $HC_LEARN_LAB/primary-cluster/.init | awk '{print $NF}')" \
      vault write pki-int-local/revoke \
      serial_number=$(cat test.local.example.com.serial.txt)

    Instead of success, Vault outputs an error:

    Error writing data to pki-int-local/revoke: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/pki-int-local/revoke
Code: 400. Errors:

* certificate with serial 09:82:7f:46:5e:92:d1:aa:7e:c3:3f:3e:88:bb:d6:84:e8:ea:97:d7 not found.

    The primary cluster is not aware of the certificate issued from the secondary cluster. This demonstrates correct and expected behavior from the pki-int-local secrets engine, since it isn't configured with support for unified CRL and OCSP.

  3. Try revoking the certificate by its serial number through the secondary cluster server. For convenience, you can set the environment variables for the secondary cluster server and token as part of the command.

    $ VAULT_ADDR=http://127.0.0.1:8210 \
  VAULT_TOKEN="$(cat ~/.vault-token)" \
      vault write pki-int-local/revoke \
      serial_number=$(cat test.local.example.com.serial.txt)

    The certificate state shows a value of revoked.

    Key                        Value
---                        -----
revocation_time            1677431870
revocation_time_rfc3339    2023-02-26T17:17:50.560556Z
state                      revoked

    You revoked the certificate.

Explore unified CRL & OCSP behavior

Diagram describing certificate issuance and revocation workflow

The goal of this section is for you to revoke the leaf certificate from the pki-int-unified PKI secrets engine. Then, you will check the list of revoked certificates on the secondary cluster server and primary cluster server. In doing so, you learn more about the PKI secrets engine behavior with the unified CRL and OCSP feature enabled.

  1. Use openssl to query the secondary cluster server OCSP about the certificate that you issued from the pki-int-unified PKI secrets engine.

    $ openssl ocsp \
    -noverify \
    -no_nonce \
    -issuer pki-int-unified.cert.pem \
    -cert test.unified.example.com.crt \
    -url http://127.0.0.1:8210/v1/pki-int-unified/ocsp

    Example expected output:

    test.unified.example.com.crt: good
    This Update: Feb 26 17:18:58 2023 GMT
    Next Update: Feb 27 05:18:58 2023 GMT

    According to the secondary cluster OCSP, the certificate is valid.

  2. Revoke the certificate by its serial number through the secondary cluster server. For convenience, you can set the environment variables for the secondary cluster server and token as part of the command.

    $ VAULT_ADDR=http://127.0.0.1:8210 \
  VAULT_TOKEN="$(cat ~/.vault-token)" \
      vault write pki-int-unified/revoke \
      serial_number=$(cat test.unified.example.com.serial.txt)

    Key                        Value
---                        -----
revocation_time            1677431976
revocation_time_rfc3339    2023-02-26T17:19:36.438238Z
state                      revoked

    You revoked the certificate.

  3. List revoked certificates on the secondary cluster server.

    $ VAULT_ADDR=http://127.0.0.1:8210 \
  VAULT_TOKEN="$(cat ~/.vault-token)" \
      vault list /pki-int-unified/certs/revoked

    Example expected output:

    Keys
----
52:21:bd:4b:3c:12:b0:93:1b:0f:6e:c7:e9:9e:90:79:54:58:8e:fb

    You revoked the certificate.

    Without the unified CRL and OCSP feature, you cannot expect the primary cluster server to have knowledge of the certificate. With it enabled, the primary cluster server should also show the certificate as revoked.

  4. Use openssl to query the primary cluster server OCSP about the certificate.

    $ openssl ocsp \
     -noverify \
     -no_nonce \
     -issuer pki-int-unified.cert.pem \
     -cert test.unified.example.com.crt \
     -url http://127.0.0.1:8200/v1/pki-int-unified/ocsp

    Example expected output:

    test.unified.example.com.crt: revoked
    This Update: Feb 26 17:50:25 2023 GMT
    Next Update: Feb 26 17:52:25 2023 GMT
    Revocation Time: Feb 26 17:19:36 2023 GMT

    With unified CRL and OCSP, the secondary cluster server now also shows the certificate revocation.

Explore cross cluster revocation behavior

Diagram showing cross cluster revocation workflow

The goal of this section is for you to revoke the leaf certificate from the pki-int-cross PKI secrets engine. In doing so, you learn more about the PKI secrets engine behavior with the cross cluster revocation feature enabled.

  1. Use openssl to query the secondary cluster server OCSP about the certificate that you issued from the pki-int-cross PKI secrets engine.

    $ openssl ocsp \
    -noverify \
    -no_nonce \
    -issuer pki-int-cross.cert.pem \
    -cert test.cross.example.com.crt \
    -url http://127.0.0.1:8210/v1/pki-int-cross/ocsp

    Example expected output:

    test.cross.example.com.crt: good
    This Update: Feb 26 17:41:41 2023 GMT
    Next Update: Feb 27 05:41:41 2023 GMT

    According to the secondary cluster OCSP, the certificate is valid.

  2. Try to revoke the certificate by its serial number through the primary cluster server. For convenience, you can set the environment variables for the primary cluster server and token as part of the command.

    $ VAULT_ADDR=http://127.0.0.1:8200 \
  VAULT_TOKEN="$(grep 'Initial Root Token' $HC_LEARN_LAB/primary-cluster/.init | awk '{print $NF}')" \
      vault write pki-int-cross/revoke \
      serial_number=$(cat test.cross.example.com.serial.txt)

    Example expected output:

    WARNING! The following warnings were returned from Vault:

* Revocation request was not found on this present node. This request will
be in a pending state until the PR cluster which issued this certificate
sees the request and revokes the certificate. If no online cluster has this
certificate, the request will eventually be removed without revoking any
certificates.

Key      Value
---      -----
state    pending

    Vault warns that it did not find a revocation request on the primary cluster server, and that the request is now pending. This means that with cross cluster revocation enabled, the secondary will learn of the request and revoke the certificate.

  3. Try querying the secondary cluster server for the certificate from pki-int-cross.

    $ openssl ocsp \
    -noverify \
    -no_nonce \
    -issuer pki-int-cross.cert.pem \
    -cert test.cross.example.com.crt \
    -url http://127.0.0.1:8210/v1/pki-int-cross/ocsp

    Example expected output:

    test.cross.example.com.crt: revoked
    This Update: Feb 26 17:42:54 2023 GMT
    Next Update: Feb 27 05:42:54 2023 GMT
    Revocation Time: Feb 26 17:42:48 2023 GMT

    The cross cluster revocation functionality revoked the certificate.

Explore unified CRL & OCSP with cross cluster revocation behavior

Diagram showing workflow with both features

The goal of this section is for you to revoke the leaf certificate from the pki-int-both PKI secrets engine from the secondary cluster server. You will then query the certificate with OCSP from the primary cluster server. In doing so, you learn more about the PKI secrets engine behavior with both the unified CRL and OCSP, with cross cluster revocation features enabled.

  1. Use openssl to query the secondary cluster server OCSP about the certificate that you issued from the pki-int-both PKI secrets engine.

    $ openssl ocsp \
    -noverify \
    -no_nonce \
    -issuer pki-int-both.cert.pem \
    -cert test.both.example.com.crt \
    -url http://127.0.0.1:8210/v1/pki-int-both/ocsp

    Example expected output:

    test.both.example.com.crt: good
    This Update: Feb 26 17:41:41 2023 GMT
    Next Update: Feb 27 05:41:41 2023 GMT

    According to the secondary cluster OCSP, the certificate is valid.

  2. Try to revoke the certificate by its serial number through the primary cluster server. For convenience, you can set the environment variables for the primary cluster server and token as part of the command.

    $ VAULT_ADDR=http://127.0.0.1:8200 \
  VAULT_TOKEN="$(grep 'Initial Root Token' $HC_LEARN_LAB/primary-cluster/.init | awk '{print $NF}')" \
      vault write pki-int-both/revoke \
      serial_number=$(cat test.both.example.com.serial.txt)

    Example expected output:

    WARNING! The following warnings were returned from Vault:

* Revocation request was not found on this present node. This request will
be in a pending state until the PR cluster which issued this certificate
sees the request and revokes the certificate. If no online cluster has this
certificate, the request will eventually be removed without revoking any
certificates.

Key      Value
---      -----
state    pending

    Vault warns that it did not find a revocation request on the primary cluster server, and that the request is now pending. This means that with cross cluster revocation enabled, the secondary will learn of the request and revoke the certificate.

  3. Try querying the secondary cluster server for the certificate from pki-int-both.

    $ openssl ocsp \
    -noverify \
    -no_nonce \
    -issuer pki-int-both.cert.pem \
    -cert test.both.example.com.crt \
    -url http://127.0.0.1:8210/v1/pki-int-both/ocsp

    Example expected output:

    test.both.example.com.crt: revoked
    This Update: Feb 26 17:42:54 2023 GMT
    Next Update: Feb 27 05:42:54 2023 GMT
    Revocation Time: Feb 26 17:42:48 2023 GMT

  4. Use openssl to query the primary cluster server OCSP about the certificate.

    $ openssl ocsp \
     -noverify \
     -no_nonce \
     -issuer pki-int-both.cert.pem \
     -cert test.both.example.com.crt \
     -url http://127.0.0.1:8200/v1/pki-int-both/ocsp

    Example expected output:

    test.both.example.com.crt: revoked
    This Update: Feb 26 17:42:54 2023 GMT
    Next Update: Feb 27 05:42:54 2023 GMT
    Revocation Time: Feb 26 17:42:48 2023 GMT

    The cross cluster revocation functionality revoked the certificate. Both the primary and secondary server clusters are aware of the revocation.

Summary

You established a lab environment with PKI secrets engines, the userpass auth method, and Performance Replication enabled.

You also learned about the PKI secrets engine unified CRL and OCSP, and cross cluster revocation features and their capabilities by issuing, querying, and revoking certificates.

Usage caveats

Size limitations exist for individual entries within Vault depending on the chosen storage backend.

If your use cases accumulate a large quantities of revoked certificates across various clusters, the unified CRL could hit limits and become unusable.

If a large amount of revocations are occurring, the traffic from the additional writes across all the clusters and/or the background/periodic threads within Vault could overload resources.

Consult Vault Limits and Maximums for more details.

Clean up

  1. Stop the Vault server processes.

    $ pkill vault

  2. Change into your home directory.

    $ cd

  3. Remove the lab directory.

    $ rm -rf $HC_LEARN_LAB

  4. Unset the environment variables.

    $ unset HC_LEARN_LAB \
    PRIMARY_UNSEAL_KEY \
    SECONDARY_ACTIVATION_TOKEN \
    SECONDARY_UNSEAL_KEY \
    VAULT_ADDR \
    VAULT_LICENSE \
    VAULT_TOKEN \

  5. Close any unneeded terminal sessions.

Next steps

To continue learning about the Vault PKI secrets engine or Performance Replication, check out the following tutorials:

Help and reference

Previous

Manage certificates with ACME clients

 Next

Generate certificates with managed keys