Standard procedure for restoring a Vault cluster

6min
|
Vault

The objective of this document is to provide a set of Standard Operating Procedures for restoring a Vault cluster from a snapshot, for either Consul or Raft Integrated Storage backends. The methods of restoration following a DR situation would be in response to data corruption or sabotage of which Disaster Recovery Replication might not be able to protect against.

Vault supports a number of storage backend types. Therefore, the exact steps to restore data to Vault will depend on your selected storage backend. The two recommended storage backend types are Consul and Integrated Storage, and so this document assumes either of these storage backends is being used.

Personas

This standard operating procedures is primarily aimed at operations personnel.

Prerequisites

The following prerequisite steps and knowledge are required in order to backup a Vault cluster. All of the following are required to understand or carry out before attempting to backup or restore Vault.

Working Knowledge of Vault: Some working knowledge of Vault is required in order to follow these Standard Operating Procedures.
Vault cluster configuration is defined: Vault (and Consul, where using it as a storage backend) infrastructure configured as per Vault Reference Architecture.
A cluster configuration as defined in either our Vault with Integrated Storage Reference Architecture is required.
Vault has been initialised: This SOP assumes you have already initialised Vault, keyholders are available with access to the unseal keys for each, that you have access to tokens with sufficient privileges for both clusters and encrypted data is stored in the storage backend.

Procedures

Follow these steps to backup Vault manually. Note that the exact steps needed to be undertaken differs dependent upon your Vault architecture (e.g. Single cluster, using Disaster Recovery or Performance Replication).

Single Vault Cluster

Bring your Vault cluster back online following the circumstances that required you to restore from backup. You will need to reinitialise your Vault cluster and log in with the new root token that was generated during its reinitialisation. Note that these will be temporary- the original unseal keys will be needed following restore.
Copy your Vault Raft Snapshot file onto a Vault cluster member and run the below command, replacing the filename with that of your snapshot file. Note, the -force option is required here since the Auto-unseal or Shamir keys will not be consistent with the snapshot data as you will be restoring a snapshot from a different cluster.
```
$ vault operator raft snapshot restore -force backup.snap
```
Once you have restored the Raft snapshot you will need to unseal your Vault cluster again using the following command
```
$ vault operator unseal [unseal_key]
```

Bring your Vault and Consul clusters back online following the circumstances that required you to restore from backup (outside the scope of these SOPs). Note that there is no need to re-initialise your Vault cluster or, in the case of Vault Enterprise, apply a license to it.
Copy your Consul Snapshot file onto a Consul cluster member and run the following command, replacing the filename with that of your snapshot file.
```
$ consul snapshot restore backup.snap
```
Once you have restored the Consul snapshot you will need to unseal your Vault cluster again using the following command
```
$ vault operator unseal [unseal_key]
```

Vault with Disaster Recovery Replication Enabled

Bring your Vault cluster back online following the circumstances that required you to restore from backup. You will need to reinitialise your Vault cluster and log in with the new root token that was generated during its reinitialisation. Note that these will be temporary- the original unseal keys will be needed following restore.
Copy your Vault Raft Snapshots for the Primary and DR replica clusters onto restored members of the respective clusters and run the below following command, replacing the filename with that of your snapshot file. Note, the -force option is required here since the Autounseal/Shamir keys will not be consistent with the snapshot data as you will be restoring a snapshot from a different cluster.
```
$ vault operator raft snapshot restore -force backup.snap
```
Once you have restored the Raft snapshot you will need to unseal your Vault cluster again using the following command
```
$ vault operator unseal [unseal_key]
```

Bring your Vault and Consul clusters back online following the circumstances that required you to restore from backup. Note that there is no need to reinitialise your Vault cluster or, in the case of Vault Enterprise, apply a license to it.
Copy your Consul snapshot files onto a member of the respective Consul cluster that is supporting Vault's Primary and Secondary DR Replica clusters.
```
$ consul snapshot restore backup.snap
```
Once you have restored the Consul snapshot, use the Primary's unseal key to unseal both the Primary and Secondary Disaster Recovery Replica Vault clusters.
```
$ vault operator unseal [unseal_key]
```

Vault with Performance Replication Enabled

Bring your Vault cluster back online following the circumstances that required you to restore from backup. You will need to reinitialise your Vault cluster and log in with the new root token that was generated during its reinitialisation. Note that these will be temporary- the original unseal keys will be needed following restore.
Copy your Vault Raft Snapshots for the Primary and Secondary Performance Replica clusters onto restored members of the respective clusters and run the below following command, replacing the filename with that of your snapshot file. Note, the -force option is required here since the Auto-unseal or Shamir keys will not be consistent with the snapshot data as you will be restoring a snapshot from a different cluster.
```
$ vault operator raft snapshot restore -force backup.snap
```
Once you have restored the Raft snapshot you will need to unseal your Vault cluster again using the following command
```
$ vault operator unseal [unseal_key]
```

Bring your Vault and Consul clusters back online following the circumstances that required you to restore from backup. Note that there is no need to reinitialise your Vault cluster or, in the case of Vault Enterprise, apply a license to it.
Copy your Consul snapshot files onto a member of the respective Consul cluster that is supporting Vault's Primary and Secondary Performance Replica clusters.
```
consul snapshot restore backup.snap
```
Once you have restored the Consul snapshot, use the Primary's unseal key to unseal both the Primary and Secondary Performance Replica Vault clusters.
```
$ vault operator unseal [unseal_key]
```

Vault upgrade standard procedure

Next Collection

App integration