Terraform
Terraform Enterprise Automated Recovery
This guide explains how to configure automated recovery for a Terraform Enterprise installation. The goal is to provide a short Mean-Time-To-Recovery (MTTR) in the event of an outage. There are two steps in the automated recovery process:
- Configure snapshots on your current install to backup data
- Provision a new Terraform Enterprise instance using the latest snapshot
This guide will walk through both of these steps.
Configure snapshots
Snapshots are taken on the Terraform Enterprise instance. That instance can have two types of data on it:
- Terraform Enterprise application data: The core product data such as run history, configuration history, state history. This data changes frequently.
- Terraform Enterprise installer data: The data used to configure Terraform Enterprise itself such as installation type, database connection settings, hostname. This data rarely changes.
In the Mounted Disk and External Services operational modes, only installer data is stored on the instance. Application data is stored in the mounted disk or in an external PostgreSQL instance.
Automated snapshots are more effective when using Mounted Disk or External Services as the amount of backed up data is smaller and less risky.
You can configure snapshots in the dashboard under Console Settings
, in the
Snapshot & Restore
section. We recommend selecting Enable Automatic
Scheduled Snapshots
. The interval depends on the operational mode, but we
recommend Daily for Mounted Disk or External Services because the snapshots
contain only configuration data, not application data.
Restore a snapshot in a new Terraform Enterprise instance
Version Checking
Note: replicatedctl
is located at /usr/local/bin/replicatedctl
. On some operating systems, /usr/local/bin
is not in the user's $PATH
. In these cases, either add /usr/local/bin
to the path or refer to replicatedctl
with the full path.
Using the restore mechanism requires Replicated version 2.17.0 or greater.
You can check the version using replicatedctl version
.
Script
Below are examples of restore scripts that are run on machine boot. There are many mechanisms that can run a script on boot (cloud-init, systemd, /etc/init.d). Which one is used is up to the user.
This script is presented as an example. Anyone using it needs to understand what it's doing and is free to modify it to meet any additional needs they have.
S3
This example uses S3 as the mechanism to store snapshots on a debian-based system.
#!/bin/bash
set -e -u -o pipefail
bucket=your_bucket_to_store_snapshots
region=region_of_the_bucket
access_key=aws_access_key_id
secret_key=aws_secret_access_key
access="--store s3 --s3-bucket $bucket --s3-region $region --s3-key-id $access_key --s3-secret-key $secret_key"
# jq is used by this script, so install it. For other Linux distros, either preinstall jq
# and remove these lines, or change to the mechanism your distro uses to install jq.
apt-get update
apt-get install -y jq
# Run the installer.
curl https://install.terraform.io/ptfe/stable | bash -s fast-timeouts
# Wait for replicated to start before proceeding
until replicatedctl system status --template '{{and (eq .Replicated "ready") (eq .Retraced "ready")}}' | grep -q true; do
sleep 1
echo "Replicated is not yet ready."
done
echo "Replicated is ready."
# This retrieves a list of all the snapshots currently available.
replicatedctl snapshot ls $access -o json > /tmp/snapshots.json
# Pull just the snapshot id out of the list of snapshots
id=$(jq -r 'sort_by(.finished) | .[-1].id // ""' /tmp/snapshots.json)
# If there are no snapshots available, exit out
if test "$id" = ""; then
echo "No snapshots found"
exit 1
fi
echo "Restoring snapshot: $id"
# Restore the detected snapshot. This ignores preflight checks to be sure the application
# is booted.
replicatedctl snapshot restore $access --dismiss-preflight-checks "$id"
# Wait until the application reports itself as running. This step can be removed if
# something upstream is prepared to wait for the application to finish booting.
until curl -f -s --connect-timeout 1 http://localhost/_health_check; do
sleep 1
done
echo
echo "Application booted!"
SFTP
This example uses sftp to store the snapshots.
#!/bin/bash
set -e -u -o pipefail
key_file=path_to_your_ssh_key
key="$(base64 -w0 "$key_file")"
host=sftp_server_hostname_or_ip
user=user_to_sftp_on_the_remote_server
access="--store sftp --sftp-host $host --sftp-user $user --sftp-key $key"
# jq is used by this script, so install it. For other Linux distros, either preinstall jq
# and remove these lines, or change to the mechanism your distro uses to install jq.
apt-get update
apt-get install -y jq
# Run the installer.
curl https://install.terraform.io/ptfe/stable | bash -s fast-timeouts
# Wait for replicated to start before proceeding
until replicatedctl system status --template '{{and (eq .Replicated "ready") (eq .Retraced "ready")}}' | grep -q true; do
sleep 1
echo "Replicated is not yet ready."
done
echo "Replicated is ready."
# This retrieves a list of all the snapshots currently available.
replicatedctl snapshot ls $access -o json > /tmp/snapshots.json
# Pull just the snapshot id out of the list of snapshots
id=$(jq -r 'sort_by(.finished) | .[-1].id // ""' /tmp/snapshots.json)
# If there are no snapshots available, exit out
if test "$id" = ""; then
echo "No snapshots found"
exit 1
fi
echo "Restoring snapshot: $id"
# Restore the detected snapshot. This ignores preflight checks to be sure the application
# is booted.
replicatedctl snapshot restore $access --dismiss-preflight-checks "$id"
# Wait until the application reports itself as running. This step can be removed if
# something upstream is prepared to wait for the application to finish booting.
until curl -f -s --connect-timeout 1 http://localhost/_health_check; do
sleep 1
done
echo
echo "Application booted!"
Local directory
This example uses a local directory to store the snapshots. If this is intended to be run on a brand new system, the local directory would be either mounted block device or a network filesystem like NFS or CIFS.
#!/bin/bash
set -e -u -o pipefail
path=absolute_path_to_directory_of_snapshots
access="--store local --path $path"
# jq is used by this script, so install it. For other Linux distros, either preinstall jq
# and remove these lines, or change to the mechanism your distro uses to install jq.
apt-get update
apt-get install -y jq
# Run the installer.
curl https://install.terraform.io/ptfe/stable | bash -s fast-timeouts
# Wait for replicated to start before proceeding
until replicatedctl system status --template '{{and (eq .Replicated "ready") (eq .Retraced "ready")}}' | grep -q true; do
sleep 1
echo "Replicated is not yet ready."
done
echo "Replicated is ready."
# This retrieves a list of all the snapshots currently available.
replicatedctl snapshot ls $access -o json > /tmp/snapshots.json
# Pull just the snapshot id out of the list of snapshots
id=$(jq -r 'sort_by(.finished) | .[-1].id // ""' /tmp/snapshots.json)
# If there are no snapshots available, exit out
if test "$id" = ""; then
echo "No snapshots found"
exit 1
fi
echo "Restoring snapshot: $id"
# Restore the detected snapshot. This ignores preflight checks to be sure the application
# is booted.
replicatedctl snapshot restore $access --dismiss-preflight-checks "$id"
# Wait until the application reports itself as running. This step can be removed if
# something upstream is prepared to wait for the application to finish booting.
until curl -f -s --connect-timeout 1 http://localhost/_health_check; do
sleep 1
done
echo
echo "Application booted!"
Airgap recovery considerations
The instructions above are tailored for the online install method. When restoring on an airgap instance, there are several additional considerations:
- The minimum version of Replicated is 2.31.0, rather than 2.17.0.
- The license file and airgap package must be in place on the new instance prior to restore. The restore process expects to find them in the same locations as they were on the original instance.
- The snapshot being used must also be from an airgap instance.
- The
install.sh
script and method used must be from the Replicated airgap installer boostrapper, using the process described for airgap installation.