Measure failover resilience

This topic describes how to measure the failover resilience for a Terraform Enterprise deployment connected to a PostgreSQL database cluster.

Overview

You can connect Terraform Enterprise to a database cluster so that the application can fail over to another database instance if there is an issue with the primary instance. Refer to the following topics for additional information:

• PostgreSQL database failover
• Connect to a PostgreSQL cluster

You can test the resilience of your failover system by develping and running test workloads against the database cluster and measuring the recovery time objective (RTO).

Note that RTO varies significantly based on organizational priorities and the complexity of the services involved.

Define test workloads

You can continuously execute a workload against Terraform Enterprise and measure the time since the last successful execution of the workload. With this continuous measurement running, you can then trigger a failover on the Postgres cluster and record the outage duration.

The following sequence of steps describe an example workload:

1. Reset workspace to cleanup any blocking run.
2. Create and upload a configuration version.
3. Create a run.
4. Wait for the plan to finish.
5. Discard the plan.

Define a protocol

Determine what success and failure means in terms of measuring RTO for your instances. The following criteria represent an example protocol:

1. Execute the workloads every fifteen seconds.
   1. If the workload does not report success within 10 seconds, the instance is unhealthy.
   2. The instance is healthy whenfive consecutive runs complete successfully.
   3. The instance is non-operational if any run fails.
2. Trigger a failover.
3. Wait until the system becomes operational.
   1. If the workload does not report success within 10 seconds, the instance is unhealthy.
   2. The instance is healthy whenfive consecutive runs complete successfully.
   3. The instance is non-operational if any run fails.
4. Complete 10 iterations.

Trigger a failover

Create a separate organization and workspace to prevent modifying the initial dataset and to enable you to repeat tests. Determine a mechanism for triggering a failover. For example, if Terraform Enterprise is connected to a database cluster hosted on AWS, you can use the relational database service (RDS) to trigger a failover in the AWS console:

$ aws rds failover-db-cluster --db-cluster-identifier <CLUSTER_NAME> --region us-west-2`

Compute metrics

Compute the RTO by logging the duration between the first failed run and the first of five consecutively successful runs. You can measure RTO using go-tfe client.

Patroni example

The following table contains example data collected by running test workloads against a Terraform Enterprise deployment connected to a PostgreSQL cluster running on Patroni:

Aurora example

The following table contains example data collected by running test workloads against a Terraform Enterprise deployment connected to a PostgreSQL cluster running on Aurora: