Terraform Enterprise backup - recommended pattern

If you are using the online installation method, configure the boot script to run the Replicated install.sh script explicitly without the airgap argument when the new VM starts up. The VM will download the installation media from the Internet and install the service.

Based on the Replicated configuration, the application will connect to the configured object store and database resources automatically.

If you are using the air-gapped installation method, use one of the following ways to ensure the installation media is available to the install configuration.

• Include the Terraform Enterprise installation media files on the machine images used to provision the instance (phoenix build practice). Use an automated build pipeline that can manage images, such as Packer. Do not install Terraform Enterprise as part of the image build because the installer needs to retrieve the host configuration data, which is not available before the VM starts. Arrange for Terraform Enterprise to install from the media on the first boot. This method gives you a faster recovery time because the installation media is already on disk, ready for installation. You can also patch the images automatically.
• Use a dedicated object store bucket or blob to store the installation media (a 'bootstrap' bucket), if you are on public cloud.
• Store your installation media in a network-based artifact repository. The time it takes to transfer the installation media from the repository to the application server could adversely impact your RTO if the media is not in close network proximity (for example, transferring from an on-premise location to a deployment in a cloud environment).

Use an Auto Scaling group (ASG) to automatically replace nodes on AWS. Select your deployment for more details.

• For both Standalone and Active/Active deployment, populate the ASG vpc_zone_identifier list with at least two subnets. If the region supports additional subnets, we recommend a minimum of three subnets since it provides n-2 AZ redundancy.

Use a zone-balanced Linux virtual machine scale set (VMSS) to automatically replace nodes on Azure. Select your deployment for more details.

• For both Standalone and Active/Active deployment, if using Terraform to deploy, use the azurerm_linux_virtual_machine_scale_set resource. Set the zones to a minimum of two (preferably three) zones in the region, and set zone_balance to true, which provides zone redundancy.

Use a regional managed instance group (MIG) to automatically replace nodes on GCP. Select your deployment for more details.

• For both Standalone and Active/Active deployment, if using Terraform to deploy, use the google_compute_region_instance_group_manager resource as this deploys a regional MIG, and thus ensures that the application server layer can automatically recover from a zone failure.

In an External Services mode scenario, the application server is running as a stateless node.

• If you deployed the instance from code (where the sensitive data listed above is available to the repaving of the node), you do not need to back up the instance. Redeploying the application server is acceptable.
• If you deployed the instance through a vRA/vRO experience and need to manually configure it after it comes up, you should back up the instance. Available VMWare operational practices are likely to already include a strategic backup solution, which we would recommend using in this case. There are many VMware backup solutions, and the relative merits are outside the scope of this guide. HashiCorp customers have successfully backed up Terraform Enterprise virtual machines using Dell EMC RP4VM and Veritas NetBackup, however, the suitability of these products for your business is also out of the scope of this guide and will need to be decided separately.

The most likely problem with the object store is service inaccessibility or corruption through human error rather than loss of durability, due to AWS's claim of eleven 9s of durability.

As a result, S3 Same-Region Replication is not explicitly required for the Terraform Enterprise object store because it does not add sufficient value: corruption on the primary S3 bucket will be replicated to the secondary automatically.

We recommend the following to ensure you back up your application data appropriately.

• Refer to AWS's S3 FAQs for information about S3's durability.
• Implement the security best practices for Amazon S3.
• Enable versioning on the bucket used as the object store. If you are using a bucket as a bootstrap store to contain installation media, enable versioning on the bucket.
• You should use S3 Standard class buckets.
• The buckets should be in the same region as the EC2 worker node(s).
• Use the VPC endpoint for S3.
• AWS Backup is not an option for object stores because S3 is not supported as a source.

The most likely problem with the object store is service inaccessibility or corruption through human error rather than loss of durability, due to Azure's claim of eleven 9s of durability.

We recommend the following to ensure you back up your application data appropriately.

• Use a zone-redundant storage account blob (ZRS).
• Implement the security recommendations for Azure blob storage.
• Enable point-in-time restores.
  • In order to do this, you will need to also enable its prerequisites on the storage account:
    • Enable soft delete for blobs and set retention to the maximum 365 days.
    • Enable the change feed
    • Enable blob versioning
  • While Microsoft recommend maintaining fewer than one thousand versions per blob, Terraform Enterprise does not overwrite existing objects, so additional blob versions will only exist as a result of workspace or accidental deletion operations.
  • Note the Azure limitations on storage container deletion.
• Use a Hot access tier, General-purpose V2 Azure blob storage account.
• The bucket should be in the same region as the worker node.
• Use the Microsoft.Storage endpoint.

The most likely problem with the object store is service inaccessibility or corruption through human error rather than loss of durability, due to GCP's claim of eleven 9s of durability.

We recommend the following to ensure you back up your application data appropriately.

• Refer to Google's Storage documentation for information about Google Cloud Storage's (GCS) durability.
• Implement the security recommendations for GCS.
• Enable object versioning.
• Use a regional Standard Storage cloud storage bucket as an object store.
• The bucket should be in the same region as the managed instance group.

For on-premise External Services deployments, as the architectural requirements include an S3-compatible storage facility, such as minIO or Dell ECS:

• If using minIO, use an active-passive configuration to ensure that objects are replicated. Enable object versioning on the buckets. Refer to minIO setup guide for more information.
• If using Dell ECS, we recommend using the default erasure coding scheme (12+4) and, therefore, a minimum of four discrete nodes in the storage cluster intraregion. Because Terraform providers comprise a significant percentage of stored slug objects (created for each workspace run), the slug size stored could be larger or smaller than 128Mb. Refer to the Dell white paper for information about ECS high availability design. We recommend you back up your ECS rig and enable cross-DC replication (refer to Multi-Region Considerations below).

In addition to the general recommendations above, consider the following AWS-specific recommendations:

• Implement the security best practices for Amazon RDS.
• Use a multi-AZ deployment. AWS will create the primary DB instance in the Primary AZ and synchronously replicate the contents to the standby instance in the Secondary AZ. Refer to AWS's high availability for RDS documentation for more information.
• Configure the PostgreSQL database in line with the HashiCorp Terraform Enterprise AWS Reference Architecture.
• Configure AWS Backup for RDS, using continuous backup and point-in-time-recovery (PITR).
• If using Aurora as the Terraform Enterprise RDS database, you automatically benefit from point-in-time recovery, continuous backup to Amazon S3, and replication across three availability zones. How many days of retention you require is a business decision, but HashiCorp recommends the maximum 35-day retention for maximum flexibility, particularly in regulated environments.
• Additionally configure database snapshots.
  • The default DB backup is taken from the standby instance once a day. Snapshots can be used to achieve an RPO of less than a day, and also to facilitate region-redundancy if they are stored in region-replicated buckets. They are also persist beyond the 35-day PITR window above.
  • Trigger DB snapshots automatically at required points during the day. The organization's needs and RPO will determine when you should trigger DB snapshots.
  • Continuously monitor the length of time it takes to do the backup and compare this to the RPO to avoid overlapping backups.
• In both cases, ensure that backups and snapshots are secure and restrict access to only required staff.
• Keep up-to-date with the AWS RDS documentation.
• Actively and continuously monitor operational health, and configure automatic event notifications.
• Store your snapshots in Amazon S3 in the same region as the platform to reduce recovery time.
• If Terraform is being used to deploy standard RDS, in the aws_db_instance resource:
  • Set backup_window to a suitable period in line with company policy and regulations. The default backup window should be 30 minutes.
  • Set multi_az to true
• If Terraform is being used to deploy Aurora RDS, in the aws_rds_cluster resource:
  • Set availability_zones to a list of at least three EC2 availability zones. AWS will increase availability zones to at least three if you specify less than three; however, we recommend using at least three to maximize the database layer's recoverability.
  • Set preferred_backup_window and preferred_maintenance_window to times convenient to your business model.
• For both RDS and Aurora RDS, if using Terraform as above, set backup_retention_period to suitable periods according to company policy and regulations. The recommended retention is the current maximum of 35 days since this maximizes the recoverability of the data; however, you should be aware of the costs associated with the potential level of data retention. Use snapshots to retain DB copies for longer than this.

In addition to the general recommendations above, consider the following Azure-specific recommendations:

• Follow the performance best practices for Azure Database for PostgreSQL.
• Enable high availability on the Azure Database for PostgreSQL service (default configuration).
  • This provides automatic single-region redundancy for recoverable errors in seconds.
  • For unrecoverable errors such as data corruption or user error, configure a point-in-time recovery which would require database snapshots.
  • Consider the overview of BC with Azure Database for PostgreSQL - Single Server.
• Configure the PostgreSQL database in line with the HashiCorp Terraform Enterprise Azure Reference Architecture and the Azure backup and restore guide.
• Keep up-to-date with the Azure Database for PostgreSQL documentation.
• Always ensure that event notifications and operational health are being actively monitored.
• Ensure that backups and snapshots are secure and restrict access to only required staff. Azure encrypted automatic database snapshots by default.
• Use servers that can support up to 16Tb storage - bear in mind the supported regions for these. This option currently provides eight days of coverage.
• If Terraform is being used to deploy Azure Database for PostgreSQL, set the azurerm_postgresql_server resource's backup_retention_days to a suitable period inline with company policy and regulations. The recommended retention is the current maximum of 35 days since this maximizes the recoverability of the data; however, you should be aware of the costs associated with the potential level of data retention. Use snapshots to retain DB copies for longer than this.

In addition to the general recommendations above, consider the following GCP-specific recommendations:

• Adopt Google CloudSQL best practices detailed here. In particular, note that exports take longer to create because an external file is created in Cloud Storage that can be used to recreate your data. Exports are unaffected if you delete the instance.
• Use a multi-AZ HA regional deployment instance so GCP automatically creates the primary DB instance in the Primary AZ and synchronously replicates the contents to the standby instance in the Secondary AZ. Failover is automatic. Refer to GCP's Replication in Cloud SQL documentation for more information.
• Configure the PostgreSQL database in line with the HashiCorp Terraform Enterprise GCP Reference Architecture.
• Keep up-to-date with the GCP CloudSQL documentation.
• Always ensure that the database is being actively monitored.
• Ensure that backups and snapshots are secure and restrict access to only required staff.
• Data in a Cloud SQL database is encrypted automatically. The configuration of Google-managed encryption versus customer-managed encryption is down to site-specific policy.
• If Terraform is being used to deploy CloudSQL, in the google_sql_database_instance resource:
  • In the settings stanza, set availability_type to REGIONAL to enable high availability
  • Under the settings stanza, in the backup_configuration subblock, set enabled and point_in_time_recovery_enabled to true, set an appropriate start_time for backups to run.

In addition to the general recommendations above, consider the following VMware-specific recommendations:

We understand that customers with private clouds are likely to have an established backup policy for databases already, possibly including a software partnership with a recognized backup vendor. In this case, for External Services mode deployments, we recommend you use these existing practices and tooling.

We make these additional recommendations for database backups:

• Refer to the PostgreSQL continuous archiving and PITR document to be able to replay write-ahead logs (WALs).
• Establish automated snapshots in line with the established RPO.

AWS has a significant number of business continuity configuration options for Redis.

If you use Terraform to deploy Terraform Enterprise, refer to AWS ElastiCache section of the Active/Active deployment guide for an example Redis configuration.

Your aws_elasticache_replication_group.tfe resource should look similar to the one found below. This configuration is for a Redis (cluster mode disabled) cluster of three nodes, one in each availability zone to confer n-2 zone redundancy.

resource "aws_elasticache_replication_group" "tfe" {
  ## ...

  num_cache_clusters = 3
  preferred_cache_cluster_azs = [var.availability_zones]
  multi_az_enabled = true
  automatic_failover_enabled = true
}

Azure Cache for Redis has built-in high availability.

If you use Terraform to deploy Terraform Enterprise, refer to the Azure Cache for Redis section of the Active/Active deployment guide.

Your azurerm_redis_cache.tfe resource should look similar to the one found below. This configuration is for a Redis (cluster mode disabled) cluster of three nodes, one in each availability zone to confer n-2 zone redundancy.

resource "azurerm_redis_cache" "tfe" {
  ## ...

  capacity  = 3
  family    = "P"
  sku_name  = "Premium"
}

The Standard Tier of the GCP Memorystore for Redis service provides high availability through replication and automatic failover capability. However, this tier provides only a second node, which provides an n-1 zone redundancy. The Standard Tier is currently the highest.

If you use Terraform to deploy Terraform Enterprise, refer to the GCP Memorystore for Redis section of the Active/Active deployment guide for an example configuration.

Active/Active deployment is unavailable for VMWare.

The following additional considerations provide an n-1 region redundancy on AWS. Since both cross-region S3 replication and Aurora read replicas can provide replicas in multiple Secondary regions, it is possible to offer greater than n-1 region redundancy if required.

• Use AWS S3 cross-region replication (CRR) on the object store; this means creating a pair of buckets, one in each region, configured to replicate from the Primary to the Secondary.
• Use S3 CRR on the buckets that store applicable database snapshots and on the bootstrap buckets that store the air-gapped installation media. Doing this locates critical data local to the ASG in the respective region.
• Use Aurora as the RDS DBaaS solution and enable cross-region read replicas.

The following additional considerations will provide an n-1 region redundancy on Azure:

• Use Azure geo-zone-redundant storage (GZRS) on the object store storage accounts. GZRS provides sixteen 9s of durability, maintains the recommended ZRS replication on objects in the Primary region, and replicates data to the Secondary region.
  • In the second region, use LRS, not ZRS. During a whole-region outage, it is part of BC planning and incident management to decide how long the outage will last and whether to persist with LRS in the Secondary region or to migrate to ZRS in that region.
  • GZRS is not supported in every region.
  • Geo-redundant data transfer happens asynchronously so there will always be a potential for data loss.
• Use read replicas in Azure Database for PostgreSQL to replicate the database to the Secondary region.
  • Not all Secondary regions can be used with all Primary regions. Refer to this matrix for details.
  • We recommend using Azure Paired Regions when region redundancy is required and you need the most up-to-date BC replicas.
  • If using Terraform to deploy the database, set geo_redundant_backup_enabled = true in the azurerm_postgresql_server resource.
• Using the Azure Cache for Redis Premium tier, you can enable geo-replication between two regions, where the writes in the Primary region are replicated to the Secondary region.
  • Read through and satisfy the geo-replication prerequisites.

The following additional considerations will provide an n-1 region redundancy on GCP:

• For cross-region object store GCS buckets, we recommend dual-region buckets since you need a separate Secondary region to replicate the object store to. We do not recommend multi-region buckets unless you intend to enable more than one Secondary region.
  • For geo-redundant storage, Cloud Storage data is redundant within at least one geographic place as soon as you upload it.
  • Be aware of the pairs of regions present in each dual-region bucket class. If you require cross-region deployment in Europe, due to EUR4 being europe-north1 and europe-west4 and the requirement to colocate the MIG in the same location as the GCS bucket, you must deploy Terraform Enterprise to one of these two regions to ensure a working instance with successful cross-region replication.
  • If the above region choices are not possible or greater than n-1 region redundancy is required, you need to either develop independent replication or use multi-region buckets. We recommend discussing this matter with a HashiCorp Customer Success Manager during the planning phase.
• CloudSQL supports cross-region replication.
  • If using Terraform to deploy the database, set location to the Secondary region in the backup_configuration subblock of the settings stanza of the google_sql_database_instance resource.
• There is no current support for geo-location of Memorystore for Redis replicas, so we recommend you repeat the above advice for each region.

Since this guide refers to multiple availability zones and maps these zones to separate VMware datacenters, multi-region deployments require connected datacenters in different countries or continents.

Repeat the recommendations in this guide for each region and use the strategic connections between regions to migrate Terraform Enterprise workloads during outages. The key concepts are to ensure:

• The S3-compatible object storage is replicated across region.
• The PostgreSQL database snapshots are made available in the BC region(s).
• You can switch global DNS/service addressing to the appropriate region in an outage.
• Full scenario testing conducted before production.

AWS has recommended backup/snapshot options to back up a Mounted Disk deployment.

Azure has recommended backup/snapshot options to back up a Mounted Disk deployment.

GCP has recommended backup/snapshot options to back up a Mounted Disk deployment.

For on-premise Mounted Disk mode deployments, refer to the Application Server VMware tab above for recommendations for server backup.

In addition:

• Make copies available in multiple datacenters to confer DC-redundancy.