Resources - Retries and Customizable Timeouts

The reality of cloud infrastructure is that it typically takes time to perform operations such as booting operating systems, discovering services, and replicating state across network edges. As the provider developer you should take known delays in resource APIs into account in the CRUD functions of the resource. Terraform supports configurable timeouts to assist in these situations.

package example

import (
    "fmt"

    "github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema"
)

func resourceExampleInstance() *schema.Resource {
    return &schema.Resource{
        CreateContext: resourceExampleInstanceCreate,
        ReadContext:   resourceExampleInstanceRead,
        UpdateContext: resourceExampleInstanceUpdate,
        DeleteContext: resourceExampleInstanceDelete,

        Schema: map[string]*schema.Schema{
            "name": {
                Type:     schema.TypeString,
                Required: true,
            },
        },
        Timeouts: &schema.ResourceTimeout{
            Create: schema.DefaultTimeout(45 * time.Minute),
        },
    }
}

In the above example we see the usage of the timeouts in the schema being configured for what is deemed the appropriate amount of time for the Create function. Read, Update, and Delete are also configurable as well as a Default. These configured timeouts can be fetched in the CRUD function logic using the (*schema.ResourceData).Timeout() method, such as d.Timeout(schema.TimeoutCreate). Practitioners can override these timeout values with resource timeouts configuration, such as:

resource "example_thing" "example" {
  # ...

  timeouts {
    create = "60m"
  }
}

Default Timeouts and Deadline Exceeded Errors

The SDK imposes the following default timeout behaviors for CRUD functions:

The *schema/Resource.Timeouts field can customize the default timeout on CRUD functions with default timeouts.

If a CRUD function timeout is exceeded, the SDK will automatically return a context.DeadlineExceeded error. To practitioners, this is shown in the Terraform CLI output as a context: deadline exceeded error. Since the context timeout and associated error handling occur outside CRUD logic in the SDK, it is not possible to capture or change this error behavior. If it is unclear how long CRUD operations may take, it is recommended to either increase the default timeout using the Timeouts field, or switch to using the WithoutTimeout CRUD functions.

Retry

The retry helper takes a timeout and a retry function.

• The timeout value specifies the maximum time Terraform will invoke the retry function. You can retrieve the timeout from the *schema.ResourceData struct by passing the timeout key (schema.TimeoutCreate) to the Timeout method.
• The retry function, which is available by importing github.com/hashicorp/terraform-plugin-sdk/v2/helper/retry, returns either a retry.NonRetryableError for unexpected errors/states or a retry.RetryableError for expected errors/states. If the function returns a retry.RetryableError, it will re-run the function.

In the context of a CREATE function, once the backend responds with the desired state, invoke the READ function. If READ errors, return that error wrapped with retry.NonRetryableError. Otherwise, return nil (no error) from the retry function.

func resourceExampleInstanceCreate(d *schema.ResourceData, meta any) error {
    name := d.Get("name").(string)
    client := meta.(*ExampleClient)
    resp, err := client.CreateInstance(name)

    if err != nil {
        return fmt.Errorf("Error creating instance: %s", err)
    }

    return retry.Retry(d.Timeout(schema.TimeoutCreate) - time.Minute, func() *retry.RetryError {
        resp, err := client.DescribeInstance(name)

        if err != nil {
            return retry.NonRetryableError(fmt.Errorf("Error describing instance: %s", err))
        }

        if resp.Status != "CREATED" {
            return retry.RetryableError(fmt.Errorf("Expected instance to be created but was in state %s", resp.Status))
        }

        err = resourceExampleInstanceRead(d, meta)
        if err != nil {
            return retry.NonRetryableError(err)
        } else {
            return nil
        }
    })
}

StateChangeConf

retry.Retry is useful for simple scenarios, particularly when the API response is either success or failure, but sometimes handling an APIs latency or eventual consistency requires more fine tuning. retry.Retry is in fact a wrapper for a another helper: retry.StateChangeConf.

Use retry.StateChangeConf when your resource has multiple states to progress though, you require fine grained control of retry and delay timing, or you want to ensure a minimum number of occurrences of a target state is reached (this is very common when dealing with eventually consistent APIs, where a response can reply back with an old state between calls before becoming consistent).

func resourceExampleInstanceCreate(d *schema.ResourceData, meta any) error {
    name := d.Get("name").(string)
    client := meta.(*ExampleClient)
    resp, err := client.CreateInstance(name)

    createStateConf := &retry.StateChangeConf{
        Pending: []string{
            client.ExampleInstanceStateRequesting,
            client.ExampleInstanceStatePending,
            client.ExampleInstanceStateCreating,
            client.ExampleInstanceStateVerifying,
        },
        Target: []string{
            client.ExampleInstanceStateCreateComplete,
        },
        Refresh: func() (any, string, error) {
            resp, err := client.DescribeInstance(name)
            if err != nil {
                0, "", err
            }
            return resp, resp.Status, nil
        },
        Timeout:    d.Timeout(schema.TimeoutCreate) - time.Minute,
        Delay:      10 * time.Second,
        MinTimeout: 5 * time.Second,
        ContinuousTargetOccurence: 5,
    }
    _, err = createStateConf.WaitForState()
    if err != nil {
        return fmt.Errorf("Error waiting for example instance (%s) to be created: %s", d.Id(), err)
    }

    return resourceExampleInstanceRead(d, meta)
}