Module creation - recommended pattern

Terraform modules are self-contained pieces of infrastructure-as-code that abstract the underlying complexity of infrastructure deployments. They speed adoption and lower the barrier of entry for Terraform end users who consume pre-built configuration. As a result, they should use coding best practices such as clear organization and the DRY ("Don't Repeat Yourself") principle wherever possible.

This guide discusses module architecture principles to help you write composable, sharable, reusable infrastructure modules. These architectural recommendations can be helpful to enterprises using any edition of Terraform. However, some features like the private registry are only present in HCP Terraform and Terraform Enterprise.

This guide complements and expands on the Terraform modules documentation.

In this guide, you will:

1. Read about the typical module creation workflow and fundamental principles for Terraform module creation.
2. Explore an example scenario that follows these principles.
3. Learn how to improve Terraform modules through collaboration.
4. Learn how to develop a module consumption workflow.

Module creation workflow

The first step in creating a new module is to find an early adopter team and gather their requirements.

Working with this early-adopter team allows you to hone the module functionality by ensuring it is sufficiently flexible through the use of input variables and outputs. In addition, they can onboard other teams with similar requirements with minimal code alterations. This eliminates code duplication and reduces time-to-market.

After you do this, there are two main steps to keep in mind.

1. Scope the requirements into appropriate modules.
2. Create the module minimum viable product (MVP).

Scope the requirements into appropriate modules

Deciding what infrastructure to include is the one of the most challenging aspects about creating a new Terraform module.

Modules should be opinionated and designed to do one thing well. If a module's function or purpose is hard to explain, the module is probably too complex. When initially scoping your module, aim for small and simple to start.

When building a module, consider three areas:

• Encapsulation: Group infrastructure that is always deployed together.
  Including more infrastructure in a module makes it easier for an end user to deploy that infrastructure but makes the module's purpose and requirements harder to understand.
• Privileges: Restrict modules to privilege boundaries.
  If infrastructure in the module is the responsibility of more than one group, using that module could accidentally violate segregation of duties. Only group resources within privilege boundaries to increase infrastructure segregation and secure your infrastructure.
• Volatility: Separate long-lived infrastructure from short-lived.
  For example, database infrastructure is relatively static while teams could deploy application servers multiple times a day. Managing database infrastructure in the same module as application servers exposes infrastructure that stores state to unnecessary churn and risk.

Create the module MVP

Modules, like any piece of code, are never complete. There will always be new module requirements and changes. Embrace this and aim for your first few module versions to meet the minimum viable product (MVP) standards. The following is a list of guidelines to keep in mind while designing your MVP.

• Always aim to deliver a module that works for at least 80% of use cases.
• Never code for edge cases in modules. An edge case is rare. A module should be a reusable block of code.
• Avoid conditional expressions in an MVP. An MVP should have a narrow scope and should not do multiple things.
• The module should only expose the most commonly modified arguments as variables. Initially, the module should only support variables that you are most likely to need.

Maximize outputs

Output as much information as possible from your module MVP even if you do not currently have a use for it. This will make your module more useful for end users who will often use multiple modules, using outputs from one module as inputs for the next.

Remember to document the outputs in the module's README.

Explore a scoping example

A team wants to provision their infrastructure, web tier application, and app tier application using Terraform.

They want their applications to be on a dedicated VPC and follow a traditional three-tier design. Their web application requires an autoscaling group. Their app tier application requires an autoscaling group, an S3 bucket and a database. The following architecture diagram depicts the desired outcome.

In this scenario, a team of Terraform producers, who write Terraform code from scratch, will build a collection of modules to provision the infrastructure and applications. The members of the team in charge of the application will consume these modules to provision the infrastructure they need.

After reviewing the consumer team's requirements, the producer team has broken up the application infrastructure into the following modules: Network, Web, App, Database, Routing, and Security.

After the Terraform producer team writes the modules, they should import them into the private registry, and advertise their availability to the respective team members for consumption. For example, the team members responsible for networking would use the network Terraform module to deploy and configure the underlying application network.

Network module

The network module is responsible for the infrastructure's networking. It contains the network access control lists (ACLs) and NAT gateway. It could also include the VPC, subnets, peering, and direct connect if they were necessary for the application.

This module contains these resources because they have high privilege and low volatility.

1. Only members of the application team that have permission to create or modify network resources should be able to use this module.
2. The resources in this module won't change very often. Grouping them in their own module protects them from unnecessary churn and risk.

The network module returns outputs that other workspaces and modules can use. If VPC creation is multi-faceted, you could eventually split this module into different modules with different functions.

Application modules

There are two application modules in this scenario — one for the web tier, the other for the app tier.

After the Terraform producers write both modules, they should distribute it to the respective team members to deploy their application. As the application team becomes more familiar with Terraform code, they can suggest infrastructure enhancements or changes via pull request in conjunction with releases of their application code.

Web module

The web module creates and manages the infrastructure needed to run the web application. It contains the load balancer and autoscaling group. It could also include EC2 instances, S3 buckets, security groups inside the application, and logging. This module takes an AMI ID of a prebuilt (via Packer) AMI that contains the latest web application code release.

This module contains these resources because they are highly encapsulated and have highly volatility.

1. The resources in this module are tightly scoped and associated specifically with the web application (for example, this module provisions an AMI containing the latest web application code release). As a result, they are grouped together into a single module so web application team members can easily deploy them.
2. The resources in this module change often (with each code release). By separating them into their own module, you decrease unnecessary churn and risk for other modules.

App module

The app module creates and manages the infrastructure needed to run the app tier application. It contains the load balancer, autoscaling group and S3 buckets. It could also include EC2 instances, security groups inside the application, and logging. This module takes an AMI ID of a prebuilt (via Packer) AMI that contains the latest app tier application code release.

This module contains these resources because they are highly encapsulated and have highly volatility.

1. The resources in this module are tightly scoped and associated specifically with the app tier application. As a result, they should be grouped together into a single module so app tier application team members can easily deploy them.
2. The resources in this module change often (with each code release). By separating them into a their own module, you decrease unnecessary churn and risk for other modules.

Database module

The database module creates and manages the infrastructure needed to run the database(s). It contains the RDS instance used by the application. It could also include all associated storage, all backup data, and logging.

This module contains these resources because they are highly privileged and have low volatility.

1. Only members of the application team that have permission to create or modify database resources should be able to use this module.
2. The team is unlikely to change the resources in this module often, so making them a separate module decreases unnecessary churn and risk.

Routing module

The routing module creates and manages the infrastructure needed for any network routing. It contains hosted zones, Route 53 and route tables. It could also include private hosted zones.

This module contains these resources because they are highly privileged and have low volatility.

1. Only members of the application team that have permission to create or modify routing resources should be able to use this module.
2. The team is unlikely to change the resources in this module often, so making them a separate module decreases unnecessary churn and risk.

Security module

The security module creates and manages the infrastructure needed for any security. It contains IAM resources. It could also include security groups and MFA.

This module contains these resources because they are highly privileged and have low volatility.

1. Only members of the application team that have permission to create or modify IAM/security resources should be able to use this module.
2. The team is unlikely to change the resources in this module often, so making them a separate module decreases unnecessary churn and risk.

Module creation tips

In addition to scoping, you should keep the following in mind while creating modules that configuration authors can compose and reference in root modules. HCP Terraform Plus Edition lets you publish modules that users deploy without writing any Terraform configuration. Follow the Create and Use No-Code Modules tutorial to create a no-code ready module that follows additional no-code ready module architecture best practices.

Nesting modules

A nested module is a reference to invoke another module from the current module (including from the root module). Nested modules can be located externally and are referred to as "child modules", or embedded inside the current workspace and are referred to as "submodules". Being able to nest modules is powerful feature; however, you should exercise caution to limit introducing errors.

For all types of nested modules, consider the following:

• Nesting modules can speed development, but can lead to unclear and unexpected outcomes. Clearly document the input variables, module behavior and outputs.
• Generally, do not nest primary modules more than two deep. Common and simple utility modules, like ones that define tags, are the exception to this rule.
• Nested modules should contain the necessary input variables to create specific resource configurations and outputs.
• Consistent naming conventions for input variables and outputs are highly recommended so modules can easily share and map input/output parameters.
• Nested modules can lead to code redundancy. Variable definitions and outputs need to be defined in nested and parent modules.

Nesting external modules

Nested external modules (module calls to child modules that contain module calls to other modules) can be useful when you need common modules that provide standardized resources and are reused by multiple application stacks, applications, and teams. External modules are often centrally managed and versioned so that new releases are validated before consumers can use them. When you need or desire to have externally located child modules, here are some considerations:

• The external module must be maintained independently and made available to any module needing to invoke it. Using the Terraform registry is good way to ensure this.
• With the Terraform registry requirements, a nested module will have its own version controlled repository, which versions separately from its parent.
• A change to the nested module can affect the parent module with no changes to the parent's calling code or version, thereby breaking the calling code's trust.
• Document how parent modules use externally sourced modules so the behavior and invocation are easily understood.
• Revisions to external modules should be backwards compatible. If backwards-compatibility is impossible, changes that need to be made to any parent modules should be clearly documented and distributed to all module consumers to avoid surprises.

Nesting submodules

Embedding one or more submodule inside your current code base enables you to cleanly separate logical components of the primary module or to create a reusable code block that can be invoked multiple times during the execution of the calling module. In the following example, ec2-instance is an embedded module that the root main.tf can reference.

root-module-directory
├── README.md
├── main.tf
└── ec2-instances
    └── main.tf

If you need or desire to use submodules, here are some considerations:

• Adding the module to your "root" module means the submodule is versioned along with the called module.
• Any changes affecting the compatibility of the two modules is discovered quickly because they must be released and tested together.
• The submodule cannot be invoked by another module outside of the source tree so there may be increased code duplication. For example, if the embedded ec2-instance module is meant to create a standardized compute instance used in multiple places, it could not be shared in this format.

Label and document module elements

Create and follow a naming convention for your module elements to make them easier to understand and work with. This can lead to higher module adoption and contribution. The following is a list of suggestions to keep your module elements more consistent.

• Use a module naming convention consistent and understandable by humans. For example:

  terraform	cloud provider	function	full name
  terraform	aws	consul cluster	terraform-aws-consul_cluster
  terraform	aws	security module	terraform-aws-security
  terraform	azure	database	terraform-azure-database

• Use a variable naming convention understandable by humans. Modules are code that will be written once and used many times, so name everything fully to improve readability and document your code as you write it.

• Document all modules. Make sure the documentation includes:

  • Required inputs: These variables should be a deliberate choice. The module will fail if they are not defined. Only set defaults for variables that should have them. For example var.vpc_id should never have a default because the value would be different every time you use the module.
  • Optional inputs: These should have a sensible default that will be acceptable in most use cases but may need adjusting. Advertise the default value. For example var.elb_idle_timeout will have a sensible default, but there is a chance that someone may need to modify it.
  • Outputs: List all the outputs of your module and wrap the important and informational ones in user-friendly output templates.

Define and use a consistent module structure

While module structure is a matter of taste, you should document the module structure and be consistent across all your modules. To keep your module structure consistent:

• Define a list of .tf files that must be in the module and what they should contain.
• Define a .gitignore (or similar) for modules.
• Create a standard way of providing examples of variables (such as a terraform.tfvars.example).
• Use a consistent directory structure with a defined set of directories, even if they may be empty.
• All modules directories should have a README detailing the purpose and use of the files within it.

Collaborate on modules

As your team develops modules, streamline your collaboration.

1. Create a roadmap for each module.
2. Gather requirements from your users and prioritize them by popularity.
   • The most common reason for not using a module is that "It does not do what I want." Gather these requirements and either add them to the roadmap or advise the user's workflow.
   • Check each requirement to determine if the use case it references is correct.
   • Publish and maintain the requirement list. Share it and involve users in the list curation process.
   • Don't prioritize edge-cases.
3. Document every decision.
4. Adopt open-source community principles within the company. Some users want to use the modules as efficiently as possible while others want to help create the modules.
   • Create a community,
   • Have a clearly defined and published contribution guide.
   • Eventually, you could allow trusted community members to own some modules.

Use source control to track modules

A Terraform module should adhere to all good code practices.

• Place modules in source control to manage release versions, collaboration, and audit trail of changes.
• Tag and document all releases to the main branch (use CHANGELOG and README as a minimum).
• Code review all changes to the main branch.
• Encourage your module users to reference by tag.
• Assign each module an owner.
• Use only one module per repository.
  • This is vital for modules to be idempotent and to function as libraries.
  • You should tag or version modules. Tagged or versioned module should be immutable.
  • Tagged releases are a requirement for the private Terraform registry.

Develop a module consumption workflow

Define and publicize a repeatable workflow that consuming teams should follow to use the modules. This workflow, like the modules themselves, should take into account user requirements.

Terraform Enterprise provides tools, like the private Terraform registry and configuration designer, that provide structure for module collaboration and consumption. These tools makes modules easier to use, and clarifies how teams use modules.

Make modules easy to use

• Private Terraform registry: The private Terraform registry provides a searchable, filterable way to manage your modules and allows consumers to browse and search for modules appropriate to their use case.
• UI: The Terraform Enterprise UI is less intimidating for Terraform novices and has a lower barrier to entry.
• Configuration designer: The configuration designer functions like interactive documentation for your private modules, or very advanced autocompletion. It results in the same Terraform code you would write in a text editor, and saves time by automatically discovering variables and searching module and workspace outputs for reusable values.

Clarify how teams use modules

• Devolved security: If each module is versioned in its own repository, repository RBAC can be used to manage who has write access, allowing relevant teams to manage related infrastructure (such as the network team owning write access on network modules).
• Fostering a code community: The best practice for module development, given the above recommendations, is to allow pull requests on all module repositories for modules stored in the private module repository. This fosters a code community within the organization, keeps module content relevant and maximally flexible and helps maintain the registry's effectiveness in the long term.
• Policy enforcement: With Sentinel, you can assign policy criteria to all Terraform plans before execution. This allows for enforcement such that only modules from the private Terraform registry can be used; this provides greater control over collaboration and adoption of company policy and/or regulatory requirements.

Next steps

In this guide, you learned about recommended Enterprise patterns for creating, collaborating and consuming Terraform modules.

• To learn more about Terraform modules, refer to the Terraform Modules documentation
• For step-by-step tutorials on using and creating Terraform modules, refer to the Reuse Configuration with Modules tutorials.
• Help developers provision self-service infrastructure by following the Create and Use No-Code Modules tutorial.