DRY using Terraform

DRY?

In the world of Infrastructure as Code (IaC), the principle of DRY (Don’t Repeat Yourself) is crucial for maintaining efficient and manageable codebases. When it comes to Terraform, one of the most popular IaC tools, applying DRY principles can significantly improve your infrastructure management in a lot of ways. But what exactly does DRY mean in the context of Terraform, and why is it so important for you and most importantly, for your current job and company?

DRY in Terraform refers to the practice of minimizing code duplication and promoting reusability across your infrastructure configurations. This approach not only reduces the potential for errors but also enhances maintainability and scalability of your infrastructure code. As your infrastructure grows in complexity, adhering to DRY principles becomes increasingly vital for scaling in today’s cloud environments.

In this blog post, we’ll not explore the concept of DRY in Terraform as there are many other blogs that talk about this. What we will do is explore two tools that allows us to apply DRY principles in our beloved Terraform file; Terragrunt and Atmos.

Terragrunt

Terragrunt is a thin wrapper around Terraform that helps manage and work with complex infrastructure-as-code (IaC) configurations by solving some common challenges developers face when working with Terraform. It’s designed to make Terraform easier to use by providing additional features like code reuse, dependency management, and automated remote state management.

Terraform, while powerful, can become difficult to manage in larger, more complex infrastructure projects due to several challenges like:

1. Duplicated Code: Terraform configurations often require similar code across different environments (e.g., staging, production). Managing these environments can lead to a lot of copy-pasting of the same code with minor modifications, which is hard to maintain.

2. Remote State Management: Terraform requires state to track resources, and in larger teams, managing Terraform state files across multiple environments can be tricky.

3. Dependency Management: Terraform doesn’t handle resource dependencies across different modules/environments well out-of-the-box. For example, if a networking module needs to be deployed before an application module, Terraform doesn’t provide a built-in solution for cross-module dependencies.

4. Managing Multiple Environments/Regions: Deploying infrastructure to multiple environments (e.g., dev, staging, production) and regions can involve many configurations and make infrastructure drift hard to control.

Benefits of Using Terragrunt

1. DRY / Reusable Configurations

Terragrunt allows you to eliminate duplicate code across environments by enabling modular and reusable Terraform configurations. It lets you use a single Terraform module across multiple environments by passing in different configurations.

• Without Terragrunt: You often end up copying and pasting the same Terraform code in different files for different environments, which can be hard to maintain.

• With Terragrunt: You can create a base Terraform module and manage different environments (e.g., dev, prod) by simply configuring input variables in Terragrunt’s terragrunt.hcl files. This reduces duplication and increases maintainability.

Example:

# terragrunt.hcl for "production"
terraform {
  source = "../modules/my-app"
}

inputs = {
  environment = "prod"
  instance_type = "m5.large"
}

# terragrunt.hcl for "development"
terraform {
  source = "../modules/my-app"
}

inputs = {
  environment = "dev"
  instance_type = "t3.micro"
}

2. Automated Remote State Management

Managing Terraform state files can be tricky, especially when using remote state backends like S3 or GCS. Terragrunt simplifies this by automatically configuring remote state for you.

• Without Terragrunt: You manually configure and update remote state settings in each Terraform configuration.

• With Terragrunt: You can define remote state settings once in Terragrunt and reuse it across all environments/modules. Terragrunt will automatically handle the configuration and lock state files to prevent conflicts.

Example of remote state configuration in terragrunt.hcl:

remote_state {
  backend = "s3"
  config = {
    bucket         = "my-terraform-state"
    key            = "envs/prod/terraform.tfstate"
    region         = "us-west-2"
    encrypt        = true
    dynamodb_table = "terraform-lock"
  }
}

3. Dependency Management

Terragrunt allows you to define dependencies between different Terraform modules. For example, if you want to ensure that your network infrastructure is deployed before your application infrastructure, Terragrunt provides a built-in way to handle these dependencies.

• Without Terragrunt: Terraform does not handle cross-module or cross-environment dependencies. You need to manually ensure that dependencies are applied in the correct order.

• With Terragrunt: You can explicitly define dependencies between modules using Terragrunt, and it will manage the execution order.

Example:

dependency "vpc" {
  config_path = "../vpc"
}

inputs = {
  vpc_id = dependency.vpc.outputs.vpc_id
}

4. Managing Multiple Environments/Regions Easily

Terragrunt simplifies managing multiple environments and regions by allowing you to keep the configurations for environments and regions in one place and reference shared modules. This makes managing environments like dev, staging, and prod much easier, and ensures consistency.

• Without Terragrunt: You would have separate Terraform configurations for each environment, which can lead to code duplication and potential drift between environments.

• With Terragrunt: You can maintain a clean, consistent structure for environments, with separate terragrunt.hcl files for each environment that specify how to configure the underlying shared modules.

For example, you can define an infrastructure layout like this:

bash
Copy code
/live
  └── /dev
      └── terragrunt.hcl
  └── /prod
      └── terragrunt.hcl

5. Built-In Workflows for apply-all, destroy-all

Terragrunt provides higher-level commands like apply-all and destroy-all, which make it easy to apply or destroy Terraform configurations across multiple modules and environments in one go.

• Without Terragrunt: You need to manually run terraform apply or terraform destroy for each module or environment.

• With Terragrunt: You can use a single command (terragrunt apply-all or terragrunt destroy-all) to apply or destroy all configurations across multiple modules and environments.

6. Easier Multi-Account or Multi-Region Setups

When managing infrastructure across different AWS accounts or regions, Terragrunt makes it easier to manage credentials and configuration by allowing you to reuse a single configuration file with different variables (e.g., different AWS account credentials).

• Without Terragrunt: You may have to manually configure provider blocks for each account or region.

• With Terragrunt: You can configure a base terragrunt.hcl that handles different accounts or regions by passing in different variables.

Use Cases for Terragrunt

• Large teams with multiple environments (dev, staging, prod): It simplifies managing the differences between environments without duplicating code.

• Infrastructure spread across multiple regions/accounts: Helps reduce complexity by handling remote state and dependencies across environments and accounts.

• Cross-module dependencies: Makes managing the order of module execution much simpler when some modules depend on others.

• Modular, DRY infrastructure: Ideal for teams that want to reuse Terraform code across environments.

Atmos

Atmos is a tool developed by CloudPosse to enhance the workflow for managing infrastructure with Terraform and Helm. It is designed to simplify and standardize the configuration of complex environments by introducing a consistent, modular, and scalable approach.

Benefits of using Atmos:

1. Hierarchical Configuration Management: Atmos allows you to define infrastructure and environment configurations hierarchically. You can share configurations between environments (like dev, staging, and prod), while allowing specific overrides.

   Example:

# catalog/terraform/stacks/dev.yaml global: region: us-west-1 account_id: "123456789012"  components: terraform: vpc: vars: cidr_block: "10.0.0.0/16"

# catalog/terraform/stacks/prod.yaml global: region: us-east-1 account_id: "987654321098"  components: terraform: vpc: vars: cidr_block: "172.16.0.0/16"

1. In this example, the vpc configuration for prod overrides the region and CIDR block while sharing common components defined globally.

2. Modular Architecture: Atmos promotes reusable modules across environments, reducing the need to redefine infrastructure repeatedly.

   Example: You can define a shared vpc module used across all environments:

# modules/vpc/main.tf module "vpc" { source = "terraform-aws-modules/vpc/aws" name = var.vpc_name cidr = var.cidr_block }

1. Then reference this module in each environment’s configuration:

# stacks/dev.yaml components: terraform: vpc: vars: vpc_name: "dev-vpc" cidr_block: "10.0.0.0/16"

1. Layering System: Atmos layers your infrastructure components (networking, apps, security) across environments, simplifying management.

   Example:

# stacks/dev.yaml components: terraform: vpc: vars: vpc_name: "dev-vpc" app: vars: app_name: "my-app-dev"

# stacks/prod.yaml components: terraform: vpc: vars: vpc_name: "prod-vpc" app: vars: app_name: "my-app-prod"

1. Here, both environments share the same module but apply different variables per layer (vpc, app).

2. YAML-Based Configuration: Atmos uses YAML for defining infrastructure stacks, making it easier to read and maintain.

   Example:

# stacks/dev.yaml global: region: us-west-1 components: terraform: eks: vars: cluster_name: "dev-cluster" node_count: 3

1. Integrated with Terraform and Helm: Atmos integrates with Terraform for infrastructure provisioning and Helm for Kubernetes, allowing you to handle both through a unified CLI.

   Example: Run Terraform plan for the eks component:

    bash Copy code atmos terraform plan eks

   Run Helm upgrade for a Kubernetes service:

    bash Copy code atmos helm upgrade service-name

2. Support for Multi-Environment Workflows: Atmos helps manage multiple environments and regions without repeating code by automatically selecting configurations based on context.

   Example: To deploy infrastructure to the prod environment:

    bash Copy code atmos terraform apply eks --stack=prod

3. Stack Configuration: Atmos introduces “stacks” to define infrastructure across environments in a repeatable manner.

   Example:

# catalog/terraform/stacks/prod.yaml global: region: us-east-1 account_id: "987654321098"  components: terraform: vpc: vars: cidr_block: "172.16.0.0/16" eks: vars: cluster_name: "prod-cluster"

1. This stack defines vpc and eks configurations for the prod environment.

2. Terraform Backend and Workspaces Management: Atmos simplifies managing Terraform backends and workspaces across environments. The backend configuration is handled automatically based on the stack.

   Example:

# stacks/prod.yaml terraform: backend: s3: bucket: "my-tf-state-prod" key: "terraform/state" region: "us-east-1"

1. CLI for Orchestration: Atmos provides a powerful CLI to run Terraform and Helm commands while automatically applying the right configurations.

   Example: To deploy to prod:

    bash Copy code atmos terraform apply eks --stack=prod

   To destroy a resource:

    bash Copy code atmos terraform destroy vpc --stack=dev

2. Standardization Across Teams: Atmos enforces a consistent structure for Terraform and Helm configurations, reducing the risk of errors and making it easier to onboard new team members.

Example Structure:

bash
Copy code
catalog/
├── terraform/
│   ├── modules/
│   └── stacks/
└── helmfile/
    ├── charts/
    └── releases/

Conclusion

Both Terragrunt and Atmos offer valuable enhancements to infrastructure management with Terraform. Terragrunt focuses on code reuse, dependency management, and remote state management, making it particularly useful for complex, multi-environment, and multi-region projects. It helps enforce consistency while reducing duplication in Terraform configurations.

Atmos, on the other hand, brings additional value by integrating Terraform and Helm management. It simplifies configuration management through its layering and stack systems, promotes modular and reusable setups, and provides a standardized workflow. This approach is especially beneficial for teams managing complex multi-environment infrastructures.

Both tools aim to help teams scale infrastructure management efficiently and consistently across multiple environments and regions. Whether you choose Terragrunt or Atmos depends on your specific needs, but both can significantly improve your infrastructure management workflows, especially in large-scale, complex environments.

References

• https://cloudposse.com/atmos/

• https://terragrunt.gruntwork.io