[HOWTO] Automate Terraform execution in Azure DevOps YAML pipeline

This is a topic to which I have dedicated and continue to dedicate many hours and which I have wanted to blog about for a long time.

When it comes to infrastructure deployments with Infrastructure as Code (IaC) – in this case Terraform – using Azure DevOps YAML pipelines I prefer the following setup assuming that there are three stages (DEV, TEST and PROD) and that the target is the Azure cloud.

There are a few important things to mention.

• To follow the “separation of concerns” principle, I always create separate service connections per target environment
• On the left side you can see two Azure DevOps environments per target environment (i.e. DEV and DEV IaC). This design allows the definition of approvals on the IaC suffixed environments without having to approve application deployments to i.e. DEV. Usually I set up approvals (visualized as checkmarks) for PROD deployments only. However to be able to review Terraform plan before applying it, approvals are a great option to achieve that.
• In the security settings of the Azure DevOps environment, assign a group/team (i.e. Project Administrators) to the Administrator role
• Assign a group/team as approvers instead of single user(s)

Let’s now have a look at the pipeline definition.

deploy\pipelines\ExampleSolution-iac.yml

This is the root file of the YAML pipeline to automate Terraform execution. The subsequent files make use of the Azure DevOps extension Azure Pipelines Terraform Tasks which therefore needs to be installed to run the pipeline successfully.

name: ExampleSolution IaC Pipeline
trigger:
  branches:
    include:
      - main
      - dev
  paths:
    include:
      - deploy/iac
variables:
  isMainBranch: $[eq(variables['Build.SourceBranch'], 'refs/heads/main')]
  isDevBranch: $[eq(variables['Build.SourceBranch'], 'refs/heads/dev')]
  isTargetMainBranch: $[eq(variables['system.pullRequest.targetBranch'], 'refs/heads/main')]
  isReleasePR: $[startsWith(variables['system.pullRequest.sourceBranch'], 'refs/heads/release')]
  isHotfixPR: $[startsWith(variables['system.pullRequest.sourceBranch'], 'refs/heads/hotfix')]
pool:
  vmImage: windows-latest
stages:
  - stage: PlanIaCDev
    displayName: Plan IaC for Development environment
    condition: eq(variables.isDevBranch, true)
    jobs:
      - template: jobs-iac-plan.yml
        parameters:
          azureSubscription: DEV_SERVICE_CONNECTION_NAME
          backendName: dev
          environmentCode: d1
  - stage: DeployIaCDev
    displayName: Deploy IaC to Development environment
    dependsOn: PlanIaCDev
    condition: and(succeeded(), eq(variables.isDevBranch, true))
    jobs:
      - template: jobs-iac-deploy.yml
        parameters:
          azureSubscription: DEV_SERVICE_CONNECTION_NAME
          backendName: dev
          environmentCode: d1
          environment: Development-IaC

  - stage: PlanIaCTest
    displayName: Plan IaC for Test environment
    condition: and(eq(variables.isTargetMainBranch, true), or(eq(variables.isReleasePR, true), eq(variables.isHotfixPR, true)))
    jobs:
      - template: jobs-iac-plan.yml
        parameters:
          azureSubscription: TEST_SERVICE_CONNECTION_NAME
          backendName: test
          environmentCode: t1
  - stage: DeployIaCTest
    displayName: Deploy IaC to Test environment
    dependsOn: PlanIaCTest
    condition: and(succeeded(), and(eq(variables.isTargetMainBranch, true), or(eq(variables.isReleasePR, true), eq(variables.isHotfixPR, true))))
    jobs:
      - template: jobs-iac-deploy.yml
        parameters:
          azureSubscription: TEST_SERVICE_CONNECTION_NAME
          backendName: test
          environmentCode: t1
          environment: Test-IaC

  - stage: PlanIaCProd
    displayName: Plan IaC for Production environment
    condition: eq(variables.isMainBranch, true)
    jobs:
      - template: jobs-iac-plan.yml
        parameters:
          azureSubscription: PROD_SERVICE_CONNECTION_NAME
          backendName: prod
          environmentCode: p1
  - stage: DeployIaCProd
    displayName: Deploy IaC to Production environment
    dependsOn: PlanIaCProd
    condition: eq(variables.isMainBranch, true)
    jobs:
      - template: jobs-iac-deploy.yml
        parameters:
          azureSubscription: PROD_SERVICE_CONNECTION_NAME
          backendName: prod
          environmentCode: p1
          environment: Production-IaC

The YAML pipeline consists of two stages per target environment. In the PlanIaC[ENV] stage, terraform plan is executed and in the DeployIaC[ENV] stage terraform apply is executed. To be able to review the plan generated by the plan stage, the deploy stage refers to the corresponding Azure DevOps IaC suffixed environment where approvals can be defined on.

deploy\pipelines\jobs-iac-plan.yml

In this part of the pipeline terraform commands related to the plan stage are executed. Besides installation, terraform init and terraform plan the correct workspace gets selected and the terraform code gets validated.

parameters:
  - name: "azureSubscription"
    type: string
    default: ""
  - name: "environmentCode"
    type: string
    default: ""
  - name: "backendName"
    type: string
    default: ""

jobs:
  - job: IaCPlan
    displayName: Plan infrastructure as code
    steps:
      - task: TerraformInstaller@0
        displayName: Install Terraform
        inputs:
          terraformVersion: 1.4.2
      - task: TerraformCLI@0
        displayName: Terraform init
        inputs:
          command: init
          backendType: azurerm
          backendServiceArm: ${{parameters.azureSubscription}}
          commandOptions: -reconfigure --backend-config=./backend/${{parameters.backendName}}.backend.tfvars
          workingDirectory: "$(System.DefaultWorkingDirectory)/deploy/iac"
      - task: TerraformCLI@0
        displayName: Terraform select workspace
        inputs:
          command: workspace
          workingDirectory: "$(System.DefaultWorkingDirectory)/deploy/iac"
          workspaceSubCommand: select
          workspaceName: ${{parameters.environmentCode}}
      - task: TerraformCLI@0
        displayName: Terraform validate
        inputs:
          command: validate
          workingDirectory: "$(System.DefaultWorkingDirectory)/deploy/iac"
      - task: TerraformCLI@0
        displayName: Terraform plan
        inputs:
          command: plan
          commandOptions: -var-file="./vars/${{parameters.backendName}}.${{parameters.environmentCode}}.tfvars"
          environmentServiceName: ${{parameters.azureSubscription}}
          workingDirectory: "$(System.DefaultWorkingDirectory)/deploy/iac"
          publishPlanResults: "ExampleSolution-iac-plan"

The Azure DevOps extension Azure Pipelines Terraform Tasks supports Terraform Plan View – a feature that renders terraform plans within the pipeline run summary (see section Terraform Plan View here). To make use of this feature, publishPlanResults is specified.

deploy\pipelines\jobs-iac-deploy.yml

Last but not least, the part of the pipeline that applies the Terraform code to the corresponding target environment.

parameters:
  - name: "azureSubscription"
    type: string
    default: ""
  - name: "environmentCode"
    type: string
    default: ""
  - name: "environment"
    type: string
    default: ""
  - name: "backendName"
    type: string
    default: ""

jobs:
  - deployment: IaCApply
    displayName: Deploy infrastructure as code
    environment: ${{parameters.environment}}
    strategy:
      runOnce:
        deploy:
          steps:
            - checkout: self
            - task: TerraformInstaller@0
              displayName: "Install Terraform"
              inputs:
                terraformVersion: 1.4.2
            - task: TerraformCLI@0
              displayName: Terraform init
              inputs:
                command: init
                backendType: azurerm
                backendServiceArm: ${{parameters.azureSubscription}}
                commandOptions: -reconfigure --backend-config=./backend/${{parameters.backendName}}.backend.tfvars
                workingDirectory: "$(System.DefaultWorkingDirectory)/deploy/iac"
            - task: TerraformCLI@0
              displayName: Terraform select workspace
              inputs:
                command: workspace
                workingDirectory: "$(System.DefaultWorkingDirectory)/deploy/iac"
                workspaceSubCommand: select
                workspaceName: ${{parameters.environmentCode}}
            - task: TerraformCLI@0
              displayName: Terraform apply
              inputs:
                command: apply
                commandOptions: -auto-approve -var-file="./vars/${{parameters.backendName}}.${{parameters.environmentCode}}.tfvars"
                environmentServiceName: ${{parameters.azureSubscription}}
                workingDirectory: "$(System.DefaultWorkingDirectory)/deploy/iac"

I thought several times about passing the terraform plan (file) created in the first stage to the terraform apply command, but have now consciously decided against it. The reason is pretty simple. If the terraform plan created in the first stage gets applied, there is no possibility to fix something by running terraform locally and then rerun the failed jobs of the pipeline.

Azure Pipelines Terraform Tasks

Finally, a few words about the extension Azure Pipelines Terraform Tasks. A few weeks ago, I figured out that the extension had changed owners.

As the extension published by Charles Zipp got unpublished from the marketplace, it’s a good idea to uninstall it and afterwards install the new one published by Jason Johnson which relies on the same source code repository. Having both extensions installed will lead to failing pipelines except you always specify fully qualified task name.

There should not be any side-effects – especially if you always used TerraformCLI@0. To be able to work with service connections that use Workload Identity federation authentication mode, version 1 of the tasks is required and therefore a switch to the extension published by Jason Johnson.