[HOWTO] Rotate Azure Key Vault secrets used by an ASP.NET Core Web API with Terraform on every deployment

Today I’m blogging about another topic that has spent quite some time on my TODO list (more than 1.5 years :S). Additionally, it took me some time to prepare the GitHub sample repository for this blog post, as there were some issues with the GitHub action workflow. Now it finally works, so let’s get started.

The WHAT

In this post I’ll explain and demonstrate how secrets that are stored in an Azure Key Vault can be rotated on every deployment of the application that uses them.

Important note: this approach is scoped to secrets that can be controlled/managed by Infrastructure as Code (IaC) – in this specific case Terraform. For example client secrets of app registrations or connection strings of other Azure resources. Regarding connection strings; try to always use managed identity authentication. Secrets that can not be controlled/managed by IaC like for example API keys to access external services are out of scope.

Assumption: application gets deployed regularly – at least weekly or monthly. The more often, the better.

The WHY

Why secret rotation at all? It’s not just for fun, it effectively minimizes the consequences of secrets leak as they are only valid until the next deployment takes place. And if a leak is noticed (i.e. someone commits a secret by mistake), a redeployment of the application will mitigate the risk.

The HOW

I’ll demonstrate and explain a possible approach using a sample ASP.NET Core Web API (.NET 9) which is deployed on an Azure App Service. As IaC tool Terraform 1.9.8 is used.

Note: Ideally, there is a separate Azure Key Vault for the ASP.NET Core Web API which only contains the necessary secrets (good practice from a security perspective as this ensures the need to know principle).

To connect the ASP.NET Core Web API with the Azure Key Vault, the NuGet package Azure.Extensions.AspNetCore.Configuration.Secrets has to be installed. Next, to enrich the (in-memory) configuration of the application during startup AddAzureKeyVault has to be called.

using Azure.Identity;
using Microsoft.AspNetCore.Authentication.JwtBearer;
using Microsoft.Identity.Web;

var builder = WebApplication.CreateBuilder(args);

var azureKeyVaultEndpoint = builder.Configuration["AzureKeyVaultEndpoint"];
if (!string.IsNullOrEmpty(azureKeyVaultEndpoint))
{
    // Add Secrets from Azure Key Vault
    builder.Configuration.AddAzureKeyVault(new Uri(azureKeyVaultEndpoint), new ManagedIdentityCredential());
}
else
{
    // Add Secrets from managed user secrets for local development
    builder.Configuration.AddUserSecrets("e5737a3a-d7aa-4968-88ea-f4c0fe1619b9");
}

The secrets from the Azure Key Vault take precedence over the default configuration which gets provided by the initialized WebApplicationBuilder. The order of the default configuration sources can be found here.

As you can see in the diagram above, the application authenticates against the Azure Key Vault using its system assigned managed identity. Authorization in Azure Key Vault is configured to use Azure role based access control (RBAC) on data plane which is the recommended configuration. For more details see Access model overview.

The following line of code ensures that the key vault is accessed using the system assigned managed identity.

builder.Configuration.AddAzureKeyVault(new Uri(azureKeyVaultEndpoint), new ManagedIdentityCredential());

As the system assigned managed identity is obviously not available in local development environment and to avoid slower startup times in local development environment due to the connection to the Azure Key Vault on every startup, I implemented sort of a fallback to the managed user secrets (secrets.json). User secrets will only be considered if AzureKeyVaultEndpoint is null which is by default true for local development environment (see appsettings.Development.json).

In case you want to test the Azure Key Vault integration locally, proceed as follows.

• Ensure user (developer) is assigned to role Key Vault Secrets User (direct or via a Entra group) on the Azure Key Vault of dev stage
• Use new VisualStudioCredential() instead of new ManagedIdentityCredential()
• Ensure you are logged in with the correct account in Visual Studio
• Apply filter to account in Visual Studio to filter out all Azure tenants except the one you want to connect to
• Ensure the settings described here

Note/Thought: a completely independent local development environment is hardly possible due to the dependency on Microsoft Entra ID – at least not without moving significantly away from the productive environment.

For the integration of the application with Microsoft Entra ID an app registration with a client secret is required. The app registration and the client secret can easily be controlled / managed by IaC (see aadapp in iac/main-entraid.tf). To be able to do secret rotation when deploying to the first stage (dev) already without affecting local development, a dedicated client secret is created for local development purposes (see localdevapppwd in iac/main-entraid.tf). To be able to set the value of AzureAd:ClientSecret in secrets.json, the client secret localdevapppwd has to be grabbed from the key vault (secret with name LocalDevClientSecret). To do so make sure your user is assigned to role Key Vault Secrets User on the Azure Key Vault of dev stage (temporary assignment possible).

The advantage of rotating secrets at the first stage already is, that errors can be detected fast and early in the process. But how is this rotation done? Let’s have a look at the GitHub Actions workflow which covers the whole CI/CD process.

name: CI/CD

on:
  push:
    branches: ["main"]

permissions:
  id-token: write
  contents: read

env:
  AZURE_CORE_OUTPUT: none
  AZURE_WEBAPP_PACKAGE_PATH: "./app.zip"
  DOTNET_VERSION: "9.0.x"
  TERRAFORM_VERSION: "1.9.8"
  TERRAFORM_ROOT_DIRECTORY: "./iac"

jobs:
  build_test_publish:
    name: Build, execute tests and publish
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4
      - name: Setup .NET
        uses: actions/setup-dotnet@v4
        with:
          dotnet-version: ${{ env.DOTNET_VERSION }}
      - name: Build
        run: dotnet build src/ArbitraryAspNetCoreWebApi.sln --configuration Release
      - name: Test
        run: dotnet test src/ArbitraryAspNetCoreWebApi.sln --configuration Release --no-build --verbosity normal
      - name: Publish
        run: dotnet publish src/ArbitraryAspNetCoreWebApi --configuration Release --output ./temp
      - name: Create Zip
        shell: pwsh
        run: |
          cd ./temp
          zip -r ../app.zip ./*
      - name: Upload artifact
        uses: actions/upload-artifact@v4
        with:
          name: arbitrary-aspnetcore-webapi
          path: ${{ env.AZURE_WEBAPP_PACKAGE_PATH }}

  iac_plan:
    name: Plan Infrastructure as Code
    runs-on: ubuntu-latest
    environment: dev
    steps:
      - uses: actions/checkout@v4
      - uses: azure/login@v2
        with:
          client-id: ${{ secrets.AZURE_CLIENT_ID }}
          tenant-id: ${{ secrets.AZURE_TENANT_ID }}
          subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: ${{ env.TERRAFORM_VERSION }}
      - name: Terraform init
        run: terraform -chdir=${{ env.TERRAFORM_ROOT_DIRECTORY }} init --backend-config=backend/dev.backend.tfvars --backend-config='client_id=${{ secrets.AZURE_CLIENT_ID }}' --backend-config='subscription_id=${{ secrets.AZURE_SUBSCRIPTION_ID }}' --backend-config='tenant_id=${{ secrets.AZURE_TENANT_ID }}' --backend-config='use_oidc=true'
      - name: Terraform plan
        run: terraform -chdir=${{ env.TERRAFORM_ROOT_DIRECTORY }} plan --var-file=vars/dev.app.tfvars --var='client_id=${{ secrets.AZURE_CLIENT_ID }}' --var='subscription_id=${{ secrets.AZURE_SUBSCRIPTION_ID }}' --var='tenant_id=${{ secrets.AZURE_TENANT_ID }}' -replace azuread_application_password.aadapppwd --state=dev.app.tfstate -out=tfplan

  iac_apply:
    name: Apply Infrastructure as Code
    runs-on: ubuntu-latest
    environment: dev-iac
    needs: iac_plan
    steps:
      - uses: actions/checkout@v4
      - uses: azure/login@v2
        with:
          client-id: ${{ secrets.AZURE_CLIENT_ID }}
          tenant-id: ${{ secrets.AZURE_TENANT_ID }}
          subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: ${{ env.TERRAFORM_VERSION }}
      - name: Terraform init
        run: terraform -chdir=${{ env.TERRAFORM_ROOT_DIRECTORY }} init --backend-config=backend/dev.backend.tfvars --backend-config='client_id=${{ secrets.AZURE_CLIENT_ID }}' --backend-config='subscription_id=${{ secrets.AZURE_SUBSCRIPTION_ID }}' --backend-config='tenant_id=${{ secrets.AZURE_TENANT_ID }}' --backend-config='use_oidc=true'
      - name: Terraform apply
        run: terraform -chdir=${{ env.TERRAFORM_ROOT_DIRECTORY }} apply --var-file=vars/dev.app.tfvars --var='client_id=${{ secrets.AZURE_CLIENT_ID }}' --var='subscription_id=${{ secrets.AZURE_SUBSCRIPTION_ID }}' --var='tenant_id=${{ secrets.AZURE_TENANT_ID }}' -replace azuread_application_password.aadapppwd --state=dev.app.tfstate -auto-approve

  deploy:
    name: Deploy to Azure Web App
    runs-on: ubuntu-latest
    environment: dev
    needs: [build_test_publish, iac_apply]
    steps:
      - name: Download artifact
        uses: actions/download-artifact@v4
      - uses: azure/login@v2
        with:
          client-id: ${{ secrets.AZURE_CLIENT_ID }}
          tenant-id: ${{ secrets.AZURE_TENANT_ID }}
          subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
      #      - name: Deploy to Azure Web App
      #        uses: azure/webapps-deploy@v3
      #        with:
      #          app-name: kv-secret-rotation-sample-appsrv-dev
      #          resource-group-name: kv-secret-rotation-sample-rg-dev
      #          package: .
      - name: Deploy to Azure Web App
        uses: azure/cli@v2
        with:
          inlineScript: |
            az webapp deploy --resource-group kv-secret-rotation-sample-rg-dev --name kv-secret-rotation-sample-appsrv-dev --src-path $GITHUB_WORKSPACE/arbitrary-aspnetcore-webapi/app.zip --track-status false
      - name: Logout
        run: |
          az logout

The place where secret rotation is triggered is highlighted. The -replace option does the trick. It ensures that every time terraform apply is executed, the azuread_application_password resource is recreated. Additionally the corresponding azurerm_key_vault_secret resource is updated as its value depends on the application password.

resource "azuread_application_password" "aadapppwd" {
  display_name   = "apppwd"
  application_id = azuread_application.aadapp.id
  end_date       = "2099-01-01T00:00:00Z"
}

resource "azurerm_key_vault_secret" "aadapppwd-secret" {
  key_vault_id = azurerm_key_vault.kv.id
  name         = "AzureAd--ClientSecret"
  value        = azuread_application_password.aadapppwd.value
}

The replace option is also used for terraform plan to be able to review the same plan which gets applied during terraform apply. The iac_apply job refers to environment dev-iac which requires a reviewer. This allows to review the plan output in the GitHub Actions workflow logs before terraform apply is executed.

The GitHub Actions workflow ensures the following process on every execution (currently on every push to main branch but it’s also possible to define an additional trigger of type schedule with a cron expression).

1. Build, execute unit tests, publish .NET Core Web API
2. Create Terraform plan
3. Apply Terraform configuration including rotation of client secret
4. Deploy .NET Core Web API to Azure

The deployment triggers an app service restart which ensures that the ASP.NET Core Web API fetches the updated/rotated client secret on startup.

During implementation of the sample I came across other mechanisms that can be used for rotation instead of the -replace option.

• The lifecycle Meta-Argument replace_triggered_by
• Resource argument rotate_when_changed

The full sample code can be found in the following GitHub repository.

https://github.com/rufer7/dotnet-webapi-using-az-key-vault-secret-rotated-by-terraform