[NoBrainer] Ensure that Azure Monitor alert email notifications are delivered

When working with Azure Monitor alerts that trigger email notifications, there is a tiny little detail to consider to make sure the email notifications get delivered as expected. But let’s first have a look at the setup of an Azure Monitor alert.

In my case, I set up an Azure Monitor metric alert and action group with Terraform as follows.

Terraform version: 1.4.6
hashicorp/azurerm provider version: 3.56.0

Azure Monitor metric alert

resource "azurerm_monitor_metric_alert" "ma-backend-api-http-server-errors" {
  name                = format("%s-%s-ma-appsrv-backend-api-http-server-errors", var.customer, var.environment)
  resource_group_name = azurerm_resource_group.rg-example.name
  scopes = [
    azurerm_windows_web_app.appsrv-backend-api.id
  ]
  description = "Action will be triggered when HTTP server error occurs."

  criteria {
    metric_namespace = "Microsoft.Web/sites"
    metric_name      = "Http5xx"
    aggregation      = "Total"
    operator         = "GreaterThanOrEqual"
    threshold        = 1
  }

  action {
    action_group_id = azurerm_monitor_action_group.ag-3rd-level.id
  }

  frequency   = "PT1H"
  window_size = "PT6H"
  severity    = 1
}

The above Terraform code creates an Azure Monitor metric alert that is triggered when the HTTP server error count of the corresponding Azure Web App is greater than or equal to 1. The azurerm_monitor_metric_alert resource creates the alert and specifies the name, resource group, and scope of the alert. The criteria block specifies the metric namespace, metric name, aggregation type, operator, and threshold that trigger the alert. The action block specifies the action group to be triggered when the alert is fired.

For more details about Azure Monitor alert rules see here.

Action Group

resource "azurerm_monitor_action_group" "ag-3rd-level" {
  name                = format("%s-%s-ag-3rdLevelSupport", var.customer, var.environment)
  resource_group_name = azurerm_resource_group.rg-example.name
  short_name          = "3rdLevel"

  arm_role_receiver {
    name                    = "MonitoringReader"
    role_id                 = "OBJECT_ID_HERE"
    use_common_alert_schema = true
  }
}

The above Terraform code creates an action group with short name 3rdLevel that can be used to trigger actions when alerts that belong to this action group are fired. The arm_role_receiver block specifies the Azure Resource Manager (ARM) role that receives notification emails of alerts assigned to this action group.

Now we come to the tiny little detail mentioned at the beginning, which is not quite obvious. The email notifications are only delivered to a users email address, if and only if, the corresponding user is assigned to the ARM role at the subscription scope.

To assign a user to an ARM role (in this case Monitoring Reader) at the subscription scope, proceed as follows.

1. Go to the Azure portal and switch to the corresponding tenant/directory
2. Search for Subscriptions in the search bar on the top and select Subscriptions
3. Select the subscription that contains the resource the metric alert was created for
4. Navigate to Access control (IAM) in the left navigation bar
5. Switch to tab Role assignments
6. Click + Add and then select Add role assignment
7. Under tab Job function roles search for the corresponding ARM role (in this case Monitoring Reader) and select it
8. Click Next on the bottom
9. Click + Select members
10. Choose the desired members (only members of type user are supported)
11. Click Select
12. Click Review + assign

If the assigned users do still not receive email notifications, see Troubleshooting problems in Azure Monitor alerts – Did not receive expected email in the docs.