What Are Labels in Prometheus Alerts?

When creating alerting rules in Prometheus, labels help you to organize, filter, and route your alerts. If you're new to alerting in Prometheus, this post will help you understand what labels are, why they matter, and how to use them.

What Are Labels in Prometheus Alerts?

Labels in Prometheus alerting rules are key-value pairs that add extra information to your alerts.

 

They help you:

 

·      Categorize alerts (e.g., severity: critical)

·      Route alerts to the right teams or channels

·      Filter and group alerts in tools like Alertmanager, Grafana, or Slack

 

Let’s look at a simple alert rule:

groups:
  - name: example_alerts
    rules:
      - alert: NodeExporterDown
        expr: up{job="node_exporter"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Node Exporter is down"
          description: "No data received from node_exporter for more than 1 minute"

Here:

·      alert: is the name of the alert – NodeExporterDown

·      labels: is where you add labels like "severity: critical"

 

Can I Add More Than One Label?

Yes! You can add as many labels as you want.

labels:
  severity: critical
  team: infra
  region: us-east-1

This makes your alert more descriptive and useful.

 

Now:

 

·      You know it's critical

·      It’s for the infra team

·      It’s happening in the us-east-1 region

 

How Labels Help with Alert Routing?

Tools like Alertmanager use labels to decide:

 

·      Where to send the alert (email, Slack, PagerDuty, etc.)

·      Whether to silence or group the alert

·      Which team should respond

 

For example,

route:
  receiver: slack-infra
  match:
    team: infra
    severity: critical

If an alert is critical and belongs to the infra team, send it to slack-infra.

 

rules_with_labels.yaml

groups:
  - name: alerts_with_labels
    rules:
      # Alert 1: Heavy load on Prometheus /metrics endpoint
      - alert: HighMetricsTraffic
        expr: avg_over_time(prometheus_http_requests_total{handler="/metrics"}[1m]) > 100
        for: 1m
        labels:
          severity: warning
          team: observability
          env: production
        annotations:
          summary: "High request rate on /metrics endpoint"
          description: "Prometheus is receiving more than 100 requests per minute on the /metrics handler"

      # Alert 2: Node Exporter Down
      - alert: NodeExporterDown
        expr: up{job="node_exporter"} == 0
        for: 2m
        labels:
          severity: critical
          team: infra
          env: production
        annotations:
          summary: "Node Exporter is not reachable"
          description: "No data received from node_exporter for more than 2 minutes. Check connectivity or service health."

      # Alert 3: High CPU Usage on a Node
      - alert: HighCPUUsage
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[1m])) * 100) > 85
        for: 1m
        labels:
          severity: warning
          team: infra
          env: production
        annotations:
          summary: "High CPU usage on instance"
          description: "CPU usage is above 85% for more than 1 minute on {{ $labels.instance }}"

let’s add rules_with_labels.yaml file to the prometheus configuration file.

 

prometheus.yml

 

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - "rules_with_labels.yaml"

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'node_exporter'
    static_configs:
      - targets: ['localhost:9100']

Start Prometheus application by executing below command.

prometheus --config.file=./prometheus.yml

 

Open the url http://localhost:9090/alerts in browser, you can see the that the three alerts are inactive.

 

Expand the alert, you can see the tags and annotations associated with.

 

You can filter the alerts by a label. For example, when you type the text warning in the search box. It filter the data.  

 

 

Previous                                                    Next                                                    Home