Prometheus: Ignore specific labels while matching

When writing PromQL queries, you often need to compare or perform operations on metrics that differ only by one or two labels. But what if some labels should be excluded from comparison? That’s where the ignoring keyword comes in!

 

Using ignoring, you can tell Prometheus to ignore specific labels while matching series during binary operations like and, or, /, +, -, *, etc.

 

What is the ignoring Keyword in Prometheus?

Prometheus metrics are made up of a metric name and a set of labels. Labels make metrics more specific, but sometimes you want to compare or combine metrics while ignoring one or more labels. That’s where ignoring comes into play.

 

Let’s take two metrics:

 

node_cpu_seconds_total{mode="idle"}

node_cpu_seconds_total{mode="system"}

 

They both track CPU time but for different modes. They share labels like cpu, instance, and job, but differ by mode.

 

If you try this:

node_cpu_seconds_total{mode="idle"} / node_cpu_seconds_total{mode="system"}

 

It won’t work, because Prometheus requires all labels to match unless told otherwise.

 

 

Using ignoring(mode)

node_cpu_seconds_total{mode="idle"}

/ ignoring(mode)

node_cpu_seconds_total{mode="system"}

 

Now, Prometheus will ignore the mode label and match the remaining labels like cpu, instance, and job.

 

Instead of per-mode results, you now get the ratio of idle time to system time per CPU.

 

You can now see, for each CPU, how much more time it spends idle compared to system time — ignoring the mode difference!

 

 

Previous                                                    Next                                                    Home