Place the node role as a label value

[jorgelbg]

I'm opening this issue to discuss if this change would make sense

At the moment the role of each server in the cluster is stored in a label with a true/false value:

metric_name{cluster="cluster1",es_client_node="true",es_data_node="false",
es_ingest_node="false", es_master_node="true",...}

This provides the ability to filter by a given role in the cluster, but it's not very friendly to dashboard solutions.

Having the role as the value of a label (...,role="client",...) could be easier to integrate into some dashboarding solutions. For instance, on Grafana, this value could be fetched from the metric with the label_values(metric, label) function. This would allow using the "values group" feature.

So far the alternative that I've found for this is to accomplish this using relabeling. If the feature is welcomed I can submit a PR.

[zwopir]

Hi @jorgelbg,

we deliberately decided to encode the role info as bool-style key/value pairs. Your proposed node role label can have multiple roles and a role label value could be for example data,ingest,client. Evaluating such a label in a prom-query (dashboard) would require the knowledge how multiple roles are encoded in the label value. With the current solution you only need to know the label names. So from my point of view I'd rather not change the way we include the role information in the metrics.

What do you think, @marthjod , @metalmatze ?

[jorgelbg]

@zwopir Thanks for the reply and for opening a discussion about this.

Regarding the multiple roles I've run into this scenario as well. I have some nodes with mixed roles in my clusters as well. Indeed, es_*_node label is very handy for filtering but makes extracting the possible roles programmatically quite complicated (unless they're hardcoded). On the other hand, having multiple roles on a single label gets tricky (as you said).

I was thinking about a similar approach to the one discussed here https://www.robustperception.io/how-to-have-labels-for-machine-roles. But perhaps this approach could cause some other issues 🤔.

[zwopir]

that's actually a very nice solution to our problem. I wasn't of it. Thanks for bringing it up!
So yes, if you could contribute a PR, that would be very nice. It shouldn't be too much work as the logic of figuring out the node role is already in the code.

Later in the life-cycle if the exporter we may even think of deprecating the es_*_node label.

[zwopir]

see #228

[zwopir]

Hi @jorgelbg,

I implemented the feature in #228. Can you build from that branch and report back if it works for you?

[jorgelbg]

Hi @zwopir, so sorry that I forgot to submit the PR for this issue 😓. I will test the changes in #228 and report here.

As a side note, if you run the prometheus node exporter (for the machine metrics) along the elasticsearch exporter, there is the node_uname_info, so a query like this one should work:

node_filesystem_avail_bytes{device="/dev/sdb"} * 
on (instance, job) group_left(nodename)  node_uname_info{nodename=~"elastic-storage.*"}

[zwopir]

thanks for sharing the query. Most of the service discoveries in prometheus offer the possibility to export machine/node meta data as a label (with some relabeling config needed), but in case you're not using a SD, your trick can be handy.

[zwopir]

implemented the feature in #228