I'm in the process of writing a Prometheus Exporter in Go to expose metrics pushed from AIX severs. The AIX servers push their metrics (in json) to a central listener (the exporter program) that converts them to standard Prometheus metrics and exposes them for scraping.
The issue I have is that the hostname for the metrics is extracted from the pushed json. I store this as a label in each metric. E.g. njmon_memory_free{lpar="myhostname"}. While this works, it's less than ideal as there doesn't seem to be a way to relabel this to the usual instance label (njmon_memory_free{instance="myhostname"}. The Prometheus relabelling happens before the scrape so the lpar label isn't there to be relabelled.
One option seems to be to rewrite the exporter so that the Prometheus server probes defined targets, each target being the lpar. In order for that to work, I'd need a means to filter the stored metrics by lpar so only metrics relating to the target/lpar are returned. Is this a practical solution or am I forced to create a dedicated listener or url for every lpar?