Lets say my spring microservice processes data. Every time a successful processing event occurs, for metrics, I update the micrometer counter. This is registered to a Graphite Registry.
registry = new GraphiteMeterRegistry(new GraphiteConfiguration(), Clock.SYSTEM, HierarchicalNameMapper.DEFAULT);
Counter counter = Counter.builder("process").tag("status","success").register(registry);
So far, it sounds good. But what if I have to create and deploy multiple instances of my service?
How do I get the aggregated count of all successful events from all the instances?
To illustrate my case further, I log the counter.count() value on each increment. Here is what i see ->
<Instance 1> <time> <package-name> Count :122
<Instance 2> <time> <package-name> Count :53
So when I run the graphite query on graphana -
process.status.success.count
I tend to get the random count from either of these instances.
What I need is a query like -
process.service-instance.status.success.count
so that I can run a summarize() function in the end.
Update Now I'm able to source data from all instances by getting the service instance ID. But that presents a new problem - Since I restart my services time and again, and my service-id changes every time, how do I source data from ONLY ACTIVE services?
Since process.*.status.success.count represents aggregate count of ALL services - dead or alive