I am incrementing a Datadog counter in python:
from datadog import initialize
from datadog import ThreadStats
stats.increment('api.request_count', tags=['environment:' + environment])
And have set the metric type to "count" and the unit to "requests per none" in the metadata for the metric.
The code runs in a docker container on a kubernetes node in a Container Engine in Google Cloud... I have docker-dd-agent (https://github.com/DataDog/docker-dd-agent) running on each node.
I can move the container to any node and it logs around 200 requests per minute. But as soon as I scale it up and launch a second container, it only logs around 100 requests per minute. If I scale down to one container again, it spikes to 200 rpm again:
What could be causing the requests to drop or get overwritten from other pods?