Im trying to setup docker in swarm mode and monitor the resource utilization of all the services/containers running in the swarm.
Docker stats on the manager node doesnt seem to show the resource utilization on the worker nodes.
Is there any way I can do this?
Thanks.
Try Ansible:
ansible docker -a "docker stats --no-stream"
Where you setup your "docker" nodes in /etc/ansible/hosts
There's no direct way to retrieve all container stats of a given service in a Swarm. You'll probably have to use more steps to discover all tasks of a service, all node addresses, and each container id. The engine api docs should help you getting started. If you need some inspiration, I'd suggest you to peek into such overview dashboards like the https://github.com/charypar/swarm-dashboard or the https://github.com/dockersamples/docker-swarm-visualizer.
Try this for CPU and Memory usage
docker stats --all --format "table {{.ID}}\t{{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}"
You can use cadvisor to deploy on each node or a swarm service extended on all nodes and collect metrics one by node.
docker run -d --name=cadvisor -p 8080:8080 --volume=/var/run:/var/run:rw --volume=/sys:/sys:ro --volume=/var/lib/docker/:/var/lib/docker:ro google/cadvisor:latest
You can do this for all worker nodes using the following bash command:
docker node ls | grep -v Leader | grep -v Reachable | cut -c 31-47 | grep -v HOSTNAME | xargs -I"SERVER" sh -c "echo SERVER; ssh SERVER docker stats --no-stream"
This command works as follows.
List all Docker swarm nodes:
docker node ls
Filter out the manager nodes:
grep -v Leader | grep -v Reachable
Select the ip-addresses of the worker nodes:
cut -c 31-47
Remove the column header from the result:
grep -v HOSTNAME
Print the ip-address of the worker node and execute docker stats on the worker node:
xargs -I"SERVER" sh -c "echo SERVER; ssh SERVER docker stats --no-stream"