0

I want to automatically scrape data from all instantiated services in my docker with Prometheus. I do this on a cluster with two workers and about 7 services. The services I want to scrape are deployed globally.

I've set Prometheus up to scrape using dns_sd_config and the target of tasks.cadvisor. This will result in a single host being returned, while it should be two services.

> tasks.cadvisor
Server:         127.0.0.11
Address:        127.0.0.11#53

Non-authoritative answer:
Name:   tasks.cadvisor
Address: 10.0.1.9

In this example I can only find a single CAdvisor node, while there are actually two.

However, when I do a lookup for a service that runs twice on the same worker node, the lookup manages to find both of the services

> tasks.nginx
Server:         127.0.0.11
Address:        127.0.0.11#53

Non-authoritative answer:
Name:   tasks.nginx
Address: 10.0.1.25
Name:   tasks.nginx
Address: 10.0.1.20

It seems like Docker DNS cannot do a lookup beyond it's own worker node. How can I set Docker up in a way that the DNS lookup returns all service instances across all workers?

Here's my current docker setup:

version: '3'
services:
  db:
    image: postgres
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.role == manager
    volumes:
      - db-data:/var/lib/postgresql/data
  backend:
    build: reggie-server
    image: requinard2/reggie-server
    command: python manage.py runserver 0.0.0.0:8000
    deploy:
      mode: global
    environment:
      - PRODUCTION=1
    depends_on:
      - db
  nginx:
    build: reggie-nginx
    image: requinard2/reggie-nginx
    deploy:
      mode: global
    ports:
      - "80:80"
      - "443:443"
    depends_on:
      - "backend"
      - "prometheus"
      - "grafana"
  prometheus:
    build: reggie-prometheus
    image: requinard2/reggie-prometheus
    ports:
      - "9090:9090"
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.role == manager
    volumes:
      - prometheus-data:/prometheus
    depends_on:
      - backend
      - cadvisor
  grafana:
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.role == manager
    image: grafana/grafana:5.1.0
    environment:
      GF_SERVER_ROOT_URL=/grafana:
    volumes:
      - grafana-data:/var/lib/grafana
    depends_on:
      - "prometheus"
  cadvisor:
    image: google/cadvisor:latest
    deploy:
      mode: global
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
    depends_on:
      - redis
  redis:
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.role == manager
    image: redis:latest
volumes:
  backend-code:
  db-data:
  grafana-data:
  prometheus-data:
4

1 回答 1

0

在摆弄它之后，我想到尝试在与我一直使用的云不同的环境中运行这个特定的问题。我使用 docker-machine 创建了两个本地实例，它立即工作。我开始挖掘了一下，结果发现我的防火墙没有正确配置。这使我的节点无法相互通信。

所以我打开了以下端口，如此处所述：

  • 2377/tcp
  • 7946/tcp&udp
  • 4789/UDP

这完全解决了问题，我的节点现在可以正确地相互通信了！

于 2019-03-21T21:15:08.937 回答