1

在 Ubuntu 20.04 服务器上,我运行 Debian 10 作为 KVM-guest,通过 macvtap-bridge 连接。在那个客人里面,我有一个 macvlan-bridge,连接着一个 docker 容器。

由于某种原因,该容器似乎没有任何网络连接,我看不到错误。

编辑2

由于我还没有得到任何答案,而且我还继续调查整个主题,我开始怀疑我的想法是否真的有效!

因此,我想提前问一个非常普遍的问题:是否有可能有一个 KVM-guest,它使用一些容器运行 docker,并通过“公共网桥”将它们全部连接起来?这样主机、KVM-guest 和 docker 容器都出现在本地局域网上,就像单独的机器一样(当然可以相互通信)。这种类型的设置称为“嵌套虚拟化”。但是我找不到一些参考资料来解释如何“公共桥接”两个级别的嵌套 VM。

因此,如果有人可以首先澄清这个问题,最好的情况是包括有关如何做到这一点的文档/说明。

/编辑2

编辑1

为了提供更多背景知识,我基本上尝试实现这样的设置:

   +-------------------------------+
   | +---+---+---+   +-----------+ |
   | |Cn1|Cn2|CnX|   |Cn1|Cn2|CnX| |
   | +---+---+---+   +---+---+---+ |
   | |  Docker   |   |  Docker   | |
   | +-----------+   +-----------+ |
   | |  macvlan  |   |  macvlan  | |
   | |VM1 eth0   |   |VM2 eth0   | |
   | +-----------+   +-----------+ |
   |    macvtap         macvtap    |
   |    +----+          +----+     |
   |    |eth0|          |eth1|     |
   |    +----+   Host   +----+     |
   +-------------------------------+
                  |
                 LAN

所以 Host、VM1、VM2 和 docker 容器应该直接连接到 LAN,就像它们是物理机器一样。因此,如果有人能指出我实现这一目标的不同方法,我也会很高兴......

/编辑1

以下是配置详细信息:

Ubuntu主机(香蕉)

/etc/网络/接口:

# The primary network interface
auto eth0
iface eth0 inet static
        address 192.168.3.3
        netmask 255.255.255.0
        gateway 192.168.3.1
        dns-search *****.local
        dns-nameserver 192.168.3.1

# USB-ethernet adapter
auto eth1
iface eth1 inet static
        address 192.168.2.12
        netmask 255.255.255.0
        broadcast 255.255.255.255
        dns-nameservers 192.168.2.250
        dns-search *****.local

ip地址:

root@banana:~# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether dc:fe:07:e1:80:e6 brd ff:ff:ff:ff:ff:ff
    inet 192.168.3.3/24 brd 192.168.3.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::b0d2:f660:df88:856f/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:e0:4c:68:13:97 brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.12/24 brd 255.255.255.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet 192.168.2.104/24 brd 192.168.2.255 scope global secondary noprefixroute eth1
       valid_lft forever preferred_lft forever
    inet6 fdc3:1fe4:1492::eb8/128 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fdc3:1fe4:1492:0:942c:ded8:d9ee:2862/64 scope global temporary dynamic 
       valid_lft 600505sec preferred_lft 81850sec
    inet6 fdc3:1fe4:1492:0:1b46:b4f9:692c:2f10/64 scope global mngtmpaddr noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::63b7:ce64:c7d1:73da/64 scope link 
       valid_lft forever preferred_lft forever
4: virbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:02:0c:0d brd ff:ff:ff:ff:ff:ff
    inet 192.168.254.1/24 brd 192.168.254.255 scope global virbr1
       valid_lft forever preferred_lft forever
    inet 169.254.70.129/16 brd 169.254.255.255 scope global noprefixroute virbr1
       valid_lft forever preferred_lft forever
5: virbr1-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master virbr1 state DOWN group default qlen 1000
    link/ether 52:54:00:02:0c:0d brd ff:ff:ff:ff:ff:ff
6: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:44:21:01 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
7: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master virbr0 state DOWN group default qlen 1000
    link/ether 52:54:00:44:21:01 brd ff:ff:ff:ff:ff:ff
8: macvtap0@eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 500
    link/ether 52:54:00:be:09:bc brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.63/24 brd 192.168.2.255 scope global noprefixroute macvtap0
       valid_lft forever preferred_lft forever
    inet6 fe80::b4bb:1ac3:ab06:b569/64 scope link 
       valid_lft forever preferred_lft forever
9: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master virbr1 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:1f:e8:10 brd ff:ff:ff:ff:ff:ff
    inet 169.254.134.94/16 brd 169.254.255.255 scope global noprefixroute vnet0
       valid_lft forever preferred_lft forever
    inet6 fe80::ea3:8aaf:a558:b1b6/64 scope link 
       valid_lft forever preferred_lft forever
    inet6 fe80::fc54:ff:fe1f:e810/64 scope link 
       valid_lft forever preferred_lft forever

Debian-guest 的 Libvirt XML-部分网络:

<interface type='direct'>
  <mac address='52:54:00:be:09:bc'/>
  <source dev='eth1' mode='bridge'/>
  <model type='virtio'/>
  <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</interface>
<interface type='network'>
  <mac address='52:54:00:1f:e8:10'/>
  <source network='isolated'/>
  <model type='virtio'/>
  <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
</interface>

Debian 来宾 (bananaLAN)

/etc/网络/接口:

# The primary network interface
auto enp1s0
iface enp1s0 inet static
        address 192.168.2.3
        netmask 255.255.255.0
        gateway 192.168.2.1

# for host communication
auto enp9s0
iface enp9s0 inet static
        address 192.168.254.3
        netmask 255.255.255.0

ip地址

root@bananaLAN:/# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:be:09:bc brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.3/24 brd 192.168.2.255 scope global enp1s0
       valid_lft forever preferred_lft forever
    inet6 fdc3:1fe4:1492:0:5054:ff:febe:9bc/64 scope global dynamic mngtmpaddr 
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:febe:9bc/64 scope link 
       valid_lft forever preferred_lft forever
3: enp9s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:1f:e8:10 brd ff:ff:ff:ff:ff:ff
    inet 192.168.254.3/24 brd 192.168.254.255 scope global enp9s0
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe1f:e810/64 scope link 
       valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:0d:e8:3a:25 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever

码头工人(程序)

网络创建

docker network create \
  --driver=macvlan \
  --subnet=192.168.2.0/24 \
  --gateway 192.168.2.1 \
  --ip-range=192.168.2.224/27 \
  -o parent=enp1s0 \
  pub_net

容器创建

docker run -d \
    --name routix \
    --network=pub_net \
    --ip 192.168.2.250 \
    -p 53:53/tcp -p 53:53/udp \
    -p 80:80 \
    -p 443:443 \
    -e TZ="Europe/Berlin" \
    -v "$(pwd)/etc-pihole/:/etc/pihole/" \
    -v "$(pwd)/etc-dnsmasq.d/:/etc/dnsmasq.d/" \
    --dns=127.0.0.1 --dns=192.168.3.1 \
    --restart=unless-stopped \
    pihole/pihole:latest

ip地址:

root@3c59f964891c:/# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
5: eth0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:c0:a8:02:fa brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.2.250/24 brd 192.168.2.255 scope global eth0
       valid_lft forever preferred_lft forever

Ping 结果

Ubuntu 主机(香蕉)到网关:

root@banana:~# ping 192.168.2.1
PING 192.168.2.1 (192.168.2.1) 56(84) Bytes Daten.
64 Bytes von 192.168.2.1: icmp_seq=1 ttl=64 Zeit=0.585 ms
64 Bytes von 192.168.2.1: icmp_seq=2 ttl=64 Zeit=0.572 ms

Ubuntu 主机(香蕉)到 LAN 主机:

root@banana:~# ping 192.168.2.20
PING 192.168.2.20 (192.168.2.20) 56(84) Bytes Daten.
64 Bytes von 192.168.2.20: icmp_seq=1 ttl=64 Zeit=0.810 ms
64 Bytes von 192.168.2.20: icmp_seq=2 ttl=64 Zeit=0.545 ms

Ubuntu 主机(香蕉)到 Debian 来宾:(这是预期的)

root@banana:~# ping 192.168.2.3
PING 192.168.2.3 (192.168.2.3) 56(84) Bytes Daten.
Von 192.168.2.12 icmp_seq=1 Zielhost nicht erreichbar
Von 192.168.2.12 icmp_seq=2 Zielhost nicht erreichbar

Ubuntu 主机(香蕉)通过独立网络连接到 Debian 来宾:

root@banana:~# ping 192.168.254.1
PING 192.168.254.1 (192.168.254.1) 56(84) Bytes Daten.
64 Bytes von 192.168.254.1: icmp_seq=1 ttl=64 Zeit=0.222 ms
64 Bytes von 192.168.254.1: icmp_seq=2 ttl=64 Zeit=0.065 ms

Debian 来宾 (bananaLAN) 到网关:

root@bananaLAN:/# ping 192.168.2.1
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=0.709 ms
64 bytes from 192.168.2.1: icmp_seq=2 ttl=64 time=0.701 ms

Debian 来宾 (bananaLAN) 到 LAN 主机:

root@bananaLAN:/# ping 192.168.2.20
PING 192.168.2.20 (192.168.2.20) 56(84) bytes of data.
64 bytes from 192.168.2.20: icmp_seq=1 ttl=64 time=0.938 ms
64 bytes from 192.168.2.20: icmp_seq=2 ttl=64 time=0.872 ms

Debian 来宾(bananaLAN)到 Ubuntu 主机:(这是预期的)

root@bananaLAN:/# ping 192.168.2.12
PING 192.168.2.12 (192.168.2.12) 56(84) bytes of data.
From 192.168.2.3 icmp_seq=1 Destination Host Unreachable
From 192.168.2.3 icmp_seq=2 Destination Host Unreachable

Debian 来宾(bananaLAN)通过独立网络连接到 Ubuntu 主机:

root@bananaLAN:/# ping 192.168.254.3
PING 192.168.254.3 (192.168.254.3) 56(84) bytes of data.
64 bytes from 192.168.254.3: icmp_seq=1 ttl=64 time=0.098 ms
64 bytes from 192.168.254.3: icmp_seq=2 ttl=64 time=0.039 ms

Docker 容器(路由)到网关:(这就是问题所在!)

root@bananaLAN:/# docker exec -it routix bash
root@3c59f964891c:/# ping 192.168.2.1
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
From 192.168.2.250 icmp_seq=1 Destination Host Unreachable
From 192.168.2.250 icmp_seq=2 Destination Host Unreachable

Docker 容器(路由)到 LAN 主机:(这就是问题所在!)

root@3c59f964891c:/# ping 192.168.2.20
PING 192.168.2.20 (192.168.2.20) 56(84) bytes of data.
From 192.168.2.250 icmp_seq=1 Destination Host Unreachable
From 192.168.2.250 icmp_seq=2 Destination Host Unreachable

Docker 容器(路由)到 Ubuntu 主机:

root@3c59f964891c:/# ping 192.168.2.12
PING 192.168.2.12 (192.168.2.12) 56(84) bytes of data.
From 192.168.2.250 icmp_seq=1 Destination Host Unreachable
From 192.168.2.250 icmp_seq=2 Destination Host Unreachable

Docker 容器(路由)到 Debian 主机:

root@3c59f964891c:/# ping 192.168.2.3 
PING 192.168.2.3 (192.168.2.3) 56(84) bytes of data.
From 192.168.2.250 icmp_seq=1 Destination Host Unreachable
From 192.168.2.250 icmp_seq=2 Destination Host Unreachable

LAN主机到网关:

~ % ping 192.168.2.1
PING 192.168.2.1 (192.168.2.1): 56 data bytes
64 bytes from 192.168.2.1: icmp_seq=0 ttl=64 time=0.392 ms
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=0.406 ms

LAN 主机到 Ubuntu 主机(香蕉):

~ % ping 192.168.2.12
PING 192.168.2.12 (192.168.2.12): 56 data bytes
64 bytes from 192.168.2.12: icmp_seq=0 ttl=64 time=0.593 ms
64 bytes from 192.168.2.12: icmp_seq=1 ttl=64 time=0.652 ms

LAN 主机到 Debian 主机(bananaLAN):

~ % ping 192.168.2.3 
PING 192.168.2.3 (192.168.2.3): 56 data bytes
64 bytes from 192.168.2.3: icmp_seq=0 ttl=64 time=0.927 ms
64 bytes from 192.168.2.3: icmp_seq=1 ttl=64 time=1.025 ms

LAN 主机到 Docker 容器(路由):(这就是问题所在!)

~ % ping 192.168.2.250
PING 192.168.2.250 (192.168.2.250): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1

路由表

网关 (192.168.2.1)

root@OpenWrt:~# ip route
default via 192.168.3.1 dev eth0.10 
192.168.2.0/24 dev br-lan scope link  src 192.168.2.1 
192.168.3.0/24 dev eth0.10 scope link  src 192.168.3.2 

Ubuntu主机(香蕉)

root@banana:~# ip route
default via 192.168.3.1 dev eth0 src 192.168.3.3 metric 202 
default via 192.168.2.1 dev eth1 proto dhcp src 192.168.2.104 metric 203 
default via 192.168.2.1 dev macvtap0 proto dhcp src 192.168.2.63 metric 208 
169.254.0.0/16 dev virbr1 scope link src 169.254.70.129 metric 204 
169.254.0.0/16 dev vnet0 scope link src 169.254.134.94 metric 209 
192.168.2.0/24 dev eth1 proto dhcp scope link src 192.168.2.104 metric 203 
192.168.2.0/24 dev macvtap0 proto dhcp scope link src 192.168.2.63 metric 208 
192.168.3.0/24 dev eth0 proto dhcp scope link src 192.168.3.3 metric 202 
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown 
192.168.254.0/24 dev virbr1 proto kernel scope link src 192.168.254.1

Debian 来宾 (bananaLAN)

root@bananaLAN:/# ip route
default via 192.168.2.1 dev enp1s0 onlink 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
192.168.2.0/24 dev enp1s0 proto kernel scope link src 192.168.2.3 
192.168.254.0/24 dev enp9s0 proto kernel scope link src 192.168.254.3

Docker 容器(路由)

root@3c59f964891c:/# ip route
default via 192.168.2.1 dev eth0 
192.168.2.0/24 dev eth0 proto kernel scope link src 192.168.2.250

可能这是某种路由问题,我不得不承认,这不是我的强项。我还仔细检查了是否完全遵循了围绕 qemu/libvirt 和 docker/macvlan 的几个文档,但显然我遗漏了一些东西。

我很感激任何帮助!非常感谢!

4

0 回答 0