0

我正在使用github 我的项目,它使用 eBPF 基于 SRv6 路由过滤/查找/重定向/丢弃数据包。eBPF 代码在 mellanox Connect5X 上运行以实现 SRv6 功能。

我的期望是mellanox Connect5X 将查看 SRv6 Destination 外部 IPv6 地址标头并在 RX 队列上进行 RSS 数据包传播。这将允许我在多个内核上运行 XDP 进行处理。

我目前的结果是,当 SRv6 数据包是多流数据包(与单流相同的负载效率)时,只使用一个 cpu 内核。

问题是即使对于 SRv6 数据包,我如何才能对 CPU 进行负载平衡?

我期待的答案的一个例子是要知道如何为 IPv6 src、dst addr 等启用 RSS。

谢谢。

4

1 回答 1

0

There is no issue with Mellanox NIC supporting basic RSS ie: outer SRC-IP + Dst-IP + protocol, but the expectation of Mellanox NIC to do RSS for SRV6 header content is incorrect. For the current library (verbs) and firmware as of today RSS and RPS can be validated by

  1. ethtool -S [interface-name] | grep packets | grep rx - for HW RSS spread on multiple RX queues
  2. grep mxl5 /proc/interupts - for queue to CPU mapping.
  3. ethtool --show-rxfh-indir [interface-name] - for identify the flow hash setting

based on the comments, there is a gap in understanding of packet format for SRv6 too. Packet format is ETh + IPv6 + next-header is 43 + srv6 header (next header can be ip/tcp/udp).

hence RSS is done on outer src-IP + dst-ip + protocol (43), the packets with different hash is spread to different queues

enter image description here

now using XDP loaded to the interface, one can filter for SRv6 headers and apply simple xor hash or murmur hash then redirect AF_XDP sockets or interface.

hence the whole expectation and assumption is incorrect

[EDIT-1] based on the live debug we have spent 1.5 hours explaining and educating the same.

[EDIT-2] address the comments raised 1. It refers to what the rx-flow-counter has already accumulated, not the increase in SRv6 packets In the live debug @takeru uses TREX packet generator to send packets to NIC, with packet format as ETH + SRC-IP-1 ... SRC-IP-n + DST-IP + Srv6. With a direct interface to interface connection, no other packets other than SRv6 packets will be recieved

2. In fact, if you check the load on the CPU in the case of SRv6 packets, you will see that only one CPU core is being loaded In the live debug, @takeru did not run top/htop, this is new information. @takeru was trying to understand if RSS on Outer IP is happening or not only. I have requested for a screenshot of CPU usage and tcpdump.

3. If it is only IPv6, the CPU load will be applied to other cores The request has been placed to run simple XDP-eBPF program which redirects/drops ipv6-Srv6 packet. @takeru did not run the same yet

4. Only IPv6 and ip / udp cases have increased the value count by the debugging method you mentioned The same thing happens with SRv6 in linux kernel I have pointed out to @takeru, the TREX packet he is generating of format ETH + Ipv6 + next-hdr routing + Srv6 header + next-header UDP. Hence the kernel statics will update as ipv6/UDP as it is not TCP or not SCTP or unknown protocol.

Note: takeru's reference github project

于 2021-02-05T07:53:58.680 回答