11

我一直在阅读有关 NIC 捕获数据包后发生的情况，并且我阅读的越多，我就越困惑。

首先，我读过传统上，在 NIC 捕获数据包后，它会被复制到内核空间中的一块内存，然后复制到用户空间，以供随后处理数据包数据的任何应用程序使用。然后我阅读了 DMA，其中 NIC 绕过 CPU 直接将数据包复制到内存中。那么网卡->内核内存->用户空间内存流还有效吗？此外，大多数 NIC（例如 Myricom）是否使用 DMA 来提高数据包捕获率？

其次，RSS（接收端缩放）在 Windows 和 Linux 系统中的工作方式是否相似？我只能在 MSDN 文章中找到关于 RSS 工作原理的详细解释，他们讨论了 RSS（和 MSI-X）在 Windows Server 2008 上的工作原理。但是 RSS 和 MSI-X 的相同概念仍然应该适用于 linux 系统，对吧?

谢谢你。

问候，雷恩

4

2 回答 2

18

How this process plays out is mostly up to the driver author and the hardware, but for the drivers I've looked at or written and the hardware I've worked with, this is usually the way it works:

  1. At driver initialization, it will allocate some number of buffers and give these to the NIC.
  2. When a packet is received by the NIC, it pulls the next address off its list of buffers, DMAs the data directly into it, and notifies the driver via an interrupt.
  3. The driver gets the interrupt, and can either turn the buffer over to the kernel or it will allocate a new kernel buffer and copy the data. "Zero copy networking" is the former and obviously requires support from the operating system. (more below on this)
  4. The driver needs to either allocate a new buffer (in the zero-copy case) or it will re-use the buffer. In either case, the buffer is given back to the NIC for future packets.

Zero-copy networking within the kernel isn't so bad. Zero-copy all the way down to userland is much harder. Userland gets data, but network packets are made up of both header and data. At the least, true zero-copy all the way to userland requires support from your NIC so that it can DMA packets into separate header/data buffers. The headers are recycled once the kernel routes the packet to its destination and verifies the checksum (for TCP, either in hardware if the NIC supports it or in software if not; note that if the kernel has to compute the checksum itself, it'd may as well copy the data, too: looking at the data incurs cache misses and copying it elsewhere can be for free with tuned code).

Even assuming all the stars align, the data isn't actually in your user buffer when it is received by the system. Until an application asks for the data, the kernel doesn't know where it will end up. Consider the case of a multi-process daemon like Apache. There are many child processes, all listening on the same socket. You can also establish a connection, fork(), and both processes are able to recv() incoming data.

TCP packets on the Internet are usually 1460 bytes of payload (MTU of 1500 = 20 byte IP header + 20 byte TCP header + 1460 bytes data). 1460 is not a power of 2 and won't match a page size on any system you'll find. This presents problems for reassembly of the data stream. Remember that TCP is stream-oriented. There is no distinction between sender writes, and two 1000 byte writes waiting at the received will be consumed entirely in a 2000 byte read.

Taking this further, consider the user buffers. These are allocated by the application. In order to be used for zero-copy all the way down, the buffer needs to be page-aligned and not share that memory page with anything else. At recv() time, the kernel could theoretically remap the old page with the one containing the data and "flip" it into place, but this is complicated by the reassembly issue above since successive packets will be on separate pages. The kernel could limit the data it hands back to each packet's payload, but this will mean a lot of additional system calls, page remapping and likely lower throughput overall.

I'm really only scratching the surface on this topic. I worked at a couple of companies in the early 2000s trying to extend the zero-copy concepts down into userland. We even implemented a TCP stack in userland and circumvented the kernel entirely for applications using the stack, but that brought its own set of problems and was never production quality. It's a very hard problem to solve.

于 2010-04-27T17:16:18.313 回答
-1

看看这篇论文，http://www.ece.virginia.edu/cheetah/documents/papers/TCPlinux.pdf它可能有助于清除一些内存管理问题

于 2010-04-26T20:21:45.350 回答