> it needs to be able to own the buffer for the duration of the entire operation for best results, so it can fill it on any ready thread, map the buffer into the driver's address space, etc.
Can it actually do that? For network write operations the OS has to split the data into MSS/PMTU-sized packets and add headers to it. For network read operations the OS has to reassemble the packets back into a stream or datagram, and it doesn't even know which process a packet is for until after the packet is read into memory.
You're making the copy regardless. Meanwhile IOCP requires O(n) read buffers for n sockets, instead of O(1) when the OS notifies you that it has enough packets to reassemble something.
If the HW has good vectored I/O support then it might be possible to send out data directly from user buffers. This is by composing a packet using two buffer descriptors, one for the header, which points to the next descriptor for the data. But there are complications:
- Almost all hardware would require the buffers to be aligned. Though I think unaligned buffers could probably be handled with a hack by copying some bytes from the user buffer to the "header" buffer.
- The user buffer would need to remain available not only until the packets are transmitted but until they are acknowledged (assuming TCP). Therefore if you want to avoid copying data to kernel buffers for the potential retransmission, the application needs to track which buffers are pending (and is notified when they can be released).
Also, I was reading that zero-copy receive is also possible in some scenarios by changing virtual memory mappings. I'm sure lots of info can be found by googling "zero-copy TCP". FreeBSD supposedly has support for zero-copy TCP.
> Therefore if you want to avoid copying data to kernel buffers for the potential retransmission, the application needs to track which buffers are pending (and is notified when they can be released).
Not necessarily. The kernel could map the page into kernel space and mark it copy-on-write so the application can't modify the kernel page. Then the application doesn't have to care when the kernel is finished with it.
> Also, I was reading that zero-copy receive is also possible in some scenarios by changing virtual memory mappings.
Not necessarily. The kernel could map the page into kernel space and mark it copy-on-write so the application can't modify the kernel page. Then the application doesn't have to care when the kernel is finished with it.
In practice, it's faster to copy the page up front than mess about with the page tables and TLB shootdowns, doubly so if you end up copying the page to break the COW anyway!
> In practice, it's faster to copy the page up front than mess about with the page tables and TLB shootdowns, doubly so if you end up copying the page to break the COW anyway!
It could be worth it when the data fills an entire page or more. And it's common that after a write, the thread will either sleep waiting on events and not get one during the milliseconds it takes for the kernel to be finished with the data, or the buffer is immediately used for read()/recv() which allows the kernel to remap the page without copying it.
But yes, that seems to be the problem in general -- we're trying to optimize something which isn't actually that slow. A memcpy() on a <500 byte packet is only tens of cycles. Even a full page is hundreds of cycles, which is on the same order as the cost of the syscall to have the OS notify the application it has finished with a buffer. None of this complexity can justify its overhead unless you're sending thousands of contiguous bytes, and at that point you're in sendfile() territory anyway.
Can it actually do that? For network write operations the OS has to split the data into MSS/PMTU-sized packets and add headers to it. For network read operations the OS has to reassemble the packets back into a stream or datagram, and it doesn't even know which process a packet is for until after the packet is read into memory.
You're making the copy regardless. Meanwhile IOCP requires O(n) read buffers for n sockets, instead of O(1) when the OS notifies you that it has enough packets to reassemble something.