2. The forking model is seductive, but wrong-headed. It's hard to make the file descriptor inheritance behave correctly, and it means the memory requirements are unpredictable (due to the copy-on-write pages)

3. Readiness I/O is not really the right way to do things, because there are obvious race conditions, and the OS can't guarantee one thread woken per pending operation (because it has no idea which threads will do what). Also, the process owns the buffer, which really limits what the OS can do... it needs to be able to own the buffer for the duration of the entire operation for best results, so it can fill it on any ready thread, map the buffer into the driver's address space, etc.

4. Poor multithreading primitives. This really annoyed me... like, you've got a bunch of pthreads stuff, but it doesn't interact with select & co. The Linux people aren't dumb, so they give you eventfd - but there's no promise of single wakeup! NT wakes up one thread per increment, and the wakeup atomically decrements the semaphore; Linux wakes up every thread waiting on the semaphore, and they all fight over it, because there's no other option.

(I'm just ignoring POSIX semaphores entirely, because they don't let you wait on them with select/poll/etc. in the first place.)

(Perhaps this and #3 ought to be the same item, because they are related. And the end result is that you need to use non-blocking IO for everything... but the non-blocking IO is crippled, because it still has to copy out into the caller's buffer. It's just a more inconvenient programming model, for no real benefit.)

I guess it just boils down to what you want: a beautifully polished turd, or a carefully engineered system assembled from turds.

You can get Linux to only wake up one thread using epoll and either EPOLLEXCLUSIVE or EPOLLONESHOT.

Though I don't really understand why you'd want to wait on a semaphore and other waitables at the same time… probably this is just my Unix bias/lack of Windows experience showing.

This. I probably also lack some specific kind of experience because I never understood why you would lock a lot of threads on a semaphore, and only want one of them to execute after a signal.

I just never saw the use case for that. Yet people complain about it a lot.

Can it actually do that? For network write operations the OS has to split the data into MSS/PMTU-sized packets and add headers to it. For network read operations the OS has to reassemble the packets back into a stream or datagram, and it doesn't even know which process a packet is for until after the packet is read into memory.

You're making the copy regardless. Meanwhile IOCP requires O(n) read buffers for n sockets, instead of O(1) when the OS notifies you that it has enough packets to reassemble something.

- Almost all hardware would require the buffers to be aligned. Though I think unaligned buffers could probably be handled with a hack by copying some bytes from the user buffer to the "header" buffer.

- The user buffer would need to remain available not only until the packets are transmitted but until they are acknowledged (assuming TCP). Therefore if you want to avoid copying data to kernel buffers for the potential retransmission, the application needs to track which buffers are pending (and is notified when they can be released).

Also, I was reading that zero-copy receive is also possible in some scenarios by changing virtual memory mappings. I'm sure lots of info can be found by googling "zero-copy TCP". FreeBSD supposedly has support for zero-copy TCP.

Not necessarily. The kernel could map the page into kernel space and mark it copy-on-write so the application can't modify the kernel page. Then the application doesn't have to care when the kernel is finished with it.

> Also, I was reading that zero-copy receive is also possible in some scenarios by changing virtual memory mappings.

You could do that regardless of IOCP too.

In practice, it's faster to copy the page up front than mess about with the page tables and TLB shootdowns, doubly so if you end up copying the page to break the COW anyway!

It could be worth it when the data fills an entire page or more. And it's common that after a write, the thread will either sleep waiting on events and not get one during the milliseconds it takes for the kernel to be finished with the data, or the buffer is immediately used for read()/recv() which allows the kernel to remap the page without copying it.

But yes, that seems to be the problem in general -- we're trying to optimize something which isn't actually that slow. A memcpy() on a <500 byte packet is only tens of cycles. Even a full page is hundreds of cycles, which is on the same order as the cost of the syscall to have the OS notify the application it has finished with a buffer. None of this complexity can justify its overhead unless you're sending thousands of contiguous bytes, and at that point you're in sendfile() territory anyway.

On modern Linux OOM killer became rather good at finding the real memory offenders. Plus this unpredictability allows, for example, to transparently enable memory compression in Linux eliminating the need for swap in many configurations.

Can Linux do that? I remember when OS X introduced it, performance improved notably because the system did much less paging. Would be nice to have that on Linux.

(Rephrase my question: If Linux can do that, do I have to take any special steps to enable it?)

EDIT: Nevermind, I should have thought of googling for it first!

Windows kernel API documentation vs. Linux kernel API documentation... there's not even a contest to be had.

and even the kernel API documentation, which, while it has its issues, I would argue is still far better than MSDN, which IME is mostly pages and pages of function prototypes with a one-line restatement of the name of the function.

MSDN:

"If this flag is set, a child process created with the bInheritHandles parameter of CreateProcess set to TRUE will inherit the object handle."

Linux:

"Enable the close-on-exec flag for the new file descriptor. Specifying this flag permits a program to avoid additional fcntl(2) F_SETFD operations to set the FD_CLOEXEC flag.

Note that the use of this flag is essential in some multithreaded programs, because using a separate fcntl(2) F_SETFD operation to set the FD_CLOEXEC flag does not suffice to avoid race conditions where one thread opens a file descriptor and attempts to set its close-on-exec flag using fcntl(2) at the same time as another thread does a fork(2) plus execve(2). Depending on the order of execution, the race may lead to the file descriptor returned by open() being unintentionally leaked to the program executed by the child process created by fork(2). (This kind of race is in principle possible for any system call that creates a file descriptor whose close-on-exec flag should be set, and various other Linux system calls provide an equivalent of the O_CLOEXEC flag to deal with this problem.)"

seems better to me.

What for? Use of NtSuspendProcess/NtResumeProcess is usually a smell of trying to do *nix style multiprocessing in Windows. For which the answer usually is: Don't.

Yeah I know, that's a perfect point to start yet another flamewar, and as such I want to add the disclaimer that I'm not making a judgment with this statement. :) It's just that this is by design: You're not supposed to use the kernel API directly in Windows, you are supposed to code against Win32/WinRT/UWP.

(Hmm...I'd hazard a guess there is documentation for these calls, but it's simply not public.)

Something as simple as the CreateProcess and family of function is a complete mess (both in the doc and in the detailed way they work...)

POSIX actually doubles as a reference documentation. I don't think such a thing really exists for Win32 -- MSDN is way too vague to achieve that purpose. Maybe people from Wine / ReactOS maintain a better doc, I don't know.

This is so patently wrong.

Firstly, the first example I thought of (CreateFile) clearly stated that you could get ERROR_FILE_NOT_FOUND, ERROR_ALREADY_EXISTS, ERROR_FILE_EXISTS, ERROR_PIPE_BUSY, ERROR_SHARING_VIOLATION, ERROR_ACCESS_DENIED.

Secondly, drivers can sometimes return error codes that the OS might not expect to be necessary, and the OS can't just coerce the error codes into something else; it'd lose information. [1] For example, if you CreateFile, a driver that mandatorily logs data might decide to return an error saying it's low on disk space even though the file already exists, which you wouldn't expect. If your code isn't specially written to handle it, you should treat it as the general failure condition it is.

Heck, you can even think of Windows as returning "TRUE" (for no error) or "FALSE" (for error), and GetLastError as just providing details you can ignore. Simply the fact that you know the possibilities are TRUE and FALSE doesn't satisfy you though, does it? So isn't the problem that Linux is just suppressing information it could actually pass through?

If anything, Windows is doing this right and Linux is doing it right by being restrictive on the returned information... but in any cases, at best you can suggest they're both "different" and maybe on a good day people would entertain the possibility. Certainly returning MORE information than you need is not something you can call out as a flaw...

[1] e.g. see http://stackoverflow.com/a/19249991

Given how poor the error reporting is under Windows (so poor that useless 32 bits hex error code, component dependant, are pretty much the only (useless) thing that users are seeing), the theory of "more of those" are better is also utterly ridiculous. I've never debugged anything looking at WinNT/Win32 error codes, or even the event log, etc, while I've in plenty of occasion debugged things with Unix errno, strace (which gives errno for failed syscalls) logs in /var/log (often dumping Unix errno + some context) or even just reading stderr.

If you want to understand how much of the getlasterror specs are missing in Win32, just take a look at the Wine unit tests in various are. For each of those cases (maybe 99% of which are undocumented in MSDN), that can correspond to programmers needing to replicate their own qualification beforehand usage of affected functions, or worse to implicitly make some (sometimes false) hypothesis. And the overwhelming majority of them are not related whatsoever to any kind of driver and whatnot, it is just that not even the upper layer errors are properly specified/documented. This is the kind of info which should appear somewhere in MS doc, and which does not. It's insanely ridiculous because it's obvious MS have it in a form or another, given it is crucial for backward compatibility (or compatibility in general, if you take the Wine case -- but I guess when you remain within MS the main issue is backward compat)

I think your lack of experience explains the issue. I do it pretty damn frequently.

> so poor that useless 32 bits hex error code, component dependant, are pretty much the only (useless) thing that users are seeing

All you need to do is look up the error code in either the headers or in MSDN [1] and then read the error message. It explains what's going on and does it far better than the minimal number of errnos possible.

And no, I didn't cherry-pick anything. Like I said, I picked the first function I thought of. If you really cared enough to provide an example you could/would have, but you didn't. And this is ignoring the fact that I then told you why the listing of error codes or lack thereof was not possible in general.

And it's not like in Linux you don't need to write and run and re-write and re-run code fifty times before you know all the edge cases. I remember having one hell of a time trying to figure out how to use epoll, for example. The documentation could easily be 2-3x as long.

Regarding WINE: WINE is not software that uses the Windows API. It replicates the Windows API. Nobody ever claimed Windows's error reporting behavior is easy to replicate. We're talking about APIs, i.e. interfaces for programmers that program TO the system. An open set of error codes makes a ton of sense for all the reasons I listed. Of course an open sense means you'll have a much harder time replicating the behaviors. Windows was not designed to be easy to copy; it was designed to be easy to use. I would've thought that would be obvious.

[1] https://msdn.microsoft.com/en-us/library/windows/desktop/ms6...

As for the openness because of installable FS; WTF? Other systems have (more) installable FS.

As for my lack of experience: WTF bis? I once got an NT error through a Win32 call. That is not even supposed to happen. It was (obviously) not useful to debug. I had to put a workaround in my code.

I know what is Wine. I once read its code to understand some of the ACL api of windows, and what some win functions are supposed to return. You would be surprised what is returned and in which case. MSDN seems to be so poor in this area that I found some example indicating not even some people working at MS know how some functions in this area are supposed to be used...

Does the POSIX documentation have anything like it? I don't recall so remind me if it does.

https://msdn.microsoft.com/en-us/library/windows/desktop/aa3...

Edit: oh, are you referring to me? I see what you mean but it's still beside the point.

What I have done - well, what I think I have done! - the UI is appalling - is added SourceTree.exe and git.exe as exclusions. My intention here is that the antimalware service won't then monitor their activity while they're doing stuff, but will still check working copies when it does a routine scan... but sadly this didn't seem to make any difference at all, literally none whatsoever, and SourceTree continued to be what I can only describe as "slow as fuck" - if you'll forgive the technical jargon.

The odd thing is, though, which just occurred to me like literally right now, it might actually be safer to add the folder exclusion after all! Because then there's zero risk of the Javascript interpreter springing into action :-|

What a time to be alive!

(But I still stand by everything I said about POSIX.)

At this point I've memorized the keys to hit to quickly disable and re-enable defender's real time protection (winkey > "defen" > enter > tab > (down to disable, up to enable) > alt + F4)

https://blog.sourcetreeapp.com/2017/04/27/sourcetree-for-win...