I admit not to being entirely up on the Windows way of doing these things, but I imagine the primitive you want if you're waiting on single flags only is NSConditionLock. Each condition lock has a particular condition, threads can attempt to lock it with no regard for the condition or only when it has a particular condition, optionally with a timeout for both. They can then unlock and, optionally, set a new condition when they do so.
Possibly a more straightforward approach is to create your NSThreads manually rather than just offloading operations into an NSOperationQueue. Each NSThread automatically has an NSRunloop so you can then use semantics like:
[object performSelector:@selector(operation:) onThread:targetThread withObject:someArgumentForOperation waitUntilDone:NO];
In which case the method 'operation:' will be called with the nominated argument on the nominated thread as soon as an opportunity arises, and the calling thread isn't blocked. Runloops solve the same problem as the classic win32 message dispatch mechanisms but invert responsibility — Cocoa deals with blocking threads, waking upon messages and issuing the appropriate function calls.