semaphore | Modern UI for Ansible | Continous Integration library

Your spin loop while (data->locked[i]); is a data race; you don't hold the lock while reading it data->locked[i], and so another thread could take the lock and write to that same variable while you are reading it. In fact, you rely on that happening. But this is undefined behavior.

Immediate practical consequences are that the compiler can delete the test (since in the absence of a data race, data->locked[i] could not change between iterations), or delete the loop altogether (since it's now an infinite loop, and nontrivial infinite loops are UB). Of course other undesired outcomes are also possible.

So you have to hold the mutex while testing the flag. If it's false, you should then hold the mutex until you set it true and do your other work; otherwise there is a race where another thread could get it first. If it's true, then drop the mutex, wait a little while, take it again, and retry.

(How long is a "little while", and what work you choose to do in between, are probably things you should test. Depending on what kind of fairness algorithms your pthread implementation uses, you might run into situations where take_forks succeeds in retaking the lock even if put_forks is also waiting to lock it.)

Of course, in a "real" program, you wouldn't do it this way in the first place; you'd use a condition variable.

If it's not the bandwidth that limits you (but I cannot check this), there is a solution less complicated than the celery and rabbitmq but it is not as scalable as the celery and rabbitmq, it will be limited by your number of CPU.

Instead of splitting calls on celery workers, you split them on multiple processes.

I modified the fetch function like this:

The key is found in the manpage (which you linked):

If O_CREAT is specified in oflag, then two additional arguments must be supplied.

That's exactly the way variadic functions work in C. A variadic function has a fixed number of parameters, each with a well-defined type, and then a ..., which can be any number of arguments of any type, with the proviso that the called function has to be able to figure out the type of each argument in order to reference it. printf is indeed the clearest example: the format string includes enough information for printf to know exactly how many and what type of arguments to expect.

Note that guessing is not permitted. If the function asks for an argument which wasn't provided or asks for a different type than the provided argument, it doesn't get a friendly error indication or a second chance. The result is Undefined Behaviour: anything can happen, including abrupt termination or a nonsensical result. Or a result which seems to make sense but has no basis in reality.

Like open, sem_open uses this to allow arguments which are only needed in certain well-defined circumstances. If it is possible that the call will create a new semaphore (or file, in the case of open), it will request the additional arguments, assuming that they are of the types specified in the documentation.

The caller is responsible for ensuring that the call is correct. Note that since variadic arguments do not have a declared type, the compiler cannot insert type conversions. So if the function is expecting a double, it cannot be called with an int, which would normally be possible. Also, if a variadic argument is supposed to be void*, it must be void*, and not, for example, 0 (or NULL), because the automatic conversion from 0 to a null pointer can't be done.

However, the caller is allowed to provide too many arguments, or conversely the called function is not required to examine every argument supplied. (It can't skip over arguments, but it doesn't need to get to the end of the list.)

In retrospect, this may seem like a terrible language feature. But it seemed like a good idea at the time, and it's not going away now.

How can I use golang.org/x/sync/errgroup to wait on and handle errors from goroutines or if there is any better way like using semaphore? For example [...] I want to cancels all those remaining in the case of any one routine returning an error (in which case that error is the one bubble back up to the caller). And it should automatically waits for all the supplied go routines to complete successfully for success case.

There are many ways to communicate error states across goroutines. errgroup does a bunch of heavy lifting though, and is appropriate for this case. Otherwise you're going to end up implementing the same thing.

To use errgroup we'll need to handle errors (and for your demo, generate some). In addition, to cancel existing goroutines, we'll use a context from errgroup.NewWithContext.

From the errgroup reference,

Package errgroup provides synchronization, error propagation, and Context cancelation for groups of goroutines working on subtasks of a common task.

Your play doesn't support any error handling. We can't collect and cancel on errors if we don't do any error handling. So I added some code to inject error handling:

The xEventGroupSync() API seems most appropriate.

If I'm understanding your requirements then something like this will work. It effectively combines the iter unordered with a channel to allow a mapping instead.

You are combining two different but related concepts; asynchronous and concurrency.

An OperationQueue always dispatches Operations onto a separate thread so you do not need to make them explicitly make them asynchronous and there is no need to override start(). You should ensure that your main() does not return until the operation is complete. This means blocking if you perform asynchronous tasks such as network operations.

It is possible to execute an Operation directly. In the case where you want concurrent execution of those operations you need to make them asynchronous. It is in this situation that you would override start()

If you want to implement a concurrent operation—that is, one that runs asynchronously with respect to the calling thread—you must write additional code to start the operation asynchronously. For example, you might spawn a separate thread, call an asynchronous system function, or do anything else to ensure that the start method starts the task and returns immediately and, in all likelihood, before the task is finished.

Most developers should never need to implement concurrent operation objects. If you always add your operations to an operation queue, you do not need to implement concurrent operations. When you submit a nonconcurrent operation to an operation queue, the queue itself creates a thread on which to run your operation. Thus, adding a nonconcurrent operation to an operation queue still results in the asynchronous execution of your operation object code. The ability to define concurrent operations is only necessary in cases where you need to execute the operation asynchronously without adding it to an operation queue.

In summary, make sure your operations are synchronous and do not override start if you want to take advantage of progress

Update

While the normal advice is not to try and make asynchronous tasks synchronous, in this case it is the only thing you can do if you want to take advantage of progress. The problem is that if you have an asynchronous operation, the queue cannot tell when it is actually complete. If the queue can't tell when an operation is complete then it can't update progress accurately for that operation.

You do need to consider the impact on the thread pool of doing this.

The alternative is not to use the inbuilt progress feature and create your own property that you update from your tasks.

The Rx does not support backpressure, so there is no easy way to fetch the records from the DB at the same tempo that the records are processed. Maybe you could use a Subject as a signaling mechanism, push a value every time a record is processed, and devise a way to use these signals at the producing site to fetch a new record from the DB when a signal is received. But it will be a messy and idiomatic solution. The TPL Dataflow is a more suitable tool than the Rx for doing this kind of work. It supports natively the BoundedCapacity configuration option.

Some comments regarding the code you've posted, that are not directly related to the backpressure issue:

The Merge operator with a maxConcurrent parameter imposes a limit on the concurrent subscriptions to the inner sequences, but this will have no effect in case the inner sequences are already up and running. So you have to ensure that the inner sequences are cold, and a handy way to do this is the Defer operator:

You have stumbled into a feature/bug in DispatchSemaphore. If you look at the stack trace and jump to the top of the stack, you'll see assembly with a message:

BUG IN CLIENT OF LIBDISPATCH: Semaphore object deallocated while in use

E.g.,

This is because DispatchSemaphore checks to see whether the semaphore’s associated value is less at deinit than at init, and if so, it fails. In short, if the value is less, libDispatch concludes that the semaphore is still being used.

This may appear to be overly conservative, as this generally happens if the client was sloppy, not because there is necessarily some serious problem. And it would be nice if it issued a meaningful exception message rather forcing us to dig through stack traces. But that is how libDispatch works, and we have to live with it.

All of that having been said, there are three possible solutions:

1. You obviously have a path of execution where you are waiting and not reaching the signal before the object is being deallocated. Change the code so that this cannot happen and your problem goes away.

2. While you should just make sure that wait and signal calls are balanced (fixing the source of the problem), you can use the approach in your question (to address the symptoms of the problem). But that deinit approach solves the problem through the use of non-local reasoning. If you change the initialization, so the value is, for example, five, you or some future programmer have to remember to also go to deinit and insert four more signal calls.

   The other approach is to instantiate the semaphore with a value of zero and then, during initialization, just signal enough times to get the value up to where you want it. Then you won’t have this problem. This keeps the resolution of the problem localized in initialization rather than trying to have to remember to adjust deinit every time you change the non-zero value during initialization.

   See https://lists.apple.com/archives/cocoa-dev/2014/Apr/msg00483.html.

3. Itai enumerated a number of reasons that one should not use semaphores at all. There are lots of other reasons, too:

   • Semaphores are incompatible with new Swift concurrency system (see Swift concurrency: Behind the scenes);
   • Semaphores can also easily introduce deadlocks if not precise in one’s code;
   • Semaphores are generally antithetical to cancellable asynchronous routines; etc.

   Nowadays, semaphores are almost always the wrong solution. If you tell us what problem you are trying to solve with the semaphore, we might be able to recommend other, better, solutions.

You said:

However, if the async function returns before deinit is called and the view controller is deinitialized, then signal() is called twice, which doesn't seem problematic. But is it safe and/or wise to do this?

Technically speaking, over-signaling does not introduce new problems, so you don't really have to worry about that. But this “just in case” over-signaling does have a hint of code smell about it. It tells you that you have cases where you are waiting but never reaching signaling, which suggests a logic error (see point 1 above).