async-task | Symfony2 Bundle builded on top of RabbitMq | Pub Sub library

You specified core size of 50 and max size of 200. So your pool will normally run with 50 threads, and when there is extra work, it will spawn additional threads, you'll see "async-task-exec-51", "async-task-exec-52" created and so on. Later, if there is not enough work for all the threads, the pool will kill some threads to get back to just 50. So it may kill thread "async-task-exec-52". The next time it has too much work for 50 threads, it will create a new thread "async-task-exec-53".

So the fact that you see "async-task-exec-7200" means that over the life time of the thread pool it has created 7200 threads, but it will still never have more than the max of 200 running at the same time.

If @Async method is waiting 10 minutes for a thread it means that you have put so much work into the pool that it has already spawned all 200 threads and they are processing, and you have filled up the queue capacity of 100, so now the parent thread has to block(wait) until there is at least a spot in the queue to put the task.

If you need to consistently handle more tasks, you will need a powerful enough machine and enough max threads in the pool. But if your work load is just very spiky, and you don't want to spend on a bigger machine and you are ok with tasks waiting longer sometimes, you might be able to get away with just raising your queue-capacity, so the work will queue up and eventually your threads might catch up (if the task creation gets slower).

Keep trying combinations of these settings to see what will be right for your workload.

You can avoid this by initializing master at the declaration site.

The easiest way is using the default keyword.

OK, figured out. The deadlock is becuase of SynchronizationContext and it's capturing as stated many times in comments to your question. Usually solution is either use await all the way up, or ConfigureAwait(false), which you doing. BUT!

As stated here (section The Blocking Hack): https://docs.microsoft.com/en-us/archive/msdn-magazine/2015/july/async-programming-brownfield-async-development

If somewhere down the line Microsoft/EF uses conversion from old event-based async pattern then ConfigureAwait(false) will have no effect, cause context will be already captured anyway.

And turns out they do such conversion, here is the source of SqlCommand for SqlServer:

https://github.com/dotnet/SqlClient/blob/bd89b6433892214eed41e97afb0a9430d11d4681/src/Microsoft.Data.SqlClient/netfx/src/Microsoft/Data/SqlClient/SqlCommand.cs

Look there for line 2996.

Now why SqlLite is working fine - because SqlLite doesn't not support async I/O, so all asyncs for SqlLite are fake, you could clearly see it in source of SqliteCommand:

https://github.com/dotnet/efcore/blob/main/src/Microsoft.Data.Sqlite.Core/SqliteCommand.cs

Line 406.

And explanation for "fake" is here - https://docs.microsoft.com/en-us/dotnet/standard/data/sqlite/async

The problem is that you are doing multiple sequenced asynchronous operations (upload to Storage and adding data to firestore) in a synchronous way in your loop and that causes the following behavior (example):

The second upload to Storage triggers before the first operation to Firestore has been completed, and this causes the fileName variable that is being used for the Firestore operation to be populated with value of the next upload.

My suggestion to fix the issue you are facing would be to either not do the operations in sequence, eg. upload everything into storage and only then add everything to firestore, or populate the part of your map that depends on the filename variable right after it's value has been populated and not in the onSuccess handler.

I'm not sure this is the best way to accomplish what I am trying to accomplish but it seems to work.

Config:

Raymond Chen posted about this in his blog The Old New Thing in the post C++/WinRT envy: Bringing thread switching tasks to C# (WPF and WinForms edition).

Reproduced here in case the source goes down:

You can utilize .GetAwaiter().GetResult()

as per your example, it would look like:

If your custom RestTemplate looks exactly like what you put in your sample code, it is of zero utility - you can just use the standard RestTemplate. Also, since you pass in a (presumably) non-request scoped RestTemplate to the constructor, it is not even doing what you think it is doing I suspect. If you really wanted it to be request-scoped and auto-proxied, you can get the same effect without subclassing it. For example:

To solve the initial problem, is you should be using await Task.WhenAll not Task.WaitAll

Task.WhenAll

Creates a task that will complete when all of the supplied tasks have completed.

However, this looks like more of a job for Parallel.ForEach

Another issue is you looping over the same list twice (nested) which is a quadratic time complexity and is definitely not thread safe

As a solve, you could create a dictionary of the changes, loop over the change set once (in parallel), and update the references in one go.