concurrentqueue | fast multi-producer , multi-consumer lock | Architecture library

First, it seems you are clogging the event handling thread. So, what I'd do is decouple event handling from the actual processing:

( Untested! Just an outline!)

REMOVED FAULTY CODE

Then in serviceInstance1, I would have Publish enqueue the orders in a BlockingCollection, on which a dedicated Thread is waiting. That thread will do the actual send. So you'll marshall the orders to that thread regardless of what you chose to do in Processor and all will be decoupled and in-order.

You probably will want to set BlockOptions according to your requirements.

Mind that this is just a coarse outline, not a complete solution. You may also want to go from there and minimize string-operations etc.

Some more thoughts that came to me since yesterday in no particular order:

• May it be beneficial to ditch the first filter to filter out empty sets of JObjects later?
• Maybe it's worth trying to use System.Text.Json instead of Newtonsoft?
• Is there a more efficient way to get from xml to json? (I was thinking "XSLT" but really not sure)
  • convert xml to json using xslt
  • xml to json using xslt
• I'd recommend to rig up a Benchmark.Net with MemoryAnalyzer to document / proove your changes have positive effects.
• Don't forget to have a look into DataFlowBockOptions to tweak the pipeline's behavior.

• DataFlow
• Walkthrough: Creating a Dataflow Pipeline
• What's so wrong about using GC.Collect()?

You need to use something that enable you to query the stream, SQL is not the best tool for it. Search for Complex Event Processing and Kafka / Event hub + Stream Analytics.

I've numbered the block in your question, so I can reference them here:

Block 1 and 3 are both running on a serial queue, thus block 3 will only run once 1 is done.

However, block 1 and 3 don't actually wait for task1/2, they just queue off work to happen asynchronously in blocks 2 and 4, which finishes near instantly.

From then on, both task 1 and 2 will be running concurrently, and finish in an arbitrary order. The only guarantee is that task1 will start before task2.

I always like to use the analogy of ordering a pizza vs making a pizza. Queuing async work is like ordering a pizza. It doesn't mean you have a pizza ready immediately, and you're not going to be blocked from doing other things while the pizzeria is baking your pizza.

Your blocks 1 and 3 are strongly ordered, so 1 will finish and finish before 3 starts. However, all the block does is order a pizza, and that's fast. It does mean pizza 1 (task 1) is done before pizza 2 (task 2), it just means you got off the first phone call before making the second.

This is indeed due to the lack of RAM on the free tier VM. Binary packages would indeed solve this. But will see whether we can do something about that as well.

Your current queue is concurrent

There are many ways to do this. Here is a TPL DataFlow Example

DataFlow has a few advantages

1. It can deal with both synchronous and asynchronous workloads.
2. You can create larger pipelines.
3. Supports Task Schedulers, and cancellations tokens.
4. Can run perpetually, or force to complete.
5. Each block can support multiple producers and consumers

It does have some disadvantages through

1. It's a bit of learning curve for the uninitiated.
2. It's designed around a pipeline, not linear collections per se, so using them can be a little unintuitive.
3. Creating your own custom blocks will require a deep dive in Stephen Toub's Twisted TPL brain.
4. It's not as light as other producer consumer frameworks, however it makes up for it with flexibility

Example

I see one error in your code, you're sleeping a background thread after every insert, don't sleep if there's more data. Instead of:

While Azure Functions may typically share a single backplane, it is not guaranteed. Resources can be spun down or up at any time and new copies of functions may not have access to the original state. As a result, if you use static fields to share data across function executions, it should be able to reload the data from an external source.

That said, this is also not necessarily preferable due to how Azure Functions are designed to be used. Azure Functions enable high-throughput via dynamic scalability. As more resources are needed to process the current workload, they can be provisioned automatically to keep throughput high.

As a result, doing too much work in a single function execution can actually interfere with overall system throughput, since there is no way for the function backplane to provision additional workers to handle the load.

If you need to preserve state, use a form of permanent storage. This could take the form of a Azure Durable Function, an Azure Storage Queue, an Azure Service Bus queue, or even a database. In addition, in order to best take advantage of your function's scalability, try to reduce the workload to manageable batches that allow for large amounts of parallel processing. While you may need to frontload your work in a single operation, you want the subsequent processing to be more granular where possible.

You called with with sync, so the call will wait until the block is scheduled and completes. The queue is inactive, so it cannot schedule blocks. Therefore, the block can't complete, and the sync can't return. Is there a different behavior you're expecting from sync?

This construct is useful if you want all the processes to wait for some condition before starting. For example, you might make an inactive queue that guards access to something that needs to initialize (logging-in for example, or reading configuration from disk). Once that has initialized, it can call .activate(), and all of these other processes will start. If the system is already initialized, the .sync {} call will return immediately.