python-forexconnect | Python binding of forexconnect api | REST library

The following code below is simulating a producer and consumer model, which will collect data from a forex broker FXCM and write to a database. Each producer process will have a session based connection with the broker.

Both producer and consumer will run indefinitely until a 'poison pill' is placed into the queue, which happens at the close of business (Friday 22:00). I have omitted this part of the code as it is irrelevant to the problem. All of the examples I can find seem to spawn a process, do some work for a short period and then join() back to the parent process. Like this one here

As mentioned above the producer will run indefinitely, and the reason for this is because it takes about 3 seconds to log in and create a session with the broker.

When running the code below you will see the Queue backlog, although this seems to be even worst when running the real code.

Not sure if it is relevant but the session is created using the python-forexconnect API, which is written in C++ and uses Boost.

The problem is the consumer is taking too long to get() items from the queue, and I am wondering if this model is the correct way to approach this type of development.

Thank-you

Sample code