superloop | network automation framework , used to manage networks | Automation library

Your predicament is a perfect fit for state machines (really, probably UML statecharts). Each different request can each be handled in its own state machine, which handle events (such as COMPLETE or ERROR indications) in a non-blocking, run-to-completion manner. As the events come in, the request's state machine moves through its different states towards completion.

For embedded systems, I often use the QP event-driven framework for such cases. In fact, when I looked up this link, I noticed the very first paragraph uses the term "non-blocking". The framework provides much more than state machines with hierarchy (states within states), which is already very powerful.

The site also has some good information on approaches to your specific problem. I would suggest starting with the site's Key Concepts page.

To get you a taste of the content and its relevance to your predicament:

In spite of the fundamental event-driven nature, most embedded systems are traditionally programmed in a sequential manner, where a program hard-codes the expected sequence of events by waiting for the specific events in various places in the execution path. This explicit waiting for events is implemented either by busy-polling or blocking on a time-delay, etc.

The sequential paradigm works well for sequential problems, where the expected sequence of events can be hard-coded in the sequential code. Trouble is that most real-life systems are not sequential, meaning that the system must handle many equally valid event sequences. The fundamental problem is that while a sequential program is waiting for one kind of event (e.g., timeout event after a time delay) it is not doing anything else and is not responsive to other events (e.g., a button press).

For this and other reasons, experts in concurrent programming have learned to be very careful with various blocking mechanisms of an RTOS, because they often lead to programs that are unresponsive, difficult to reason about, and unsafe. Instead, experts recommend [...] event-driven programming.

You can also do state machines yourself without using an event-driven framework like the QP, but you will end up re-inventing the wheel IMO.