fifo | A simple redis based task queue for python | BPM library

Suppose we have a situation where we need FIFO data structure. For example, consume some events in the order they came in.

Additionally, we need to clear the entire queue from time to time.

std::queue seems like the perfect fit for doing that, but unfortunately it lacks a function for clearing the container.

So at this point, we have 2 alternatives:

std::queue

• we asked the STL lib what we need. Granted, the STL lib will give us more: it will give us an std::deque disguised as a std::queue
• we got back only a part from what we need, namely the pop front and push back but without clear
• we will have to "emulate" clear somehow, without the naive way of looping and popping

std::deque

• we asked the STL lib what we need
• we got back what we asked for, but we've got too much: we also got push front and pop back

Overall, we either received too few or too much, never exactly what we really wanted.

Here is the thing that took me by surprise, while I was trying to provide clear functionality for using with std::queue which is a member var of my object

I have a data structure containing a set of std::pair. There are two important properties I require for this data structure:

• I can check set membership fast.
• FIFO

So as a C++ beginner armed with cppreference.com I went for std::queue> frontierPointsUV{}; where I have typedef std::pair PointUV; and I include the implementation of PointUVHash in the appendix below.

My questions are

1. Is this a sensible way to meet the 2 requirements above?
2. How do I check set membership? I've tried frontierPointsUV.c.contains for set membership but I get "has no member" errors.
3. How to I push and pop (or insert and erase)? I've been unsuccessful with trying to call these modifiers on either of the containers.

Appendix - Hash implementation

I am running into the following error when starting up containers on my Raspberry Pi 3B on Raspbian Buster:

I'm trying to calculate how many days the stock for an item has been sitting at a site.

There are two tables: Stock table shows the items and stock currently on hand and Receipts table show the dates when the site has received stock and quantity.

I want to do a left outer join to see all the items in the Stock table and only the rows from the Receipts table with the date where there is still stock left from.

Stock

I am having a lot of issues handling concurrent runs of a StateMachine (Step Function) that does have a GlueJob task in it.

The state machine is initiated by a Lambda that gets trigger by a FIFO SQS queue.

The lambda gets the message, checks how many of state machine instances are running and if this number is below the GlueJob concurrent runs threshold, it starts the State Machine.

The problem I am having is that this check fails most of the time. The state machine starts although there is not enough concurrency available for my GlueJob. Obviously, the message the SQS queue passes to lambda gets processed, so if the state machine fails for this reason, that message is gone forever (unless I catch the exception and send back a new message to the queue).

I believe this behavior is due to the speed messages gets processed by my lambda (although it's a FIFO queue, so 1 message at a time), and the fact that my checker cannot keep up.

I have implemented some time.sleep() here and there to see if things get better, but no substantial improvement.

I would like to ask you if you have ever had issues like this one and how you got them programmatically solved.

Thanks in advance!

This is my checker:

Id like to ask, is safe having a variant like this?

I'm looking for ways to count the number of trailing newlines from possibly binary data either:

• read from standard input
• or already in a shell variable (then of course the "binary" excludes at least 0x0) using POSIX or coreutils utilities or maybe Perl.

This should work without temporary files or FIFOs.

When the input is in a shell variable, I already have the following (possibly ugly but) working solution:

I'm revisiting the STM chapter of Marlow's book. There, it is stated that:

When multiple threads block on an MVar, they are guaranteed to be woken up in FIFO order

However, the same can't be done on the STM case:

A transaction can block on an arbitrary condition, so the runtime doesn't know whether any individual transaction will be able to make progress after the TVar is changed; it must run the transaction to find out. Hence, when there are multiple transactions that might be unblocked, we have to run them all; after all, they might all be able to continue now.

What I don't get is why from this it follows that

Because the runtime has to run all the blocked transactions, there is no guarantee that threads will be unblocked in FIFO order ...

I'd expect that even though we have to run all the transactions in an STM block, we can still wake the threads up in a FIFO order. So I guess I'm missing some important details.

I have a A table to store the product name, the quantity and input day of goods in stock. In adverse conditions, invoices for the above products will be sent later. I will put the product name and invoice quantity in B table. The problem here is that I want to check the quantity of goods with invoice and without invoice. Updating goods with invoices will follow FIFO.

Example:

Table A

Table B

I solved it by creating a new C table with the same data as the A table

Table C

Then I get the data in table B group by good_id and store it in $data. Using php to foreach $data and check condition:

I updated C table ORDER BY created_at DESC limit 1 as follows:

• if (tableC.current_num - $data['num'] < 0) then update current_number = 0, invoice_number = $data['num'] - tableC.current_num. Update value $data['num'] = $data['num'] - tableC.current_num

• if (tableC.current_num - $data['num'] > 0) or (tableC.current_num - $data['num'] = 0) then update current_number = tableC.current_num - $data['num'], invoice_number = $data['num'].

table C after update

I solved the problem with php like that. However, with a dataset of about 100,000 rows, I think the backend processing will take a long time. Can someone give me a smarter way to handle this?