cdc | mysql binlog parser into rabbitmq | Pub Sub library

I am trying to get the test code of the pinata-party working (https://medium.com/pinata/how-to-create-nfts-like-nba-top-shot-with-flow-and-ipfs-701296944bf).

It works fine to the point that I try and send a transaction:

flow transactions send --code "./transactions/MintPinataParty.cdc" --signer emulator-account

When I send that I get the error:

❌ Transaction Error execution error code 1006: [Error Code: 1006] invalid proposal key: public key 0 on account f8d6e0586b0a20c7 does not have a valid signature: [Error Code: 1009] invalid envelope key: public key 0 on account f8d6e0586b0a20c7 does not have a valid signature: signature is not valid

Anyone have any idea where this is coming from?

Thanks.

I am using SQL Server RDS as the source database and Apache-Kafka as the target in AWS DMS. I want to receive both the data and control records on every CDC changes made in the source database but I am only getting data records in case of CRUD commands and control record in case of the DDL commands. I went through the AWS DMS documentation but couldn't find anything relevant.

Is it possible to get both the control and data records in the Kafka topic?

I have a request to store the date on which a specific field was changed in a table. For example, In my dbo.User table, we need to know when the IsActive flag was changed. With history.

I am proposing this:

1. New schema - History.

2. New table - [History].User_History

I'm having trouble in setting up a task migrating the data in a RDS Database (PostgreSQL, engine 10.15) into an S3 bucket in the initial migration + CDC mode. Both endpoints are configured and tested successfully. I have created the task twice, both times it ran a couple of hours at most, the first time the initial dump went fine and some of the incremental dumps took place as well, the second time only the initial dump finished and no incremental dump was performed before the task failed.

The error message is now:

I'm trying to create a mview in Oracle. It would be for a report I run everyday, so I would just need it updated before the execution, on demand.

I'm getting an error when a query consuming a stream is being executed by a task. The error only appears when the query is being executed via a task.

In querying information_schema.task_history, I can see the task status is FAILED with error code 091111. I haven't been able to find any documentation on error codes so I'm mostly relying on the error message Stream my_stream not found.

The stream is being created with SHOW_INITIAL_ROWS parameter set to TRUE. This is because the source table has existed for quite some time and I would like the task to handle the past data in addition to incoming data.

What I've Noticed

SYSTEM$STREAM_HAS_DATA returns False until a new CDC becomes apparent. Since SHOW_INITIAL_ROWS is set to TRUE, when I query the stream I get the same number of rows returned as when I query the table itself. However, SYSTEM$STREAM_HAS_DATA still returns False.

What I've Tried

1. Can query the stream

I've confirmed the task owner has access to the stream by using this role and querying.

SELECT * FROM my_stream LIMIT 5; -- Works.

This confirms that the stream does in fact exist.

1. Executing an UPDATE command does make SYSTEM$STREAM_HAS_DATA return TRUE with all the rows (and not just the diff from this one command).

2. I can run the SQL of the task itself. Going into the History page, I can copy and paste the query and run it.

This confirms the query itself works.

1. Subsequent changes are in fact handled by the task.

Where I Need Help

• I need the task to handle the stream without manual intervention and execution to make the stream looks like it exists

I'm assuming that by executing the query manually, something is happening behind the scene that makes this stream accessible. An example of this would be a stream being created on at able by its owner enables change tracking on that table. However, I've been unable to find what would be causing a scenario where a stream is unfindable until queried.

Update: Step by Step Instructions to Reproduce Bug

Ran into the bug where it's much easier to see what's going on. From there, was able to come up with step by step instructions to reproduce the bug.

First, without show_initial_rows

I try to use a dll which is written in C++. It has this function:

Say I'm working with data with hierarchical indices:

Public CDC Data

The goal is to have those hierarchical indices represented in a pandas dataframe and grouped.

This is as close as I've gotten

I am using Visual C++ 2019 with MFC, on Windows 10 Home Premium. The video mode is 3840*2160 40-60 Hz (AMD FreeSync) 30 bit/pixel: 10 bit / color part, 1 073 741 824 colors.

I can give colors with COLORREF = unsigned int (32 bits), what is interpreted as (red | (green << 8) | (blue << 16)), this has only 16 777 216 colors. How can I give 1 073 741 824 colors? At now, 16M colors is converted to 1G colors. I need the method without conversion.

For example: CDC::SetPixel, FillSolidRect, SetTextColor, SetBkColor, Line, CPen/CBrush constructor, CBitmap, etc. Thank you.

(I want to save the time of conversion. For example, speed of key Page Up/Down in my own IDE is about 10 Hz, what is very slow. I generated the picture in the main memory (with CreateCompatibleBitmap, CreateCompatibleDC, BitBlt). When I drawed directly to the display, this was more slower. I tried SetBkMode OPAQUE and TRANSPARENT so, TextOut and DrawText so.)