rocksdb | Optimize rocksdb compaction

Getting this error while building docker images on Mac OS BigSur with M1 chip.

What I've tried: Installed docker for Apple Silicon Graphic M1 from docker site

It fails while trying to install RocksDB from Docker

I'm querying whether a key exists in a RocksDB database using Python. The API (see bottom) implies it returns a two-element tuple. So I receive both tuple elements:

I'm trying to finish some applications that use RocksDB state backend in the incremental mode and I want to keep a savepoint to start use in the next execution. Whenever I try do finish, this error shows.

Errors:

We have a kstreams app doing kstream-kstable inner join. Both the topics are high volume with 256 partitions each. kstreams App is deployed on 8 nodes with 8 GB heap each right now. We see that the heap memory keeps constantly growing and eventually OOM happens. I am not able to get the heap dump as its running in a container which gets killed when that happens. But, I have tried a few things to gain confidence that it is related to the state stores/ktable related stuff. Without the below RocksDBConfigSetter the memory gets used up pretty quick, but with the below it is slowed down to some extent. Need some guidance to proceed further , thanks

I added below 3 properties,

I'm reading 2 kafka topics in the same flink jobs.

• Stream1: Messages comes from the first topic are saved to rocksdb, then it will union with stream2.
• Stream2: Messages comes from the second topic are enriched with state saved by stream1, then it will union with the stream1.

Topic1 and topic 2 are different sources but basically the output is the same for two sources. I have to just enrich data come from topic2 with the data come from topic1.

Here is flow;

I want to try "Runtime Upgrade" and "Storage Migration". I tried by steps as following,but "on_runtime_upgrade" function did not work. Unable to verify that the "on_runtime_upgrade" function is called. The implemented debug log is not output.

1. Download "substrate-node-template 3.0".
2. Compile it.
3. I ran the node with the following command. "target/release/node-template --dev -l runtime = debug".
4. Implement "on_runtime_upgrade" in "palet-template".The program I tried "substrate-node-template/pallets/template/lib.rs" is listed below.
5. Compile it by "cargo build --release -p node-template-runtime" command.
6. Upload "node_template_runtime.compact.wasm" by using "sudo" & "setcode" command.
7. I checked the execution log of the node, but I couldn't check the log set to "on_runtime_upgrade".

--lib.rs--

I have an architecture question regarding the union of more than two streams in Apache Flink.

We are having three and sometime more streams that are some kind of code book with whom we have to enrich main stream. Code book streams are compacted Kafka topics. Code books are something that doesn't change so often, eg currency. Main stream is a fast event stream. Our goal is to enrich main stream with code books.

There are three possible ways as I see it to do it:

1. Make a union of all code books and then join it with main stream and store the enrichment data as managed, keyed state (so when compact events from kafka expire I have the codebooks saved in state). This is now only way that I tired to do it. Deserilized Kafka topic messages which are in JSON to POJOs eg. Currency, OrganizationUnit and so on. I made one big wrapper class CodebookData with all code books eg:

I am experimenting with my new Flink cluster(3 Different Machines-> 1 Job Manager, 2-> Task Managers) using RocksDB as State Backend however the checkpointing behaviour I am getting is a little confusing.

More specifically, I have designed a simple WordCount example and my data source is netcat. When I submit my job, the job manager assigns it to a random task manager(no replication as well). I provide some words and then I kill the currenlty running task manager. After a while, the job restarts in the other task manager and I can provide some new words. The confusing part is that state from the first task manager is preserved even when I have killed it.

To my understanding, RocksDB maintains its state in a local directory of the running task manager, so what I expected was when the first task manager was killed to lose the entire state and start counting words from the beginning. So Flink seems to somehow maintain its state in the memory(?) or broadcasts it through JobManager?

Am I missing something?

I´m trying to load a .csv in to my rocksdb database, but it fails and show me this error:

Got error 10 'Operation aborted:Failed to acquire lock due to rocksdb_max_row_locks limit' from ROCKSDB

I've tried with SET SESSION rocksdb_max_row_locks=1073741824; but same error always.

Anyone can help me?