key-value-store | Implemet Key-Value store using BTree | Key Value Database library

You don't need to use .get:

Choosing hash over string has many benefits and some drawbacks depending on the use cases. If you are going to choose hash, it is better to design your json object as hash fields & values such as;

Sure, you do it like this:

There is already sqlitedict which appears to meet all your needs.

From the documentation:

Don't write a template metaprogram, where it is not necessary. Try this simple solution (CTMap stands for compile time map):

A KTable is a logical abstraction of a table that is updated over time. Additionally, you can think of it not as a materialized table, but as a changelog stream that consists of all update records to the table. Compare https://docs.confluent.io/current/streams/concepts.html#duality-of-streams-and-tables. Hence, conceptually a KTable is something hybrid if you wish, however, it's easier to think of it as a table that is updated over time.

Internally, a KTable is implemented using RocksDB and a topic in Kafka. RocksDB stores the current data of the table (note, that RocksDB is not an in-memory store, and can write to disk). At the same time, each update to the KTable (ie, to RocksDB) is written into the corresponding Kafka topic. The Kafka topic is used for fault-tolerance reasons (note, that RocksDB itself is considered ephemeral and writing to disk via RocksDB does not provide fault-tolerance, but the used changelog topic), and is configured with log compaction enabled to make sure that the latest state of RocksDB can be restored by reading from the topic.

If you have a KTable that is created by a windowed aggregation, the Kafka topic is configured with compact,delete to expired old data (ie, old windows) to avoid that the table (ie, RocksDB) grows unbounded.

Instead of RocksDB, you can also use an in-memory store for a KTable that does not write to disk. This store would also have a changelog topic that tracks all updates to the store for fault-tolerance reasons.

If you add a store manually via builder.addStateStore() you can also add RocksDB or in-memory stores. In this case, you can enable changelogging for fault-tolerance similar to a KTable (note, that when a KTable is created, internally, it uses the exact same API -- ie, a KTable is a higher level abstractions hiding some internal details).

For caching: this is implemented within Kafka Streams and on top of a store (either RocksDB or in-memory) and you can enable/disable is for "plain" stores you add manually, of for KTables. Compare https://docs.confluent.io/current/streams/developer-guide/memory-mgmt.html Thus, caching is independent of RocksDB caching.

To your title question: Yes, they do.

To your hypothetical sorted set implementation question: No, you can't.

One, you're mistaken on the implementation of deque; it's not a plain "item per node" linked list, it's a block of items per node (64 on the CPython reference interpreter, though that's an implementation detail). And aside from the head and tail blocks, internal blocks are never left empty, so insertion midway through the deque isn't particularly cheap, it still has to move a bunch of stuff around. It's not O(n) like a mid-list insertion, as it takes advantage of some efficiencies in rotation to rotate, append to one side or the other, then rotate back, but it's a far cry from insertion at a known point in a linked list, remaining O(n) (though with large constant divisors thanks to shuffling whole blocks being cheaper than moving each of the individual items).

Two, each lookup in a deque is O(n), not O(1) like a list; it has a constant divisor of 64 as stated previously, and it's drops to O(1) near either end of the deque, but it's still O(n) in general, which scales poorly for large deques. bisect searches are O(log n) under the assumption that indexing the sequence is O(1); for a deque, they'd be O(n log n), as they'd perform log n O(n) indexing operations. This matches your results from testing; bisect+deque is significantly worse.

Java's TreeMap isn't implemented in terms of binary search and a linked list in any event; linked lists are no good for this, since ultimately a full binary search must traverse back and forth enough that it does O(n) total work, even if it only has to compare against O(log n) elements. A tree map needs a tree structure of some sort, you can't just fake it with a linked list and a good algorithm.

Built-in alternatives include:

1. insort of a normal list: Sure it's O(n) overall, but the expensive part (finding where to insert) is O(log n), and it's only the "make room" step that's O(n), and it's a really cheap O(n) (basically a memcpy). Not acceptable for truly huge lists, but you'd be surprised how large a list you'd need before the overhead was noticeable against Python's slowness.
2. Delayed, buffered sorting: If lookups are infrequent, but insertions are common, defer the sort until needed to minimize the number of sorting operations; just append the new elements to the end without sorting and set a "needs sorting" flag, and re-sort before a lookup when the flag is set. The TimSort algorithm does very well when the input is already mostly sorted (much closer to O(n) than a general purpose sort without optimizations for partially sorted typically can do), so it may be fine.
3. If you only need the smallest element at any given time, the heapq module can do that with true O(log n) insertions and removals, and gets the minimum with O(1) (it's always index 0).
4. Use a sqlite3 database (possible via shelve), indexed as appropriate; sqlite3 indices default to using a B-tree, meaning queries ordered using the index key(s) get the results back in sorted order "for free".

Otherwise, you'll have to install a third-party module that provides a proper sorted set-like type.

I have searched both in the past and in the present for an open-source solution to this problem but I have not found one that satisfies my appetite. This time I decided to start building my own and discuss openly its implementation that also covers the null case, i.e. missing data scenario.

Do notice that secondary index is very close to adjacency list representation, a core element in my TRIADB project and that is the main reason behind searching for a solution.

Let's start with one line code using numpy

But Badger is not about writing "an" entry:

My writes are really slow. Why?

Are you creating a new transaction for every single key update? This will lead to very low throughput.

To get best write performance, batch up multiple writes inside a transaction using single DB.Update() call.
You could also have multiple such DB.Update() calls being made concurrently from multiple goroutines.

That leads to issue 396:

I was looking for fast storage in Go and so my first try was BoltDB. I need a lot of single-write transactions. Bolt was able to do about 240 rq/s.

I just tested Badger and I got a crazy 10k rq/s. I am just baffled

That is because:

LSM tree has an advantage compared to B+ tree when it comes to writes.
Also, values are stored separately in value log files so writes are much faster.

You can read more about the design here.

One of the main point (hard to replicate with simple read/write of files) is:

Key-Value separation

The major performance cost of LSM-trees is the compaction process. During compactions, multiple files are read into memory, sorted, and written back. Sorting is essential for efficient retrieval, for both key lookups and range iterations. With sorting, the key lookups would only require accessing at most one file per level (excluding level zero, where we’d need to check all the files). Iterations would result in sequential access to multiple files.

Each file is of fixed size, to enhance caching. Values tend to be larger than keys. When you store values along with the keys, the amount of data that needs to be compacted grows significantly.

In Badger, only a pointer to the value in the value log is stored alongside the key. Badger employs delta encoding for keys to reduce the effective size even further. Assuming 16 bytes per key and 16 bytes per value pointer, a single 64MB file can store two million key-value pairs.