berkeleydb | Berkeley DB bindings for go

In a Perl script of mine, I have to write a mix of UTf-8 and raw bytes into files.

I have a big string in which everything is encoded as UTF-8. In that "source" string, UTF-8 characters are just like they should be (that is, UTF-8-valid byte sequences), while the "raw bytes" have been stored as if they were codepoints of the value held by the raw byte. So, in the source string, a "raw" byte of 0x50 would be stored as one 0x50 byte; whereas a "raw" byte of 0xff would be stored as a 0xc3 0xbf two-byte utf-8-valid sequence. When I write these "raw" bytes back, I need to put them back to single-byte form.

I have other data structures allowing me to know which parts of the string represent what kind of data. A list of fields, types, lengths, etc.

When writing in a plain file, I write each field in turn, either directly (if it's UTF-8) or by encoding its value to ISO-8859-1 if it's meant to be raw bytes. It works perfectly.

Now, in some cases, I need to write the value not directly to a file, but as a record of a BerkeleyDB (Btree, but that's mostly irrelevant) database. To do that, I need to write ALL the values that compose my record, in a single write operation. Which means that I need to have a scalar that holds a mix of UTF-8 and raw bytes.

Example:

Input Scalar (all hex values): 61 C3 8B 00 C3 BF

Expected Output Format: 2 UTF-8 characters, then 2 raw bytes.

Expected Output: 61 C3 8B 00 FF

At first, I created a string by concatenating the same values I was writing to my file from an empty string. And I tried writing that very string to a "standard" file without adding encoding. I got '?' characters instead of all my raw bytes over 0x7f (because, obviously, Perl decided to consider my string to be UTF-8).

Then, to try and tell Perl that it was already encoded, and to "please not try to be smart about it", I tried to encode the UTF-8 parts into "UTF-8", encode the binary parts into "ISO-8859-1", and concatenate everything. Then I wrote it. This time, the bytes looked perfect, but the parts which were already UTF-8 had been "double-encoded", that is, each byte of a multi-byte character had been seen as its codepoint...

I thought Perl wasn't supposed to re-encode "internal" UTF-8 into "encoded" UTF-8, if it was internally marked as UTF-8. The string holding all the values in UTF-8 comes from a C API, which sets the UTF-8 marker (or is supposed to, at the very least), to let Perl know it is already decoded.

Any idea about what I did miss there?

Is there a way to tell Perl what I want to do is just put a bunch of bytes one after another, and to please not try to interpret them in any way? The file I write to is opened as ">:raw" for that very reason, but I guess I need a way to specify that a given scalar is "raw" too?

Epilogue: I found the cause of the problem. The $bigInputString was supposed to be entirely composed of UTF-8 encoded data. But it did contain raw bytes with big values, because of a bug in C (turns out a "char" (not "unsigned char") is best tested with bitwise operators, instead of a " > 127"... ahem). So, "big" bytes weren't split into a two-bytes UTF-8 sequence, in the C API.

Which means the $bigInputString, created from the bad C data, didn't have the expected contents, and Perl rightfully didn't like it either.

After I corrected the bug, the string correctly encoded to UTF-8 (for the parts I wanted to keep as UTF-8) or LATIN-1 (for the "raw bytes" I wanted to convert back), and I got no further problems.

Sorry for wasting your time, guys. But I still learned some things, so I'll keep this here. Moral of the story, Devel::Peek is GOOD for debugging (thanks ikegami), and one should always double check, instead of assuming. Granted, I was in a hurry on friday, but the fault is still mine.

So, thanks to everyone who helped, or tried to, and special thanks to ikegami (again), who used quite a bit of his time helping me.

1. I'm trying to save binary data of images in Key-Value Store
2. 1st, read data using "fread" function. 2nd, save it into RocksDB. 3rd, Get the data from RocksDB and restore the data into form of image.
3. Now I don't know whether I have problem in 2nd step of 3rd step.

2nd step Put

Soon I will publish a new version of my library, which is composed of several modules. For this release, I'm now using Gradle 7, so I'm thinking about changing something that I feel needs to be fixed:

This is the library declaration (for example, the thread module):

group: 'com.intellisrc', name: 'thread', version: '2.8.0'

when published, it generates a jar with the name: thread-2.8.0.jar, which I would prefer to be: intellisrc-thread-2.8.0.jar as it is more descriptive (other than that, it is working without issues).

To fix it, one option is to change the artifact-id, so the declaration will become:

group: 'com.intellisrc', name: 'intellisrc-thread', version: '2.8.0'

But I would prefer not to change it as consumers would have to update it and it might be confusing.

I guess that if I can change the jar name and the publication files (pom.xml, module.json), it should work.

So far I was able to change the jar file name that it is generated inside build/libs/ using:

I am trying to choose a database for a newly developing application. There are so many alternatives and it’s so easy to choose a wrong one. First of all, there is a requirement to not use database servers. A required database should be a static or dynamic C++ library. The data that needs to be stored is an array of records. They vary but are fixed for a given dataset (so they can be stored in a table). The information in each row could be from several hundred bytes up to several megabytes. And a number of rows may be millions for now and expected to grow.

The index of the row could be used as a key. No need to maintain a separate key column.

Data is inserted sequentially. Read access will be performed only by iterating all the data or some segment of it sequentially (May need to iterate with steps like each 5th).

1. I don’t think that relational DBs are good feet for many reasons. a. They are mostly server-based. I know about SQLite but as far as I know, it stores data in one file which I assume may lead to issues related to maximum file size. b. We don’t need the power that SQL provides instead we would like to have more flexibility in stored data types.
2. There are Key/Value non-SQL dbms like BerkeleyDB, RocksDB, or something like luxio for lighter alternatives. The functionality they provide is more than enough for the task. And this might be the right choice however I don’t know how well they are optimized for such case where we have continuous integer keys. The associative key access (which is not required for us) may have some overhead in performance.
3. I know there are some type of non-SQL databases called “wide-column” which I am not familiar with. However, the name sounds like it is perfect for our task. All databases I can find are server of claud based. If you know dbm-like library for such type of database please advise. I am not experienced in databases so please correct me if I am wrong in any of 3 above stamens.

Is there a built-in way in SQLite (or similar) to keep the best of both worlds SQL / NoSQL, for small projects, i.e.:

• stored in a (flat) file like SQLite (no client/server scheme, no server to install; more precisely : nothing else to install except pip install )
• possibility to store rows as dict, without having a common structure for each row, like NoSQL databases
• support of simple queries

Example: