kandi background
Explore Kits

HaloDB | A fast, log structured keyvalue store. | Key Value Database library

 by   yahoo Java Version: v0.5.6 License: Apache-2.0

 by   yahoo Java Version: v0.5.6 License: Apache-2.0

Download this library from

kandi X-RAY | HaloDB Summary

HaloDB is a Java library typically used in Database, Key Value Database applications. HaloDB has no vulnerabilities, it has build file available, it has a Permissive License and it has low support. However HaloDB has 80 bugs. You can download it from GitHub.
HaloDB is a fast and simple embedded key-value store written in Java. HaloDB is suitable for IO bound workloads, and is capable of handling high throughput reads and writes at submillisecond latencies. HaloDB was written for a high-throughput, low latency distributed key-value database that powers multiple ad platforms at Yahoo, therefore all its design choices and optimizations were primarily for this use case. Basic design principles employed in HaloDB are not new. Refer to this document for more details about the motivation for HaloDB and its inspirations. HaloDB comprises of two main components: an index in memory which stores all the keys, and append-only log files on the persistent layer which stores all the data. To reduce Java garbage collection pressure the index is allocated in native memory, outside the Java heap.
Support
Support
Quality
Quality
Security
Security
License
License
Reuse
Reuse

kandi-support Support

  • HaloDB has a low active ecosystem.
  • It has 406 star(s) with 82 fork(s). There are 28 watchers for this library.
  • It had no major release in the last 12 months.
  • There are 21 open issues and 13 have been closed. On average issues are closed in 164 days. There are 6 open pull requests and 0 closed requests.
  • It has a neutral sentiment in the developer community.
  • The latest version of HaloDB is v0.5.6
HaloDB Support
Best in #Key Value Database
Average in #Key Value Database
HaloDB Support
Best in #Key Value Database
Average in #Key Value Database

quality kandi Quality

  • HaloDB has 80 bugs (7 blocker, 14 critical, 35 major, 24 minor) and 392 code smells.
HaloDB Quality
Best in #Key Value Database
Average in #Key Value Database
HaloDB Quality
Best in #Key Value Database
Average in #Key Value Database

securitySecurity

  • HaloDB has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.
  • HaloDB code analysis shows 0 unresolved vulnerabilities.
  • There are 24 security hotspots that need review.
HaloDB Security
Best in #Key Value Database
Average in #Key Value Database
HaloDB Security
Best in #Key Value Database
Average in #Key Value Database

license License

  • HaloDB is licensed under the Apache-2.0 License. This license is Permissive.
  • Permissive licenses have the least restrictions, and you can use them in most projects.
HaloDB License
Best in #Key Value Database
Average in #Key Value Database
HaloDB License
Best in #Key Value Database
Average in #Key Value Database

buildReuse

  • HaloDB releases are available to install and integrate.
  • Build file is available. You can build the component from source.
  • Installation instructions are not available. Examples and code snippets are available.
  • HaloDB saves you 5402 person hours of effort in developing the same functionality from scratch.
  • It has 11329 lines of code, 990 functions and 99 files.
  • It has high code complexity. Code complexity directly impacts maintainability of the code.
HaloDB Reuse
Best in #Key Value Database
Average in #Key Value Database
HaloDB Reuse
Best in #Key Value Database
Average in #Key Value Database
Top functions reviewed by kandi - BETA

kandi has reviewed HaloDB and discovered the below as its top functions. This is intended to give you an instant insight into HaloDB implemented functionality, and help decide if they suit your requirements.

  • Updates the data in the given storage .
  • Take a snapshot .
  • Puts an entry in the hash table .
  • Compares two byte arrays .
  • Prints out to the specified log
  • Reads a record .
  • Stops compaction thread .
  • Get an approximate histogram of buckets .
  • Returns a map of string values as a map .
  • Compares the block with the specified key .

HaloDB Key Features

A fast, log structured key-value store.

Basic Operations.

copy iconCopydownload iconDownload
            // Open a db with default options.
            HaloDBOptions options = new HaloDBOptions();
    
            // Size of each data file will be 1GB.
            options.setMaxFileSize(1024 * 1024 * 1024);

            // Size of each tombstone file will be 64MB
            // Large file size mean less file count but will slow down db open time. But if set
            // file size too small, it will result large amount of tombstone files under db folder
            options.setMaxTombstoneFileSize(64 * 1024 * 1024);

            // Set the number of threads used to scan index and tombstone files in parallel
            // to build in-memory index during db open. It must be a positive number which is
            // not greater than Runtime.getRuntime().availableProcessors().
            // It is used to speed up db open time.
            options.setBuildIndexThreads(8);

            // The threshold at which page cache is synced to disk.
            // data will be durable only if it is flushed to disk, therefore
            // more data will be lost if this value is set too high. Setting
            // this value too low might interfere with read and write performance.
            options.setFlushDataSizeBytes(10 * 1024 * 1024);
    
            // The percentage of stale data in a data file at which the file will be compacted.
            // This value helps control write and space amplification. Increasing this value will
            // reduce write amplification but will increase space amplification.
            // This along with the compactionJobRate below is the most important setting
            // for tuning HaloDB performance. If this is set to x then write amplification 
            // will be approximately 1/x. 
            options.setCompactionThresholdPerFile(0.7);
    
            // Controls how fast the compaction job should run.
            // This is the amount of data which will be copied by the compaction thread per second.
            // Optimal value depends on the compactionThresholdPerFile option.
            options.setCompactionJobRate(50 * 1024 * 1024);
    
            // Setting this value is important as it helps to preallocate enough
            // memory for the off-heap cache. If the value is too low the db might
            // need to rehash the cache. For a db of size n set this value to 2*n.
            options.setNumberOfRecords(100_000_000);
            
            // Delete operation for a key will write a tombstone record to a tombstone file.
            // the tombstone record can be removed only when all previous version of that key
            // has been deleted by the compaction job.
            // enabling this option will delete during startup all tombstone records whose previous
            // versions were removed from the data file.
            options.setCleanUpTombstonesDuringOpen(true);
    
            // HaloDB does native memory allocation for the in-memory index.
            // Enabling this option will release all allocated memory back to the kernel when the db is closed.
            // This option is not necessary if the JVM is shutdown when the db is closed, as in that case
            // allocated memory is released automatically by the kernel.
            // If using in-memory index without memory pool this option,
            // depending on the number of records in the database,
            // could be a slow as we need to call _free_ for each record.
            options.setCleanUpInMemoryIndexOnClose(false);
            
            // ** settings for memory pool **
            options.setUseMemoryPool(true);
    
            // Hash table implementation in HaloDB is similar to that of ConcurrentHashMap in Java 7.
            // Hash table is divided into segments and each segment manages its own native memory.
            // The number of segments is twice the number of cores in the machine.
            // A segment's memory is further divided into chunks whose size can be configured here. 
            options.setMemoryPoolChunkSize(2 * 1024 * 1024);
    
            // using a memory pool requires us to declare the size of keys in advance.
            // Any write request with key length greater than the declared value will fail, but it
            // is still possible to store keys smaller than this declared size. 
            options.setFixedKeySize(8);
    
            // Represents a database instance and provides all methods for operating on the database.
            HaloDB db = null;
    
            // The directory will be created if it doesn't exist and all database files will be stored in this directory
            String directory = "directory";
    
            // Open the database. Directory will be created if it doesn't exist.
            // If we are opening an existing database HaloDB needs to scan all the
            // index files to create the in-memory index, which, depending on the db size, might take a few minutes.
            db = HaloDB.open(directory, options);
    
            // key and values are byte arrays. Key size is restricted to 128 bytes.
            byte[] key1 = Ints.toByteArray(200);
            byte[] value1 = "Value for key 1".getBytes();
    
            byte[] key2 = Ints.toByteArray(300);
            byte[] value2 = "Value for key 2".getBytes();
    
            // add the key-value pair to the database.
            db.put(key1, value1);
            db.put(key2, value2);
    
            // read the value from the database.
            value1 = db.get(key1);
            value2 = db.get(key2);
    
            // delete a key from the database.
            db.delete(key1);
    
            // Open an iterator and iterate through all the key-value records.
            HaloDBIterator iterator = db.newIterator();
            while (iterator.hasNext()) {
                Record record = iterator.next();
                System.out.println(Ints.fromByteArray(record.getKey()));
                System.out.println(new String(record.getValue()));
            }
    
            // get stats and print it.
            HaloDBStats stats = db.stats();
            System.out.println(stats.toString());
    
            // reset stats
            db.resetStats();
            
            // pause background compaction thread.
            // if a file is being compacted the thread
            // will block until the compaction is complete.
            db.pauseCompaction();
            
            // resume background compaction thread.
            db.resumeCompaction();
            
            // repeatedly calling pause/resume compaction methods will have no effect.

            // Close the database.
            db.close();

Community Discussions

Trending Discussions on Key Value Database
  • Laravel how to "properly" store & retrieve models in a Redis hash
  • Can compacted Kafka topic be used as key-value database?
Trending Discussions on Key Value Database

QUESTION

Laravel how to "properly" store & retrieve models in a Redis hash

Asked 2021-Jul-08 at 17:02

I'm developing a Laravel application & started using Redis as a caching system. I'm thinking of caching the data of all of a specific model I have, as a user may make an API request that this model is involved in quite often. Would a valid solution be storing each model in a hash, where the field is that record's unique ID, and the values are just the unique model's data, or is this use case too complicated for a simple key value database like Redis? I"m also curious as to how I would create model instances from the hash, when I retrieve all the data from it. Replies are appreciated!

ANSWER

Answered 2021-Jul-08 at 17:02

Short answer: Yes, you can store a model, or collections, or basically anything in the key-value caching of Redis. As long as the key provided is unique and can be retraced. Redis could even be used as a primary database.

Long answer

Ultimately, I think it depends on the implementation. There is a lot of optimization that can be done before someone can/should consider caching all models. For "simple" records that involve large datasets, I would advise to first optimize your queries and code and check the results. Examples:

  1. Select only data you need, not entire models.
  2. Use the Database Query Builder for interacting with the database when targeting large records, rather than Eloquent (Eloquent is significantly slower due to the Active Record pattern).
  3. Consider using the toBase() method. This retrieves all data but does not create the Eloquent model, saving precious resources.
  4. Use tools like the Laravel debugbar to analyze and discover potential long query loads.

For large datasets that do not change often or optimization is not possible anymore: caching is the way to go!

There is no right answer here, but maybe this helps you on your way! There are plenty of packages that implement similar behaviour.

Source https://stackoverflow.com/questions/68305332

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install HaloDB

You can download it from GitHub.
You can use HaloDB like any standard Java library. Please include the the jar files in your classpath. You can also use any IDE and you can run and debug the HaloDB component as you would do with any other Java program. Best practice is to use a build tool that supports dependency management such as Maven or Gradle. For Maven installation, please refer maven.apache.org. For Gradle installation, please refer gradle.org .

Support

Contributions are most welcome. Please refer to the CONTRIBUTING guide.

DOWNLOAD this Library from

Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from
over 430 million Knowledge Items
Find more libraries
Reuse Solution Kits and Libraries Curated by Popular Use Cases

Save this library and start creating your kit

Explore Related Topics

Share this Page

share link
Reuse Pre-built Kits with HaloDB
Consider Popular Key Value Database Libraries
Compare Key Value Database Libraries with Highest Support
Compare Key Value Database Libraries with Highest Quality
Compare Key Value Database Libraries with Highest Security
Compare Key Value Database Libraries with Permissive License
Compare Key Value Database Libraries with Highest Reuse
Find, review, and download reusable Libraries, Code Snippets, Cloud APIs from
over 430 million Knowledge Items
Find more libraries
Reuse Solution Kits and Libraries Curated by Popular Use Cases

Save this library and start creating your kit

  • © 2022 Open Weaver Inc.