distributedlog | high performance replicated log service | Architecture library

by twitter Java Version: 0.3.51-RC1 License: Apache-2.0

X-Ray Key Features Code Snippets Community Discussions(2)Vulnerabilities Install Support

kandi X-RAY | distributedlog Summary

distributedlog is a Java library typically used in Architecture applications. distributedlog has no bugs, it has no vulnerabilities, it has build file available, it has a Permissive License and it has high support. You can download it from GitHub, Maven.

Apache DistributedLog is an open source project of The Apache Software Foundation (ASF). The Apache DistributedLog project originated from Twitter.

Support

Quality

Security

License

Reuse

Support

distributedlog has a highly active ecosystem.

It has 2256 star(s) with 288 fork(s). There are 168 watchers for this library.

It had no major release in the last 12 months.

distributedlog has no issues reported. There are no pull requests.

It has a neutral sentiment in the developer community.

The latest version of distributedlog is 0.3.51-RC1

Quality

distributedlog has no bugs reported.

Security

distributedlog has no vulnerabilities reported, and its dependent libraries have no vulnerabilities reported.

License

distributedlog is licensed under the Apache-2.0 License. This license is Permissive.

Permissive licenses have the least restrictions, and you can use them in most projects.

Reuse

distributedlog releases are available to install and integrate.

Deployable package is available in Maven.

Build file is available. You can build the component from source.

Top functions reviewed by kandi - BETA

kandi has reviewed distributedlog and discovered the below as its top functions. This is intended to give you an instant insight into distributedlog implemented functionality, and help decide if they suit your requirements.

Asynchronously gets ledger list .
Creates missing metadata .
Acquires a stream .
Performs benchmark .
Fetch log records from a given segment .
Watch lock .
Sends a streaming request to the given address .
Attempts to transmit the next record .
Creates in progress log segment .
Builds a new Zookeeper connection .

Get all kandi verified functions for this library.

distributedlog Key Features

No Key Features are available at this moment for distributedlog.

distributedlog Examples and Code Snippets

No Code Snippets are available at this moment for distributedlog.

Community Discussions

Trending Discussions on distributedlog

Cannot resolve dependency in maven

When does a leader ACKs client in DistributedLog?

QUESTION

Cannot resolve dependency in maven

Asked 2020-Dec-11 at 23:35

I have to make a university project and to do that I have to clone Bookkeeper from github to create some test cases. My problem is that when I import the project with maven it shows me the error:

...

ANSWER

Answered 2020-Dec-11 at 18:33

Why you are using snapshot version of the library? Snapshot versions should not be used (and are not intended as released version by any project). Please use below dependency instead of snapshot.

Source https://stackoverflow.com/questions/65254569

QUESTION

When does a leader ACKs client in DistributedLog?

Asked 2018-Jul-30 at 16:31

I'm having difficulty to understand when does a leader actually ACKs client. Here is part of a DistributedLog documentation:

Each batched entry appended to a log segment will be assigned a monotonically increasing entry id by the log segment writer. All the entries are written asynchronously in a pipeline. The log segment writer therefore updates an in-memory pointer, called LAP (LastAddPushed), which is the entry id of the last batched entry pushed to log segment store by the writer. The entries could be written out of order but only be acknowledged in entry id order. Along with the successful acknowledges, the log segment writer also updates an in-memory pointer, called LAC (LastAddConfirmed). LAC is the entry id of the last entry that already acknowledged by the writer. All the entries written between LAC and LAP are unacknowledged data, which they are not visible to readers.

The readers can read entries up to LAC as those entries are known to be durably replicated - thereby can be safely read without the risk of violating read ordering. The writer includes the current LAC in each entry that it sends to BookKeeper. Therefore each subsequent entry makes the records in the previous entry visible to the readers. LAC updates can be piggybacked on the next entry that are written by the writer. Since readers are strictly followers, they can leverage LAC to read durable data from any of the replicas without need for any communication or coordination with the writer.

DL introduces one type of system record, which is called control record - it acts as the commit request in two-phases-commit algorithm. If no application records arrive within the specified SLA, the writer will generate a control record. With writing the control record, it would advance the LAC of the log stream. The control record is added either immediately after receiving acknowledges from writing a user record or periodically if no application records are added. It is configured as part of writer's flushing policy. While control log records are present in the physical log stream, they are not delivered by the log readers to the application.

Now consider the following scenario:

Leader publishes message to Bookkeeper
Followers get the messages, append to log and send ACK to leader
Leader gets the confirmation from followers, increments LAC and replies client that messages is committed.
NOW: Leader fails before it could piggyback to followers that LAC has been incremented.
The question is: Since potential leader is not aware of the fact that LAC has been incremented, it becomes the new leader and truncates the log to old LAC, which means we have lost an entry in the log that has been confirmed by previous leader.

As a result, client has been confirmed that the message has been successfully written, but it has been lost.

...

ANSWER

Answered 2018-Jan-10 at 00:37

Since potential leader is not aware of the fact that LAC has been incremented, it becomes the new leader and truncates the log to old LAC, which means we have lost an entry in the log that has been confirmed by previous leader.

There are several cases:

1) if the leader gracefully close the log, it will seal the log segment that it is writing to. The LAC will be advanced and it will also be recorded as part of the log segment metadata (which is stored in the metadata store).

2) if the leader crashes and doesn't close the log gracefully, a potential leader comes up, it will go through a recovery process. What the new leader does will be:

a) It will attempt the seal the last log segment that written by previous leader. The seal process is done by bookkeeper client, which includes two parts: (a) it will fence the log segment. fencing enforces that no more writes can happen in this log segment. (b) it will then do a forward-recovery from the last known LAC and recover entries that were written but not committed yet.
b) After recover the last log segment, the new leader will open a new log segment to write entries.

Hope this explains your question.

DistributedLog also has a paper published in ICDE 2017. You can get it from here.

Source https://stackoverflow.com/questions/48010841

Community Discussions, Code Snippets contain sources that include Stack Exchange Network

Vulnerabilities

No vulnerabilities reported

Install distributedlog

You can download it from GitHub, Maven.
You can use distributedlog like any standard Java library. Please include the the jar files in your classpath. You can also use any IDE and you can run and debug the distributedlog component as you would do with any other Java program. Best practice is to use a build tool that supports dependency management such as Maven or Gradle. For Maven installation, please refer maven.apache.org. For Gradle installation, please refer gradle.org .