paxos | This is an implementation of Paxos by Itay Neeman | Architecture library

Assume there were network failures such that every acceptor accepted a different value. Without the ability to change their value in future rounds, no progress could ever be made and you have a "livelock".

After acceptance if another proposer comes along and prepares a message, it will receive at least one reply containing the previously accepted value. The Paxos rules then require that the second proposer MUST propose the previously accepted value. The value it wants to propose is overridden by the first value. This ensures that only a single value can be chosen for a single instance of the Paxos algorithm.

Your second interpretation is correct ("It's like one log entry in Raft").

You are also correct that Basic Paxos can't choose multiple values, it just chooses one, like a single log entry in Raft. To choose a series of values you need to chain multiple Basic Paxos instances together, like in Multi-Paxos or Raft.

The error message says that the JVM failed to commit memory, because mmap syscall returned with the error code 12 (ENOMEM).

This typically happens when the process reaches one of the OS memory limits:

• The number of memory mappings exceeded vm.max_map_count sysctl limit. This is a quite common problem with Cassandra, since Cassandra tends to mmap thousands files, and the default limit is rather low - around 65K.

  How to check: wc -l /proc//maps where is Java process ID.

  How to fix: sudo sysctl vm.max_map_count=1000000

• Total amount of the process' virtual memory exceeded RLIMIT_AS

  How to check: ulimit -v, cat /proc//status | grep Vm

  How to fix: ulimit -v unlimited before starting Cassandra in the same shell.

• Overcommit is disabled, and the overcommit ratio is too low.

  How to check: sysctl vm.overcommit_memory, sysctl vm.overcommit_ratio, cat /proc/meminfo

  How to fix: sudo sysctl vm.overcommit_memory=0

To answer your specific question:

WHY it can eliminate lots of Prepare RPC?

In the paper Paxos Made Simple page 10 it says:

A newly chosen leader executes phase 1 for infinitely many instances of the consensus algorithm—in the scenario above, for instances 135–137 and all instances greater than 139.

That is saying that if a leader broadcasts Prepare(135,n) which is a prepare for instance 135 using ballot number n then it is valid that this can be defined as applying to all instances >=135 that are not yet fixed. We can reason that it is safe for any node to be "spamming" out prepare messages for an infinite number of the unfixed positions in our log stream. This is because for each position each acceptor uses the rules of Paxos for that position. We can compress that infinite set of prepare messages down to a single one that applies to all higher unfixed positions. We then eliminate all but one prepare message for the term of a stable leader. So it is fantastic optimisation.

You asked about any example code. I wrote an implementation of multi-paxos using functional programming in Scala that aims to be true to the paper Paxos Made Simple over at https://github.com/trex-paxos/trex. The core state is PaxosData, the message protocol is at the bottom of PaxosProtcol and the algorithm is a set of message matching functions in PaxosAlgorithm. The algorithm takes the current immutable state and an immutable message as input and outputs the next immutable state for the node. Common behaviours are written as partial functions that have full unit tests. These partial functions are composed into complete functions used by leaders, followers and candidate leaders. There is a write up at this blog.

It adds additional messages to the basic set as optimisations speed up log replication. Those involve some implementation details that Lamport does not get into in his paper. An example is that negative acknowledgements are used to pass information between nodes to try to avoid interrupting a stable leader due to only one failed network link between a node and the leader. TRex tries to keep those features to a minimum to create a basic but complete solution.

An answer that you might find helpful about Multi-Paxos is this one that discusses why Multi-Paxos is called that https://stackoverflow.com/a/26619261/329496

There is also this one about how the original Part-Time Parliament paper uses a leader and also describes a stable leader running multi-Paxos https://stackoverflow.com/a/46012211/329496

Finally, you might enjoy my defence of Paxos post The Trial Of Paxos Algorithm.

But where is that API layer situated? Does that refer to the location proxy that the paper says clients use to locate the relevant spanservers?

That's correct, it's in the API proxy.

I had assumed that the timestamp could be chosen by any of the Paxos leaders involved in the multi-site read, but apparently not.

To negotiate a timestamp, strong read requests must contact the Paxos leader for each Spanner group involved in that read.

Based on the dig output you shared zevrant-oauth2-service-db is resolving to 92.242.140.2 but it looks like the IP address of your K8s service is 10.97.75.171 (ClusterIP) (based on the output you shared too).

If you hit 10.97.75.171 5432 you should be able to access your Postgres database, provided that you don't have any Kubernetes Network Policy and/or firewall blocking access. Make sure you that in your Postgres config you are binding the server to 0.0.0.0 otherwise if it's something like localhost you will only be able to get to it from the pod.

So the question is what is 92.242.140.2? Wny is coredns responding to a query to zevrant-oauth2-service-db with 92.242.140.2? Is there a DNS forwarder configured in coredns? Is there a default domain configured that is not part of svc.cluster.local?

The major reason why this query fails is that it's incorrect - Cassandra works fast only when you have full partition key, and then you can do the range query inside that partition. In your case, you have partition key consisting of 5 columns, but you're providing only one in the query, and Cassandra needs to perform scanning of the whole data to find where the corresponding rows are located. I really wondering that it worked before...

To solve your problem you need to change the table structure to have partition/primary key matching your queries - all data modeling for Cassandra starts with queries that should be executed. I recommend to take DS220 course at DataStax Academy on data modelling.

The new node member selects a random subset of peers(seed members) of cluster as targets for gossip messages while joining the cluster based on the configuration provided to it. The address of at least one alive node member / seed member address of the cluster must be known to the new node member so that it can join the cluster. In case the seed member (introducer) gets killed at the initial stage, there is no other way for the new node to convey its existence and it shall become isolated unless there is alternate seed address in subset that is alive and known to the new node member. However, if the seed member (introducer) gets killed at a stage where the initial gossip of new node has been communicated to another node of the cluster, there shall not be much impact unless the new node is required for sure to reach a specific another node.

Once joined, the other nodes shall introduce it to gossips, this allows the new node member to gradually receive information about all the nodes of the cluster. The “gossip information” that new node is added is included in the packets such as (ping, ping-req, ack) that is used by the protocol to send messages to other nodes at regular intervals which is termed as “infection-style” dissemination mechanism. Accordingly, the membership list maintained by each node member shall get updated whenever a new node member joins or leaves the cluster.