Distributed-Systems | MIT课程《Distributed Systems 》学习和翻译 | Architecture library

It was a problem with the constructors of vidFile. Basically I using the Arrays.arraycopy arguements in reverse (I was copying from the empty array into the full array)

You should try creating a virtual environment to check if this solves your problem

there's a varietey to achieve this, but most of them consist of creating a venv -> activating the venv and then starting the python program while the venv is active

Side-note: An IDE like Pycharm can be configured to activate the venv automatically to run your local server, using the "Configurations" dialog.

It's important to point out that Kafka guarantees item ordering in the order it was written, not necessarily in the intended order.

In order to guarantee order you provide a key, the producer will use that key in the default partitioner algorithm, it's approximately this:

Consistency is an overloaded term which has multiple meanings. But usually, when people say that a system is consistent they mean that the system is free of contradictions and its behavior matches the specification. Obviously, there are a lot of consistent distributed systems which don't use logs.

Examples:

1. (joke) If you only write data and never read then /dev/null is consistent because by definition you can't read data so you can't end up with contradiction

2. Distributed version control systems (like Git) are consistent because they avoid contradictions by keeping both versions (branches) and delegating the resolving of the conflicts to the clients (merge/rebase).

3. CRDT + Quorum reads/writes. Source code has a complex structure, and it's impossible to think about a universal auto merge algorithm. But with simpler structures it's possible. If we need to implement a trivial comment system and we don't care about the order and delete/edit capabilities, then we can store comments in a set and use set-union as the automated conflict resolution strategy. CRDT protects against conflicts, while quorum reads/writes gives us the wall-clock order: once an operation finishes its effects are visible to all follow-up reads. The lack of conflicts and wall-clock time guarantee the lack of contradictions.

4. Quorum reads/writes + single client. If a system has only one client then the conflicts are also impossible so the quorum reads/writes provides consistency as well.

People usually confuse consistency with linearizability, but even if we talk about linearizability then there are ways to use it without logs. The most famous protocol for achieving linearizability is Paxos.

Paxos is about building a distributed write-once register (Multi-Paxos is about distributed append-only log). It seems that regular Paxos (non Multi-Paxos) has a very narrow set of applications but:

1. We still can build consistent distributed systems on top of it. For example, we can use Paxos to overcome the downside of 2PC (2PC blocks a system if a coordinator fails): since a coordinator needs to decide to abort or commit a transaction only once, so it's fine to use write-once register to store that decision.

2. It has has a variant allowing to use it as a rewritable register without logs. I described this approach in the How Paxos Works post and implemented it in the Gryadka project with 500-lines of JavaScript. Also, the idea behind it was independently checked with TLA+ by Greg Rogers and Tobias Schottdorf.