kademlia | A golang implementation of the Kademlia DHT | Stream Processing library

Does this flow look all right?

It seems roughly correct, but your wording is not very precise.

Each node has a routing table by which it organizes the neighbors it knows about and another table in which it organizes the data it is asked to store by others. Nodes have a quasi-random ID that determines their position in the routing keyspace. The hashes of keys for stored data don't precisely match any particular node ID, so the data is stored on the nodes whose ID is closest to the hash, as determined by the distance metric. That's how node IDs and key hashes are used for both.

When you perform a lookup for data (i.e. find_value) you ask the remote nodes for the k-closest neighbor set they have in their routing table, which will allow you to home in on the k-closest set for a particular target key. The same query also asks the remote node to return any data they have matching that target ID.

When you perform a find_node on the other hand you're only asking them for the closest neighbors but not for data. This is primarily used for routing table maintenance where you're not looking for any data.

Those are the abstract operations, if needed an actual implementation could separate the lookup from the data retrieval, i.e. first perform a find_node and then use the result set to perform one or more separate get operations that don't involve additional neighbor lookups (similar to the store operation).

Since kademlia is UDP-based you can't really serve arbitrary files because those could easily exceed reasonable UDP packet sizes. So in practice kademlia usually just serves as a hash table for small binary values (e.g. contact information, public keys and such). Bulk operations are either performed by other protocols bootstrapped off those values or by additional operations beyond those mentioned in the kademlia paper.

What the paper describes is only the basic functionality for a routing algorithm and most basic key value storage. It is a spherical cow in a vacuum. Actual implementations usually need additional features or work around security and reliability problems faced on the public internet.

But then what's bootstrapping node? Is it another process that's always running? What if the bootstrapping node gets turned off?

That's covered in this question (by example of the bittorrent DHT)

Kademlia is an abstract routing algorithm combined with a set of operations required to build a distributed hash table. The concept of broadcasts does not exist in kademlia-as-algorithm.

But concrete implementations can add features on top of this foundation. Since kademlia provides the iterative find_node procedure (there's no forwarding!) you can locate a node and then exchange any number and type of additional messages for which they have mutual support.

The confusion stems from different versions of the paper.

This is from the pre-print version and mostly used to outline basic properties of kademlia in a theoretical manner and still reflected in §2.2 and §3 of the full version.

Many real-world implementations implement this approach but they don't implement bucket splitting, merging or node multihoming.

It involves putting contacts into the ith bucket that shares i prefix bits with the node. Which means the layout uses distances relative to the node's own ID.

This is described in section §2.4.

To implement refinements such as handling highly unbalanced trees described towards the end of §2.4 or deeper non-local splitting described in §4.2 one needs to associate each bucket with the keyspace range it covers, this can be expressed similar to CIDR ranges, i.e. a start ID and the number of prefix bits shared to mask off the tail of the ID.

Splitting a bucket is performed by increasing the number of prefix bits by one and setting the added bit to 0 and 1 respectively for two new buckets.

Unlike the flat layout this structure does not involve distances relative to the node's own ID, although some decisions are based on whether the node's own ID would fall into a bucket.

Since the number of buckets in such a routing table varies over time it has to represented in a resizable data structure, this is mentioned in §2.4. Since access can't be done by a fixed index anymore since the exact bucket that will cover any specific node ID is not known until the prefix-ranges have been examined some kind of O(log n) search is needed if one wants to avoid scanning the whole bucket list each time.
Sorting the buckets by the lowest ID that the bucket would cover is a natural approach to achieve this. BTrees or sorted arrays combined with binary search can be used to achieve this.

Regardless which approach you take, populating a response to find_node requests with the correct set of contacts that match the request's target is not trivial since any single bucket may be insufficient to fill it and thus multiple buckets need to be traversed. It may be simpler to scan the whole routing table for the best available candidates for the reply.

It's not correct because

• binary.Uvarint() can only decode numbers within 64 bits, and your rawBytes is 256 bits
• The "varint" encoding (as commented in https://golang.org/src/encoding/binary/varint.go) is basically not compatible with raw bytes.

You have to use the math/big package for usage like that. Here is my revised version of your snippet:

The length was chosen because SHA1, used as hash function for the hash table keys, outputs 160bits and that was the most widely used hash function at the time.

The routing algorithm itself does not require that specific length to work, all it needs is the key space being large enough to avoid collisions in randomly chosen IDs. 128bit IDs would provide 64bits of collision space, which should be sufficient unless you intend to address grey goo.

But in addition to the routing algorithm itself cryptographic concerns may also be relevant. Networks that use encryption benefit from node IDs doubling as the node's public key and commonly deployed ECC algorithms require public keys of at least 256 bits. Additionally resistance against (currently hypothetical) quantum attacks has inflated recommended hash function sizes well beyond 128 bits since they would cut collision resistance to N/3 down from N/2 for classical attacks.

Not sure why this was downvoted. These are excellent questions.

1- What I don't understand is if this is all decentralized who remove from the DHT peer that no longer host a file content?

I think that DHT entries are regularly re-broadcast. So if a peer goes away, its DHT entries will no longer be broadcast and the network will forget about the data it provides unless some other node has it.

2- What prevent someone from storing large amount of data for free inside the DHT?

Unless you re-publish or somebody else is interested in the data, it will vanish. The amount of data that you can store directly in a DHT entry is limited. So you can make other nodes store some of your data by putting data directly into DHT entries, but the effort outweighs the benefits.

3- What prevent someone from disrupting the network by adding large number of invalid peer for a popular file.

I think there are some mechanisms envisioned in IPFS to protect the DHT against attacks. However, I don't think the current implementation is all that sophisticated. I don't think that current IPFS would deal well with a large scale distributed DDOS attack.

4- What prevent a bad actor from joining the DHT ring and not following the routing protocol thus preventing discovery message from reaching correct nodes.

I think a single node would be insufficient to do much damage, because a node will ask multiple peers. You would have to have multiple nodes to do significant damage.

But IPFS as it is now would not survive a sophisticated attack by state actors.

For a basic kademlia implementation you only need 2 bit arithmetic operations on the IDs: xor and comparison. For both cases the ID conceptually is a 160bit unsigned integer with overflow, i.e. modulo 2^160 arithmetic. It can be decomposed into a 20bytes or 5×u32 array, assuming correct endianness conversion in the latter case. The most common endianness for network protocols is big-endian, so byte 0 will contain the most significant 8 bits out of 160.

Then the xor or comparisons can be applied on a subunit by subunit basis. I.e. xor is just an xor for all the bytes, the comparison is a binary array comparison.

Using bigint library functions are probably sufficient for implementation but not optimal because they have size and signedness overhead compared to implementing the necessary bit-twiddling on fixed-sized arrays.

A more complete implementation may also need some additional arithmetic and utility functions.

Could I also just convert each NodeID into decimal and XOR those values?

Considering the size of the numbers decimal representation is not particularly useful. For the human reader heaxadecimal or the individual bits are more useful and computers operates on binary and practically never on decimal.

First a headsup: the paper you are linking to is the pre-proceedings version only containing the basic sketch without later refinements. The 160-bucket array routing table layout is a simplified approached for the proof of the paper, later revisions introduce a more sophisticated tree-based table.

I.e. bucket[0] would have node ids with 160 leading zeros (only 1 node) and bucket[159] would have nodes with no leading zeros (50% of the entire address space).

Well, you can do it this way, but it's simpler to just count the leading zeros in the xor distance and use that as index. I.e. 0 shared prefix bits = no (0) leading zeroes = buckets[0] = bucket furthest from your own ID.

Question Using this implementation, when finding the closest k-nodes to a target nodeId would I just count the leading zeros for the target and return everything in that k-bucket?

The following is assuming that your're asking how to answer a remote node's queries.

The buckets in the flat routing table layout are organized with respect to your own node ID. When answering queries for some arbitrary target ID then this is not necessarily aligned with the closeness towards that target. So the simplest approach is to just scan all populated buckets in your routing table and calculate the N closest nodes relative to the query's target address and then return those as a response. Avoiding a full scan would involve some arithmetic on the xor metric to find the correct local buckets, but I have only done that for the tree-based layout, not the flat layout.

How concerned are you about efficiency? As a lower bound someone has to send a packet to all N nodes in the network to propagate an update to all nodes.

The most naive approach is to simply forward every message to all entries in your routing table. This will not do since it obviously leads to forwarding storms.

The second most naive approach is to forward updates, i.e. newer data. This will result in N * log(N) traffic.

If all your nodes are trusted and you don't care about the last quantum of efficiency you can already stop here.

If nodes are not trusted you will need a mechanism to limit who can send updates and to verify packets.

If you also care about efficiency you can add randomized backoff before forwarding and tracking which routing table entry already has which version to prune unnecessary forwarding attempts.

If you don't want to gossip with the whole network but only a subset thereof you can implement subnetworks which interested nodes can join, i.e. subscribe to. Bittorrent Enhancement Proposal 50 describes such an approach.