Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Experimental dht client #8061

Merged
merged 3 commits into from
May 14, 2021
Merged

Experimental dht client #8061

merged 3 commits into from
May 14, 2021

Conversation

aschmahmann
Copy link
Contributor

@aschmahmann aschmahmann commented Apr 8, 2021

This feature enables an option for an experimental DHT client that makes a few key optimizations over the standard client.

  1. It periodically crawls the network to find all the peers and then finds the closest peers to the target in Kademlia space based on the local list of peers
  2. It does not wait on all peers from a given query to return, but instead tries to guess when we'll be done hearing from peers and instead be waiting on timeouts.
    • The current metric is to wait for some percentage (30%) of the closest bucket-size (i.e. 20) peers to return and then wait for some dead time (a few hundred ms) where no new responses come in

Depends on:

A couple quirks about the current state of this PR and the client:

  • I've manually set the experimental option to always on for debugging purposes (will turn off before merge)
    • turned off
  • The DHT client crawls for updates once an hour and takes about 10 minutes to complete in my tests on the live network
    • This means the node's DHT is unusable for the first 10 minutes of operation
    • This means many of the DHT tests will fail since the node will either not have an updated view of the network or will simply have an empty routing table
  • Given the utility of a faster DHT client I'm hoping that an experimental client that's much much faster and that the community can hammer on will be appreciated while we work on improving the client to handle more of the edge cases

This PR also includes an experimental provider system which utilizes the experimental dht client and is bundled with it - the originating PR was #8070

@aschmahmann aschmahmann mentioned this pull request Apr 8, 2021
71 tasks
@aschmahmann aschmahmann added this to the go-ipfs 0.9 milestone Apr 8, 2021
@aschmahmann aschmahmann force-pushed the feat/experimental-dht-client branch 2 times, most recently from 3556ad2 to 476444c Compare May 13, 2021 05:07
@aschmahmann aschmahmann marked this pull request as ready for review May 13, 2021 06:06
core/commands/stat_dht.go Show resolved Hide resolved
@aschmahmann aschmahmann force-pushed the feat/experimental-dht-client branch from 30b7a2a to d9d6ba3 Compare May 14, 2021 08:18
The experimental AcceleratedDHTClient can be enabled from the config

When enabled it modifies the output of the `ipfs stats dht` command.
@aschmahmann aschmahmann force-pushed the feat/experimental-dht-client branch from d9d6ba3 to 2fd55d1 Compare May 14, 2021 08:31
The batched provider system is enabled when the experimental AcceleratedDHTClient is enabled

There is also an `ipfs stats provide` command which gives stats about the providing/reproviding system when the batched provider system is enabled
@aschmahmann aschmahmann merged commit 0bd3b6e into master May 14, 2021
Comment on lines +47 to +48
cmds.StringArg("dht", false, true, "The DHT whose table should be listed (wanserver, lanserver, wan, lan). "+
"wan and lan refer to client routing tables. When using the experimental DHT client only WAN is supported. Defaults to wan and lan."),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are wanserver/lanserver and are they new? Can the description explain 🙏


```
ipfs config --json Experimental.AcceleratedDHTClient true
```
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a note that if the daemon is running it must be restarted?

@hacdias hacdias deleted the feat/experimental-dht-client branch May 9, 2023 11:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants