Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

experimental provider system #8070

Merged

Conversation

aschmahmann
Copy link
Contributor

@aschmahmann aschmahmann commented Apr 13, 2021

Adds an experimental provider system that makes use of the experimental dht client.

Depends on ipfs/go-ipfs-provider#34, links to #8061

The point of this system is to be able to quickly provide by being sympathetic to how Kademlia works. For the time being it tries to get records into the DHT very quickly at the expense of having a spike in resource usage during that time.

It also comes with a few extras that have been added for sanity purposes during testing/evaluation and that may be useful in production including:

  • A stat command to track how many provider records we are emitting (only for the experimental system)
  • Tracking when we last provided our records so that rebooting the node doesn't cause us to automatically start reproviding unless we need to

@aschmahmann aschmahmann mentioned this pull request Apr 13, 2021
71 tasks
@aschmahmann aschmahmann added this to the go-ipfs 0.9 milestone Apr 13, 2021
@aschmahmann aschmahmann force-pushed the feat/experimental-provider-system branch from 3fbb6aa to 105b730 Compare April 22, 2021 23:13
@aschmahmann aschmahmann force-pushed the feat/experimental-provider-system branch from acbd82f to dbdee7f Compare May 13, 2021 04:44
@aschmahmann aschmahmann force-pushed the feat/experimental-dht-client branch 2 times, most recently from 3556ad2 to 476444c Compare May 13, 2021 05:07
@aschmahmann aschmahmann force-pushed the feat/experimental-provider-system branch from dbdee7f to c5a49d2 Compare May 13, 2021 07:00
@aschmahmann aschmahmann marked this pull request as ready for review May 13, 2021 07:00
@aschmahmann aschmahmann force-pushed the feat/experimental-provider-system branch from c5a49d2 to 35c25f1 Compare May 13, 2021 07:11
Copy link
Member

@lidel lidel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The way experiment is introduced LGTM, some small style tweaks inline.

ShortDescription: `
Returns statistics about the content the node is advertising.

This interface is not stable and may change from release to release.
Copy link
Member

@lidel lidel May 13, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💭 (not related to this PR, but in general) would be cool if go-ipfs-cmds had a flag to indicate "experimental" and "deprecated" hint, so markdown generated by https://github.com/ipfs/http-api-docs could reflect that in some visual way.
Perhaps a simple boolean flag would do? How could I kick this off? PR against go-ipfs-cmds?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure, probably? Maybe start an issue in go-ipfs for tracking and collecting feedback before starting on the PR.

core/commands/stat_provide.go Outdated Show resolved Hide resolved
Copy link
Contributor

@gammazero gammazero left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, with a couple very minor comments

core/node/provider.go Outdated Show resolved Hide resolved
docs/experimental-features.md Outdated Show resolved Hide resolved
@aschmahmann aschmahmann force-pushed the feat/experimental-dht-client branch 2 times, most recently from d9d6ba3 to 2fd55d1 Compare May 14, 2021 08:31
The batched provider system is enabled when the experimental AcceleratedDHTClient is enabled

There is also an `ipfs stats provide` command which gives stats about the providing/reproviding system when the batched provider system is enabled
@aschmahmann aschmahmann force-pushed the feat/experimental-provider-system branch from e226399 to acd4683 Compare May 14, 2021 08:42
@aschmahmann aschmahmann merged commit a67cab1 into feat/experimental-dht-client May 14, 2021
@aschmahmann aschmahmann mentioned this pull request May 14, 2021
2 tasks
@hacdias hacdias deleted the feat/experimental-provider-system branch May 9, 2023 11:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants