Better control excessive traffic to Dgraph #2678
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
golangcibot
reviewed
Oct 19, 2018
golangcibot
reviewed
Oct 19, 2018
danielmai
added a commit
to dgraph-io/pydgraph
that referenced
this pull request
Dec 13, 2018
This test case to read-index-after-write within the same txn is no longer valid. See dgraph-io/dgraph#2678
danielmai
added a commit
to dgraph-io/pydgraph
that referenced
this pull request
Dec 19, 2018
This test case to read-index-after-write within the same txn is no longer valid. See dgraph-io/dgraph#2678
martinmr
pushed a commit
to dgraph-io/pydgraph
that referenced
this pull request
Dec 20, 2018
This test case to read-index-after-write within the same txn is no longer valid. See dgraph-io/dgraph#2678
dna2github
pushed a commit
to dna2fork/dgraph
that referenced
this pull request
Jul 19, 2019
- Proposal retries generate a lot of traffic for an already jammed system. Now we consider each proposal retry to have an exponentially increasing weight (2^retry), which is limited against `pending_proposal` flag. - For all the retries, the pending proposals rate limiter bleeds the count back slowly over time, to help with the congestion. - Remove `btree` concept, which was the cause of deadlocks during LRU.gets. The reason it was put in place was because Dgraph used to not write commits to disk earlier (keeping them in memory). That has changed now, with each commit going to disk before a read is done. So, we don't need to maintain this data structure. This speeds up mutations considerably. Note that the lack of `btree` also means that a txn won't be able to read back its own uncommitted writes to secondary indices. I think that's a rare use case and hence a fair tradeoff, given the complexity and performance cost of having to overlay this structure on the DB. * Working towards speeding up live loader. * Wait before re-proposing, so we don't create too much work. * Make rollup happen only once every 5m. * Slowly bleed back to the proposal limit. * Remove btree, gain performance. * Works nicely with a lot of traffic now. * Make tests work in query pacakge by ensuring that our commits are written to disk. * Fix systest, testtxn and other tests. * Add an Ignore method to keep linter happy (not used in this PR).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
pending_proposalflag.btreeconcept, which was the cause of deadlocks during LRU.gets. The reason it was put in place was because Dgraph used to not write commits to disk earlier (keeping them in memory). That has changed now, with each commit going to disk before a read is done. So, we don't need to maintain this data structure. This speeds up mutations considerably.Note that the lack of btree also means that a txn won't be able to read back its own uncommitted writes to secondary indices. I think that's a rare use case and hence a fair tradeoff, given the complexity and cost of having to overlay this structure on the DB.
This change is