BSIP67: Dynamic Market Fees #133

OpenLedgerApp · 2018-12-17T15:54:28Z

Dear BitShares Community,

We would like to introduce the Dynamic Market Fees BSIP.
The purpose is to support high-volume trading and market makers. Thus making it more profitable for them. We believe it will bring more people to BitShares.

As per BitShares Core Team request we have spent some time drafting this BSIP.
And here's the resulting BSIP:
https://github.com/openledger/bsips/blob/bsip-dynamicmarketfees/bsip-00XX%20Dynamic%20market%20fees.md

The pull request is here:
#133
Forum thread is here:
https://bitsharestalk.org/index.php?topic=27589.0
Please share your opinion.
If you think it might help BitShares, please voice your opinion and vote for it, when the worker is created.

Sincerely,
Denis Sokolov
OpenLedger

pmconrad · 2019-01-22T13:46:22Z

Please squash this PR and use a sensible commit message.

pmconrad · 2019-01-22T13:55:29Z

bsip-00XX Dynamic market fees.md

+One of the ways to increase the trade volume can be Dynamic Market Fees that will provide additional discounts for active traders.
+For example, to set lower fees for trading larger volumes.
+
+So, instead of market fee, an asset owner can optionally set up dynamic_market_fee_percent property that may look like a following example table.


Please elaborate. Since the given table is only meant as an example, how is that property to be set up and evaluated, and which degrees of freedom does this allow?

A table is an example. UIA owner can edit this table, set up a percent and trading volume.

That still doesnt define what the degrees of freedom are

pmconrad · 2019-01-22T13:58:58Z

This proposal introduces a significant cost in terms of CPU and RAM for all node operators. IMO this needs to be balanced against the potential benefits.
I agree that this might incentivize market makers, but it is unclear if this can justify the significant cost.

OpenLedgerApp · 2019-01-29T08:29:39Z

That's a good point! We have thought about it. We are planning to check and monitor performance KPIs at the beginning of implementation. It will allow us to make sure CPU and RAM are tracked. If we face some issues and we don't have any turnaround, it will be indicated ASAP.

ryanRfox · 2019-01-30T15:02:37Z

This proposal introduces a significant cost in terms of CPU and RAM for all node operators. IMO this needs to be balanced against the potential benefits.
I agree that this might incentivize market makers, but it is unclear if this can justify the significant cost.

@sschiessl-bcp has suggested elsewhere that we add a Risks section to the BSIP Template. @pmconrad may I request your craft your comment above into a paragraph highlighting the potential risk(s) you foresee? My desire is the identified risks will be added to the PR by @OpenLedgerApp

pmconrad · 2019-01-31T13:02:27Z

Suggestion:

Risks

CPU Overhead

The proposed fee calculations will have to be performed on every trade. This adds significant burden on top of the already quite heavyweight market engine. While this is probably not a big problem with today's transaction volume, it will further limit how far our chain scales in terms of tx/s.

RAM Overhead

The proposed calculations require logging the trade volume of all traders over a period of time, separately for each traded asset. With both the number of users and the number of assets growing over time, the required amount of RAM will grow squarely over time. This does not scale well.

ryanRfox · 2019-01-31T13:29:55Z

Thank you @pmconrad for adding the Risks section. May I request @OpenLedgerApp update the PR to include the Risks just prior to the Summary for Shareholders?

ryanRfox · 2019-01-31T13:41:41Z

My thoughts on calculation impacts/mitigation:

I now understand why performing the calculation on every order would be problematic. Can we calculate a static table hourly, daily so the market can just lookup the fee?
I now understand that storing complete order history for a month to use in calculation will consume substantial RAM. Can we mitigate this by calculating it into a volume average per hour, day?
What other options can be explored to reduce the scaling impacts within this specification?
I feel dynamic market fees are a valuable addition to the protocol and may provide an incentive for further adoption, so hope we continue to debate this topic.

OpenLedgerApp · 2019-01-31T13:56:07Z

@pmconrad Thanks for the input! we will add this section to the BSIP description.
@ryanRfox we are analyzing your thoughts and will provide our feedback.
In any case, we should think how we can have some metrics in advance (or at the beginning of implementation) to decrease RISKS of this BSIP.

pmconrad · 2019-02-01T10:57:47Z

I now understand why performing the calculation on every order would be problematic. Can we calculate a static table hourly, daily so the market can just lookup the fee?

We must update the trade volume per user and asset for every trade.

I now understand that storing complete order history for a month to use in calculation will consume substantial RAM. Can we mitigate this by calculating it into a volume average per hour, day?

This is already described in the section "30-day volume method". It requires two values per user and asset.

sschiessl-bcp · 2019-02-21T09:36:45Z

I would very much like this BSIP in two steps. First, introduce maker-taker, then maybe dynamic fees like indicated above.

Limit order would have fixed fee plus market fee, both for maker and taker. But, when filled

maker may receive a refund for fixed fee (maybe 50%, committee-parameter)
asset owner can define how much % of the market fee is refunded to the maker

Thoughts?

xeroc · 2019-02-22T10:51:39Z

I would very much like this BSIP in two steps. First, introduce maker-taker, then maybe dynamic fees like indicated above.

This!
"Keep it simple stupid" and do one thing at a time. Also, a BSIP that proposes only maker/taker fees will be much easier to get approved than something that introduces it together with something that adds significant additional burden to the chain

OpenLedgerApp · 2019-03-11T16:01:59Z

We have made a prototype in order to measure performance

As you can see in the picture attached, our worker has no significant influence on BitShares performance:

For more details please refer the prototype. Here is the link - https://github.com/openledger/bitshares-core/commits/issue-dmf

abitmore · 2019-03-12T09:25:28Z

@OpenLedgerApp try test with 1M users? and around 1K trading pairs (although there can be more)?

OpenLedgerApp · 2019-03-27T14:01:52Z

@OpenLedgerApp try test with 1M users? and around 1K trading pairs (although there can be more)?

There is a result:

The sources are located - https://github.com/openledger/bitshares-core/commits/issue-dmf

ryanRfox · 2019-04-03T00:39:19Z

Sorry, I'm not understanding the graph. What is the X-axis representing? It seems like there are both red and blue plots for about 2/3 of the data, but only blue when the value jump. What accounts for the jump? Are there red values as well?

pmconrad · 2019-04-03T10:17:19Z

AFAICS your test creates 1 M users but only 1000 traders, and it trades only 1 asset pair.

OpenLedgerApp · 2019-04-19T07:02:51Z

AFAICS your test creates 1 M users but only 1000 traders, and it trades only 1 asset pair.

We have made test with 100 asset pair.

As you can see in the picture attached, our worker has no significant influence on BitShares performance:

sources are located - https://github.com/openledger/bitshares-core/commits/issue-dmf

pmconrad · 2019-04-24T14:12:46Z

Now you have 100 assets and 1000 traders, but each trader is using only 1 or 2 assets. Like abit said, please repeat your test with 1M traders and 1k trading pairs where each trader trades in a significant portion of these pairs.

Also, please explain your graphics, as Ryan has requested. I don't understand them either.

OpenLedgerApp · 2019-04-25T14:13:08Z

We will update our test in the near future.

OpenLedgerApp · 2019-04-29T15:27:12Z

Now you have 100 assets and 1000 traders, but each trader is using only 1 or 2 assets. Like abit said, please repeat your test with 1M traders and 1k trading pairs where each trader trades in a significant portion of these pairs.

We have modified the tests. There are 100 traders and 100 assets where each trader trades 100 assets. We have got the following results: Origin bitshares takes 2095465 milliseconds, and with DMF logic 2099182 milliseconds.
You are welcome to change tests' parameters and rerun them:
// How many traders to create
const static auto traders_count = 100;
// Count of cycles for trading
const static auto iterations = 1;
// How many assets to sell/buy
const static auto uia_to_sell = 20;
// How many assets to create
const static auto ui_assets_count = 100;

New sources are located openledger/bitshares-core@e4f9f44

Also, please explain your graphics, as Ryan has requested. I don't understand them either.

Red line shows how much time origin bitshares take and an other line shows how much time the same test takes with DMF logic.

abitmore · 2019-06-09T11:59:02Z

@ryanRfox please assign BSIP number.

OpenLedgerApp · 2019-07-26T15:06:52Z

We would like to put our BSIP on voting.
The prototype is ready for voting. We have finished all test and made Pull Request.
There are some worries about computation resources.
That why, we have made several performance tests, which showed good results.

What should we do to put this BSIP on voting?

@ryanRfox please let us know. Thanks in advance!

abitmore · 2019-07-26T19:22:32Z

bsip-00XX Dynamic market fees.md

+account_id_type owner;
+asset_id_type asset;
+timestamp first_trade_date; //first trade date for this asset
+share_type total_volume; //total bought asset volume


We can't use share_type for this data due to the high possibility of overflow. I guess a 128-bit data type would be sufficient, and in the implementation still need to deal with overflows.

abitmore · 2019-07-26T20:12:22Z

@OpenLedgerApp please update the document and rename the file with the assigned BSIP number.

sschiessl-bcp · 2019-07-29T22:26:54Z

Please address all comments, and update your document.

Also I am still wondering if only maker/taker differentiation should be an option as well when voting as the simple version.

pmconrad

For more realistic data I have slightly modified your existing implementation to compute the trade statistics for all trades since the beginning of time while still applying the old market fee calculation rules.
As of block 35M there are roundabout 800,000 trade statistics objects generated which is about 32MB raw, plus index overhead. Replay time is slightly worse but acceptable IMO (1-2%). Both are acceptable IMO. Also, there is some room for improvement in the implementation, e. g. trade statistics should be removed if there were no trades within the last 30 days.

IMO this BSIP is not ready for voting, because the specification is virtually non-existent. Please see #170 for an example what the "Specification" section should contain.
It is also confusing because it presents several different options without making it clear which option is the one that will be implemented. Please focus on one option only. Considered alternatives should be mentioned briefly in the "Discussion" section, together with a reason why they were dismissed.

OpenLedgerApp added 30 commits December 14, 2018 18:17

test

2074c23

test 2

015c210

initial draft commit

ad28bd4

test 2

38eecaa

test3

7af44c0

test 5

0b09a84

test 5

2954b7c

test6

ed77dfc

test7

aa8daf3

test 78

a30b126

test8

3e02f82

test9

d25bb10

test9

88eda95

TEST10

3b3a39b

10

340db90

11

cedf2cf

11

9419c6c

11

1db21e0

12

9a3276c

13

e5592e8

14

8d2bed6

15

64eadd8

15

a88200f

16

2df002e

17

15f7d50

18

2b2766f

draft

b64a867

draft

e907ae6

bsip draft 1

abd61f0

draft 1

d41c1ab

pmconrad reviewed Jan 22, 2019

View reviewed changes

ryanRfox changed the title ~~BSIP Draft: Dynamic Market Fees~~ BSIP67: Dynamic Market Fees Jun 9, 2019

abitmore reviewed Jul 26, 2019

View reviewed changes

pmconrad suggested changes Aug 1, 2019

View reviewed changes

This was referenced Aug 16, 2019

New BSIP: coin-days as market fees (core token holders have a discount when paying market fees) #191

Open

BSIP-0043 further improvements: share taker market fees to maker's registrar and referrer #192

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BSIP67: Dynamic Market Fees #133

BSIP67: Dynamic Market Fees #133

OpenLedgerApp commented Dec 17, 2018 •

edited

Loading

pmconrad commented Jan 22, 2019

pmconrad Jan 22, 2019

OpenLedgerApp Jan 28, 2019

sschiessl-bcp Jul 29, 2019

pmconrad commented Jan 22, 2019

OpenLedgerApp commented Jan 29, 2019

ryanRfox commented Jan 30, 2019

pmconrad commented Jan 31, 2019

ryanRfox commented Jan 31, 2019

ryanRfox commented Jan 31, 2019

OpenLedgerApp commented Jan 31, 2019

pmconrad commented Feb 1, 2019

sschiessl-bcp commented Feb 21, 2019 •

edited

Loading

xeroc commented Feb 22, 2019

OpenLedgerApp commented Mar 11, 2019

abitmore commented Mar 12, 2019

OpenLedgerApp commented Mar 27, 2019 •

edited

Loading

ryanRfox commented Apr 3, 2019

pmconrad commented Apr 3, 2019

OpenLedgerApp commented Apr 19, 2019 •

edited

Loading

pmconrad commented Apr 24, 2019

OpenLedgerApp commented Apr 25, 2019

OpenLedgerApp commented Apr 29, 2019 •

edited

Loading

abitmore commented Jun 9, 2019

OpenLedgerApp commented Jul 26, 2019 •

edited

Loading

abitmore Jul 26, 2019

abitmore commented Jul 26, 2019

sschiessl-bcp commented Jul 29, 2019

pmconrad left a comment

BSIP67: Dynamic Market Fees #133

Are you sure you want to change the base?

BSIP67: Dynamic Market Fees #133

Conversation

OpenLedgerApp commented Dec 17, 2018 • edited Loading

pmconrad commented Jan 22, 2019

pmconrad Jan 22, 2019

Choose a reason for hiding this comment

OpenLedgerApp Jan 28, 2019

Choose a reason for hiding this comment

sschiessl-bcp Jul 29, 2019

Choose a reason for hiding this comment

pmconrad commented Jan 22, 2019

OpenLedgerApp commented Jan 29, 2019

ryanRfox commented Jan 30, 2019

pmconrad commented Jan 31, 2019

Risks

CPU Overhead

RAM Overhead

ryanRfox commented Jan 31, 2019

ryanRfox commented Jan 31, 2019

OpenLedgerApp commented Jan 31, 2019

pmconrad commented Feb 1, 2019

sschiessl-bcp commented Feb 21, 2019 • edited Loading

xeroc commented Feb 22, 2019

OpenLedgerApp commented Mar 11, 2019

abitmore commented Mar 12, 2019

OpenLedgerApp commented Mar 27, 2019 • edited Loading

ryanRfox commented Apr 3, 2019

pmconrad commented Apr 3, 2019

OpenLedgerApp commented Apr 19, 2019 • edited Loading

pmconrad commented Apr 24, 2019

OpenLedgerApp commented Apr 25, 2019

OpenLedgerApp commented Apr 29, 2019 • edited Loading

abitmore commented Jun 9, 2019

OpenLedgerApp commented Jul 26, 2019 • edited Loading

abitmore Jul 26, 2019

Choose a reason for hiding this comment

abitmore commented Jul 26, 2019

sschiessl-bcp commented Jul 29, 2019

pmconrad left a comment

Choose a reason for hiding this comment

OpenLedgerApp commented Dec 17, 2018 •

edited

Loading

sschiessl-bcp commented Feb 21, 2019 •

edited

Loading

OpenLedgerApp commented Mar 27, 2019 •

edited

Loading

OpenLedgerApp commented Apr 19, 2019 •

edited

Loading

OpenLedgerApp commented Apr 29, 2019 •

edited

Loading

OpenLedgerApp commented Jul 26, 2019 •

edited

Loading