Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add wallet api get_account_history_by_operations for exchanges #430

Merged
merged 11 commits into from
Dec 20, 2017
Merged

add wallet api get_account_history_by_operations for exchanges #430

merged 11 commits into from
Dec 20, 2017

Conversation

secp256
Copy link

@secp256 secp256 commented Oct 19, 2017

get_account_history_by_operations returns the most recent operations on the named account. txID will be included in returned transactions.
parameters:account_name_or_id [operation ids] start limit_num

operation ids species the operation type you want to retrieve, for example, [0] stands for transfer operation.

cli_wallet call:

new >>> get_account_history_by_operations yun-bts [] 1 10
total_count : 10 
result_count : 10 
2015-11-06T00:16:15 Transfer 0.00010000 BTC from hpyhacking to yun-bts -- Unlock wallet to see memo.   (Fee: 0.00032999 BTC) transaction_id : cb29b5dfab92c735b49c7d61523a10002c87c09f 
2015-11-06T00:15:24 Transfer 1.1000 CNY from laomao to yun-bts -- Unlock wallet to see memo.   (Fee: 0.8462 CNY) transaction_id : ac592467762f90f34edaaa7fcfc08f7ef92341b0 
2015-11-06T00:10:48 Transfer 0.2000 USD from hpyhacking-ex to yun-bts -- Unlock wallet to see memo.   (Fee: 0.1334 USD) transaction_id : cab7a23568bb75faa86d6d1e68ca59cc3e4b9c90 
2015-11-05T23:58:57 Transfer 0.1000 USD from laomao to yun-bts -- Unlock wallet to see memo.   (Fee: 0.1336 USD) transaction_id : e20ee10ef99917f33fe20c60cbf35813024ef666 
2015-11-05T23:58:09 Transfer 0.1000 USD from hpyhacking-ex to yun-bts -- Unlock wallet to see memo.   (Fee: 0.1336 USD) transaction_id : e1381bc266bcc7105b09de142d32b1201452d7ce 
2015-11-05T23:56:30 Transfer 18 BTS from laomao to yun-bts -- Unlock wallet to see memo.   (Fee: 41.79686 BTS) transaction_id : 9f3d08afbe4a3f7d36bfc0411b58ea2a1f330e11 
2015-11-05T21:37:48 account_upgrade_operation yun-bts fee: 20000 BTS transaction_id : b6fd4f88c21ec10741f3b235ef7584a18cd45845 
2015-11-05T21:37:33 Transfer 10000 BTS from hpyhacking to yun-bts -- Unlock wallet to see memo.   (Fee: 42.10936 BTS) transaction_id : 8f947605f9ef66c95cec956a14ac84f5afa9c813 
2015-11-05T21:34:27 Transfer 13000 BTS from hpyhacking-ex to yun-bts -- Unlock wallet to see memo.   (Fee: 41.79686 BTS) transaction_id : 0ad35d6d74c6bfc06a21deb87ba6b5b5423be157 
2015-11-05T21:33:21 Create Account 'yun-bts'   (Fee: 190.29296 BTS) transaction_id : f3f06c56e5e7535c4cb606f8e788c8621deb0b02 

get_account_history_by_operations yun-bts [0] 1 10
new >>> get_account_history_by_operations yun-bts [0] 1 10
total_count : 10 
result_count : 8 
2015-11-06T00:16:15 Transfer 0.00010000 BTC from hpyhacking to yun-bts -- Unlock wallet to see memo.   (Fee: 0.00032999 BTC) transaction_id : cb29b5dfab92c735b49c7d61523a10002c87c09f 
2015-11-06T00:15:24 Transfer 1.1000 CNY from laomao to yun-bts -- Unlock wallet to see memo.   (Fee: 0.8462 CNY) transaction_id : ac592467762f90f34edaaa7fcfc08f7ef92341b0 
2015-11-06T00:10:48 Transfer 0.2000 USD from hpyhacking-ex to yun-bts -- Unlock wallet to see memo.   (Fee: 0.1334 USD) transaction_id : cab7a23568bb75faa86d6d1e68ca59cc3e4b9c90 
2015-11-05T23:58:57 Transfer 0.1000 USD from laomao to yun-bts -- Unlock wallet to see memo.   (Fee: 0.1336 USD) transaction_id : e20ee10ef99917f33fe20c60cbf35813024ef666 
2015-11-05T23:58:09 Transfer 0.1000 USD from hpyhacking-ex to yun-bts -- Unlock wallet to see memo.   (Fee: 0.1336 USD) transaction_id : e1381bc266bcc7105b09de142d32b1201452d7ce 
2015-11-05T23:56:30 Transfer 18 BTS from laomao to yun-bts -- Unlock wallet to see memo.   (Fee: 41.79686 BTS) transaction_id : 9f3d08afbe4a3f7d36bfc0411b58ea2a1f330e11 
2015-11-05T21:37:33 Transfer 10000 BTS from hpyhacking to yun-bts -- Unlock wallet to see memo.   (Fee: 42.10936 BTS) transaction_id : 8f947605f9ef66c95cec956a14ac84f5afa9c813 
2015-11-05T21:34:27 Transfer 13000 BTS from hpyhacking-ex to yun-bts -- Unlock wallet to see memo.   (Fee: 41.79686 BTS) transaction_id : 0ad35d6d74c6bfc06a21deb87ba6b5b5423be157 

walet api call:

$ curl --data '{"jsonrpc": "2.0", "method": "get_account_history_by_operations", "params": ["1.2.96993",[0], 1, 5], "id": 1}' http://127.0.0.1:8091/rpc

results:

{
  "id": 1,
  "jsonrpc": "2.0",
  "result": {
    "total_count": 5,
    "result_count": 3,
    "details": [{
      "memo": "",
      "description": "Transfer 18 BTS from laomao to yun-bts -- Unlock wallet to see memo.   (Fee: 41.79686 BTS)",
      "op": {
        "id": "1.11.196121",
        "op": [0, {
          "fee": {
            "amount": 4179686,
            "asset_id": "1.3.0"
          },
          "from": "1.2.23355",
          "to": "1.2.96993",
          "amount": {
            "amount": 1800000,
            "asset_id": "1.3.0"
          },
          "memo": {
            "from": "BTS7oS8BEunaYxXiDVabbKEoKfznGqMWdBSwHF81PegDiQbW3rqMD",
            "to": "BTS81JWvhxW5YQ52rDC2fzCddi3BopYrEtT2qza3qu55s7iFAjLLo",
            "nonce": "370372552766600",
            "message": "0cff9f0a37da1f8c8aef5bec4344a5ed"
          },
          "extensions": []
        }],
        "result": [0, {}],
        "block_num": 666734,
        "trx_in_block": 0,
        "op_in_trx": 0,
        "virtual_op": 31013
      },
      "transaction_id": "9f3d08afbe4a3f7d36bfc0411b58ea2a1f330e11"
    }, {
      "memo": "",
      "description": "Transfer 10000 BTS from hpyhacking to yun-bts -- Unlock wallet to see memo.   (Fee: 42.10936 BTS)",
      "op": {
        "id": "1.11.195209",
        "op": [0, {
          "fee": {
            "amount": 4210936,
            "asset_id": "1.3.0"
          },
          "from": "1.2.24166",
          "to": "1.2.96993",
          "amount": {
            "amount": 1000000000,
            "asset_id": "1.3.0"
          },
          "memo": {
            "from": "BTS6fphvFQZn6naZzF6G7pMH291X7KtB4VKDNm552XoXu2QuTp5F5",
            "to": "BTS81JWvhxW5YQ52rDC2fzCddi3BopYrEtT2qza3qu55s7iFAjLLo",
            "nonce": "370370416804876",
            "message": "b11e8bf248dfb18c1d14649f2d4c10adfedcef9423d77767628418762eb5e604"
          },
          "extensions": []
        }],
        "result": [0, {}],
        "block_num": 663961,
        "trx_in_block": 0,
        "op_in_trx": 0,
        "virtual_op": 30101
      },
      "transaction_id": "8f947605f9ef66c95cec956a14ac84f5afa9c813"
    }, {
      "memo": "",
      "description": "Transfer 13000 BTS from hpyhacking-ex to yun-bts -- Unlock wallet to see memo.   (Fee: 41.79686 BTS)",
      "op": {
        "id": "1.11.195189",
        "op": [0, {
          "fee": {
            "amount": 4179686,
            "asset_id": "1.3.0"
          },
          "from": "1.2.95938",
          "to": "1.2.96993",
          "amount": {
            "amount": 1300000000,
            "asset_id": "1.3.0"
          },
          "memo": {
            "from": "BTS6kRdAQcHdo7PxNzcGQXYth4k86EmpdJH8Q5KpGojAcu4SNdqvs",
            "to": "BTS81JWvhxW5YQ52rDC2fzCddi3BopYrEtT2qza3qu55s7iFAjLLo",
            "nonce": "370370369659146",
            "message": "9d8d6508130d129d56cbc5888fec9ef4"
          },
          "extensions": []
        }],
        "result": [0, {}],
        "block_num": 663899,
        "trx_in_block": 0,
        "op_in_trx": 0,
        "virtual_op": 30081
      },
      "transaction_id": "0ad35d6d74c6bfc06a21deb87ba6b5b5423be157"
    }]
  }
}

@oxarbitrage
Copy link
Member

the problem here is that you call get_relative_account_history with the limit from the new api get_account_history_by_operations. as operations pulled from get_relative_account_history does not match the one provided by the new call you can and will get less results than limit.

we tried this and several other methods to make this call and similar calls to resolve similar issues, they all have some flaw and we concluded to move account history from ram into a database(elasticsearch was selected) indexing everything to make all kind of searches directly against the database instead of creating and maintain new api calls for everything needed.

this is work in progress and a first version of this work will be posted very soon.

Copy link
Member

@abitmore abitmore left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The logic in this PR is OK. Only need a small change, we can merge it after it's done. There are implicit logic/limitations about the filter and limit param, best if the document is clearer.

@@ -369,7 +382,7 @@ class wallet_api
vector<call_order_object> get_call_orders(string a, uint32_t limit)const;
vector<force_settlement_object> get_settle_orders(string a, uint32_t limit)const;

/** Returns the collateral_bid object for the given MPA
/** Returns the most recent operations on the named account, with transaction id.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please revert this change.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. I have updated the document.

Copy link
Member

@abitmore abitmore left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good now. Thank you. @oxarbitrage please review.

@@ -112,16 +112,15 @@ namespace graphene { namespace app {
operation_history_id_type start = operation_history_id_type())const;

/**
* @brief Get operations relevant to the specificed account
* @brief Get operations relevant to the specificed account, return a list of operation history objects with transaction id
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Get operations relevant to the specificed account, return a list of operation history objects with transaction id"
should be something more like:
"Get operations relevant to the specified account filtering by operation type"
same everywhere.
the parameter name i think it should be "operation_types" instead of "operation_indexs" or "operation_ids".

rest looks good.

This code does not affect performance of the node so i don't have any reason for not to add except for the limit thing that is now specified as a max number ... still don't like much calling for 100 results and getting 1. much more better to do it with the elasticsearch plugin and wrapper as: http://209.188.21.157:5000/get_account_history?account_id=1.2.282&from=100&size=10&sort_by=-operation_type&operation_type=0

anyways, i am not going to oppose to it as there is not infrastructure for elasticsearch yet, if you guys think it is needed then lets make that small changes and add it.

thanks @gxchain123

Copy link
Member

@oxarbitrage oxarbitrage Dec 14, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

another limitation of elasticsearch is that the cli_wallet cant make calls into it yet. i think that in the future the cli will access the database for specific calls like this but not present atm. thats another argument to merge this code at least by now.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated.

@oxarbitrage
Copy link
Member

oxarbitrage commented Dec 16, 2017

Something to consider, i was testing the cli wallet call.

To make it i started a node following the history of 2 accounts xeroc and zdbz(bot).

Started node as:

programs/witness_node/witness_node --data-dir data/my-blockprod --rpc-endpoint "127.0.0.1:8090" --track-account "\"1.2.282"\" "\"1.2.126225"\"

I added a counter dump to the while on line https://github.com/gxchain123/bitshares-core/blob/e4fb550e38354da467997d1e8019072509ff86e9/libraries/wallet/wallet.cpp#L2848. when calling xeroc(account with less than 900 operations) some calls will return fast:

new >>> get_account_history_by_operations xeroc [0] 1 1
get_account_history_by_operations xeroc [0] 1 1
2653723ms th_a       wallet.cpp:2900               get_account_history_ ] count: 0 
total_count : 1 
result_count : 1 
2015-10-21T10:46:30 Transfer 60 BTS from bittrex-deposit to xeroc -- Unlock wallet to see memo.   (Fee: 41.79686 BTS) transaction_id : cbbc0a75068291fdc6c84da842cd621a67f06501 

new >>> 

In some others it needs to start looping throw the account history like:

new >>> get_account_history_by_operations xeroc [1] 1 1
get_account_history_by_operations xeroc [1] 1 1
2725277ms th_a       wallet.cpp:2900               get_account_history_ ] count: 0 
2725278ms th_a       wallet.cpp:2900               get_account_history_ ] count: 1 
2725279ms th_a       wallet.cpp:2900               get_account_history_ ] count: 2 
2725280ms th_a       wallet.cpp:2900               get_account_history_ ] count: 3 
2725281ms th_a       wallet.cpp:2900               get_account_history_ ] count: 4 
...
2725346ms th_a       wallet.cpp:2900               get_account_history_ ] count: 176 
2725346ms th_a       wallet.cpp:2900               get_account_history_ ] count: 177 
total_count : 178 
result_count : 1 
2015-11-16T16:39:06 limit_order_create_operation xeroc fee: 0.00011441 BTC   result: 1.7.4951 transaction_id : 7974380e383f4b4e2ca3d27df523c52204e6554f 

new >>> 

Now trying some calls against the other account i am following zdbz(3.5 millon ops):

new >>> get_account_history_by_operations zdbz [] 0 1
get_account_history_by_operations zdbz [] 0 1
2958953ms th_a       wallet.cpp:2900               get_account_history_ ] count: 0 
total_count : 1 
result_count : 1 
2017-12-16T01:49:09 limit_order_create_operation zdbz fee: 0.01213 BTS   result: 1.7.43159361 transaction_id : 9355cd30dd7c65f75763836a87388603c9701728 

new >>> 

new >>> get_account_history_by_operations zdbz [1] 0 1
get_account_history_by_operations zdbz [1] 0 1
2971137ms th_a       wallet.cpp:2900               get_account_history_ ] count: 0 
total_count : 1 
result_count : 1 
2017-12-16T01:49:09 limit_order_create_operation zdbz fee: 0.01213 BTS   result: 1.7.43159361 transaction_id : 9355cd30dd7c65f75763836a87388603c9701728 

new >>> 

all good until there but when:

new >>> get_account_history_by_operations zdbz [] 0 2
get_account_history_by_operations zdbz [] 0 2
3131631ms th_a       wallet.cpp:2900               get_account_history_ ] count: 32735 
3131632ms th_a       wallet.cpp:2900               get_account_history_ ] count: 32736 
3131632ms th_a       wallet.cpp:2900               get_account_history_ ] count: 32737
...
530613ms th_a       wallet.cpp:2900               get_account_history_ ] count: 4704802 
530614ms th_a       wallet.cpp:2900               get_account_history_ ] count: 4704803 
530614ms th_a       wallet.cpp:2900               get_account_history_ ] count: 4704804 
^C

the cli will get trapped in an infinite loop as account haves 3,5M operations and loop already at 4,7M.

we need to at least stop the loop when the count is greater than total_ops.

the cpu usage of the witness_node and the cli_wallet go increasing as the infinite loop is running, cli_wallet more drastically but witness_node also goes climbing.

what do you think @abitmore ?

i made a custom binary search some time ago for other feature that is more efficient(finds data in lot less iterations) than looping throw all the account, i think it can be adapted to this. code was never included to the node as elasticsearch plugin replaced it but it can be something good to have in the cli wallet.
Above not work as the scenario was different.

let me know what do you think guys, sorry to be so picky here but this are certain problems i already faced before.

@secp256
Copy link
Author

secp256 commented Dec 18, 2017

how about adding a counter to stop the loop ?

@oxarbitrage
Copy link
Member

thank you @zhuliting . yes, that will help, the same call on the 3.5 mill account now at least stops at some point:

new >>> get_account_history_by_operations zdbz [] 0 2
get_account_history_by_operations zdbz [] 0 2
232098ms th_a       wallet.cpp:2904               get_account_history_ ] count: 0 
232098ms th_a       wallet.cpp:2904               get_account_history_ ] count: 1 
232099ms th_a       wallet.cpp:2904               get_account_history_ ] count: 2 
...
533860ms th_a       wallet.cpp:2904               get_account_history_ ] count: 1820357 
total_count : 0 
result_count : 0 
new >>> 

however i still don't get why that request will need to loop throw the account so much. i am missing something or the logic is not very good in that cases ?

please note i am just asking to get all the op types from an account, all this calls will loop throw the account and i think it is not needed:

get_account_history_by_operations zdbz [] 0 2
get_account_history_by_operations zdbz [] 1 5
get_account_history_by_operations zdbz [] 10 2
... etc

@secp256
Copy link
Author

secp256 commented Dec 19, 2017

sorry for my mistake.

new >>> 
get_account_history_by_operations zdbz [] 1 5
new >>> get_account_history_by_operations zdbz [] 1 5
2229867ms th_a       wallet.cpp:2908               get_account_history_ ] counter: 3647495
2229880ms th_a       wallet.cpp:2908               get_account_history_ ] counter: 3647500
total_count : 3647500 
result_count : 5 
2017-12-19T06:40:51 limit_order_create_operation zdbz fee: 0.01213 BTS   result: 1.7.43577636 transaction_id : 31b57c89d434fd85358e29d29cdbcfd30f03b61b 
2017-12-19T06:40:51 limit_order_create_operation zdbz fee: 0.01213 BTS   result: 1.7.43577635 transaction_id : 180ba2aa17a1b7678bb10083e064787fab89ed42 
2017-12-19T06:40:51 limit_order_cancel_operation zdbz fee: 0.00121 BTS   result: 342.18333 BTS transaction_id : e3919c5653cd66a15272dcb2bb5abbb6fe8e7a14 
2017-12-19T06:40:51 limit_order_cancel_operation zdbz fee: 0.00121 BTS   result: 0.016537 OPEN.ETH transaction_id : e0c7e75ffa0cc883e464b2c9d0ebdfe8b74633fc 
2017-12-19T06:40:51 limit_order_create_operation zdbz fee: 0.01213 BTS   result: 1.7.43577636 transaction_id : 31b57c89d434fd85358e29d29cdbcfd30f03b61b 


new >>> 

new >>> 
get_account_history_by_operations zdbz [] 10 2
new >>> get_account_history_by_operations zdbz [] 10 2
2235078ms th_a       wallet.cpp:2908               get_account_history_ ] counter: 3647495
2235083ms th_a       wallet.cpp:2908               get_account_history_ ] counter: 3647497
total_count : 3647497 
result_count : 2 
2017-12-19T06:40:51 limit_order_cancel_operation zdbz fee: 0.00121 BTS   result: 0.016537 OPEN.ETH transaction_id : e0c7e75ffa0cc883e464b2c9d0ebdfe8b74633fc 
2017-12-19T06:40:51 limit_order_cancel_operation zdbz fee: 0.00121 BTS   result: 0.016537 OPEN.ETH transaction_id : e0c7e75ffa0cc883e464b2c9d0ebdfe8b74633fc


uint32_t counter = start;
while (limit > 0 && counter <= stats.total_ops) {
ilog("counter: ${c}", ("c", counter));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pls lets remove this line to merge. thx.

Copy link
Member

@oxarbitrage oxarbitrage left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good enough now @zhuliting. thank you for your work on this. lets just remove the counter ilog to merge.

@secp256
Copy link
Author

secp256 commented Dec 20, 2017

counter is redundant in current code

@oxarbitrage oxarbitrage merged commit f9af1b5 into bitshares:develop Dec 20, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants