Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

--addrindex doesn't work on preprocessed blocks #303

Closed
manan19 opened this issue Feb 17, 2015 · 5 comments
Closed

--addrindex doesn't work on preprocessed blocks #303

manan19 opened this issue Feb 17, 2015 · 5 comments

Comments

@manan19
Copy link

manan19 commented Feb 17, 2015

Generating address index with --addrindex option succeeds when processing blocks from scratch.

But generating the address index on a fully synced node throws the following errors -

11:40:46 2015-02-17 [ERR] ADXR: Couldn't get referenced txOut (cf2ab4e635099dfc8f2c0438c2d7eb6fdbc57ce3c7d497869e1466f03f872d61:0): <nil>

To reproduce bug, drop the address index from a fully synced node and then try to rengenerate.

$ btcd --testnet --dropaddrindex --rpclisten=127.0.0.1:18334 --listen=127.0.0.1:18555
11:38:37 2015-02-17 [INF] BTCD: Version 0.9.0-beta
11:38:37 2015-02-17 [INF] BTCD: Loading block database from '/Users/manan/Library/Application Support/Btcd/data/testnet/blocks_leveldb'
11:38:37 2015-02-17 [INF] BTCD: Block database loaded with block height 323335
11:38:37 2015-02-17 [INF] BTCD: Deleting entire addrindex.
11:39:12 2015-02-17 [INF] BTCD: Successfully deleted addrindex, exiting

$ btcd --testnet --addrindex --rpclisten=127.0.0.1:18334 --listen=127.0.0.1:18555
11:40:32 2015-02-17 [INF] BTCD: Version 0.9.0-beta
11:40:32 2015-02-17 [INF] BTCD: Loading block database from '/Users/manan/Library/Application Support/Btcd/data/testnet/blocks_leveldb'
11:40:32 2015-02-17 [INF] BTCD: Block database loaded with block height 323335
11:40:32 2015-02-17 [WRN] AMGR: Skipping bound address: address 127.0.0.1 is not routable
11:40:32 2015-02-17 [INF] BMGR: Generating initial block node index.  This may take a while...
11:40:33 2015-02-17 [INF] BMGR: Block index generation complete
11:40:33 2015-02-17 [INF] SRVR: Server listening on 127.0.0.1:18555
11:40:33 2015-02-17 [INF] ADXR: Building up address index from height 0 to 323335.
11:40:33 2015-02-17 [INF] RPCS: RPC server listening on 127.0.0.1:18334
11:40:33 2015-02-17 [INF] AMGR: Loaded 1742 addresses from file '/Users/manan/Library/Application Support/Btcd/data/testnet/peers.json'
11:40:33 2015-02-17 [INF] DISC: 6 addresses found from DNS seed testnet-seed.bitcoin.petertodd.org
11:40:33 2015-02-17 [INF] DISC: 1 addresses found from DNS seed testnet-seed.bluematt.me
11:40:34 2015-02-17 [INF] DISC: 41 addresses found from DNS seed testnet-seed.bitcoin.schildbach.de
11:40:43 2015-02-17 [INF] ADXR: Indexed addresses of 20539 blocks in the last 10s (31844 transactions, height 20538, 2012-08-18 16:34:20 -0700 PDT)
11:40:43 2015-02-17 [INF] BMGR: New valid peer 78.46.18.137:18333 (outbound) (/Satoshi:0.8.3/)
11:40:43 2015-02-17 [INF] BMGR: Syncing to block height 323335 from peer 78.46.18.137:18333
11:40:46 2015-02-17 [ERR] ADXR: Couldn't get referenced txOut (cf2ab4e635099dfc8f2c0438c2d7eb6fdbc57ce3c7d497869e1466f03f872d61:0): <nil>
11:40:46 2015-02-17 [ERR] ADXR: Couldn't get referenced txOut (cf2ab4e635099dfc8f2c0438c2d7eb6fdbc57ce3c7d497869e1466f03f872d61:1): <nil>
11:40:48 2015-02-17 [ERR] ADXR: Couldn't get referenced txOut (32638ada91a0d21a4236ceb824a32bc18736c5cd8b466222ac822c359db22d61:0): <nil>
11:40:49 2015-02-17 [ERR] ADXR: Couldn't get referenced txOut (32638ada91a0d21a4236ceb824a32bc18736c5cd8b466222ac822c359db22d61:1): <nil>
11:40:53 2015-02-17 [INF] ADXR: Indexed addresses of 37753 blocks in the last 10.23s (65137 transactions, height 58291, 2013-03-13 00:38:29 -0700 PDT)
11:40:53 2015-02-17 [INF] BMGR: New valid peer 54.210.107.2:18333 (outbound) (/Satoshi:0.9.2.1/)
11:41:02 2015-02-17 [ERR] ADXR: Couldn't get referenced txOut (58e04fb6fd999ec59ce4b6d3716affb37967028b990883c7bd70a4e9b6392d61:0): <nil>
11:41:02 2015-02-17 [ERR] ADXR: Couldn't get referenced txOut (58e04fb6fd999ec59ce4b6d3716affb37967028b990883c7bd70a4e9b6392d61:1): <nil>
11:41:03 2015-02-17 [INF] DISC: DNS discovery failed on seed testnet-seed.alexykot.me: lookup testnet-seed.alexykot.me: no such host
11:41:03 2015-02-17 [INF] ADXR: Indexed addresses of 8335 blocks in the last 10s (52082 transactions, height 66626, 2013-04-14 12:19:30 -0700 PDT)
11:41:03 2015-02-17 [INF] BMGR: New valid peer 54.237.211.7:18333 (outbound) (/Satoshi:0.9.1/)

@Roasbeef
Copy link
Member

Using the getrawtransaction RPC, are you able to successfully query any of those txid's?

@jrick
Copy link
Member

jrick commented Feb 17, 2015

By that error message, this looks like a non-nil interface to a nil concrete type.

@manan19
Copy link
Author

manan19 commented Feb 17, 2015

@Roasbeef No.

$ btcctl --rpcserver=localhost:18334 getrawtransaction cf2ab4e635099dfc8f2c0438c2d7eb6fdbc57ce3c7d497869e1466f03f872d61 --notls
rpcCommand: -5: No information available about transaction

$ btcctl --rpcserver=localhost:18334 getrawtransaction 32638ada91a0d21a4236ceb824a32bc18736c5cd8b466222ac822c359db22d61 --notls
rpcCommand: -5: No information available about transaction

@Roasbeef
Copy link
Member

@manan19 looks like a case of block corruption caused by goleveldb, similar to: #199. For now, you'll have to nuke your DB and start over with a fresh sync. In the near future we'll be moving away from storing blocks within leveldb. We'll instead store blocks as flat files which should significantly increase performance, and avoid errors like this tied to the implementation of goleveldb

Also the logging error message here isn't as clear as it should be. It currently couples an error returned by the db during lookup, with an alternate case of db not finding anything for the particular query. I'll have a patch soon to separate the cases into distinct error messages, which should help with future debugging.

@davecgh
Copy link
Member

davecgh commented Feb 24, 2015

Closing this as the issue is what @Roasbeef described.

@davecgh davecgh closed this as completed Feb 24, 2015
aakselrod added a commit to aakselrod/btcd that referenced this issue Mar 20, 2015
Fix btcsuite#303 by changing the addrindex key prefix to 3 characters so that
it's easy to check length when dropping the index. To drop the old
index, check to make sure we aren't dropping any entries that end in
"sx" or "tx" as those aren't part of the addrindex.

Fix btcsuite#346 by changing the pointers in the mempool's addrindex map to
wire.ShaHash 32-byte values. This lets them be deleted even if the
transaction data changes places in memory upon expanding the maps.

Change the way addrindex uint32s are stored to big-endian in order to
sort the transactions on disk in chronological/dependency order.

Change the "searchrawtransactions" RPC call to return transactions
from the database before the memory pool so that they're returned in
order. This commit DOES NOT do topological sorting of the memory pool
transactions to ensure they're returned in dependency order. This may
be a good idea for a future enhancement.

Add addrindex versioning to automatically drop the old/incompatible
version of the index and rebuild with the new sort method and key
prefix.
aakselrod added a commit to aakselrod/btcd that referenced this issue Mar 20, 2015
Fix btcsuite#303 by changing the addrindex key prefix to 3 characters so that
it's easy to check length when dropping the index. To drop the old
index, check to make sure we aren't dropping any entries that end in
"sx" or "tx" as those aren't part of the addrindex. Update test to
deal with the new prefix length.

Fix btcsuite#346 by changing the pointers in the mempool's addrindex map to
wire.ShaHash 32-byte values. This lets them be deleted even if the
transaction data changes places in memory upon expanding the maps.

Change the way addrindex uint32s are stored to big-endian in order to
sort the transactions on disk in chronological/dependency order.

Change the "searchrawtransactions" RPC call to return transactions
from the database before the memory pool so that they're returned in
order. This commit DOES NOT do topological sorting of the memory pool
transactions to ensure they're returned in dependency order. This may
be a good idea for a future enhancement.

Add addrindex versioning to automatically drop the old/incompatible
version of the index and rebuild with the new sort method and key
prefix.
aakselrod added a commit to aakselrod/btcd that referenced this issue Mar 23, 2015
Fix btcsuite#303 by changing the addrindex key prefix to 3 characters so that
it's easy to check length when dropping the index. To drop the old
index, check to make sure we aren't dropping any entries that end in
"sx" or "tx" as those aren't part of the addrindex. Update test to
deal with the new prefix length.

Fix btcsuite#346 by changing the pointers in the mempool's addrindex map to
wire.ShaHash 32-byte values. This lets them be deleted even if the
transaction data changes places in memory upon expanding the maps.

Change the way addrindex uint32s are stored to big-endian in order to
sort the transactions on disk in chronological/dependency order.

Change the "searchrawtransactions" RPC call to return transactions
from the database before the memory pool so that they're returned in
order. This commit DOES NOT do topological sorting of the memory pool
transactions to ensure they're returned in dependency order. This may
be a good idea for a future enhancement.

Add addrindex versioning to automatically drop the old/incompatible
version of the index and rebuild with the new sort method and key
prefix.
aakselrod added a commit to aakselrod/btcd that referenced this issue Mar 23, 2015
Fix btcsuite#303 by changing the addrindex key prefix to 3 characters so that
it's easy to check length when dropping the index. To drop the old
index, check to make sure we aren't dropping any entries that end in
"sx" or "tx" as those aren't part of the addrindex. Update test to
deal with the new prefix length.

Fix btcsuite#346 by changing the pointers in the mempool's addrindex map to
wire.ShaHash 32-byte values. This lets them be deleted even if the
transaction data changes places in memory upon expanding the maps.

Change the way addrindex uint32s are stored to big-endian in order to
sort the transactions on disk in chronological/dependency order.

Change the "searchrawtransactions" RPC call to return transactions
from the database before the memory pool so that they're returned in
order. This commit DOES NOT do topological sorting of the memory pool
transactions to ensure they're returned in dependency order. This may
be a good idea for a future enhancement.

Add addrindex versioning to automatically drop the old/incompatible
version of the index and rebuild with the new sort method and key
prefix.
jcvernaleo added a commit to jcvernaleo/btcd that referenced this issue Sep 8, 2016
jcvernaleo added a commit to jcvernaleo/btcd that referenced this issue Sep 8, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants