blockchain: rolling merkle root

[cfromknecht]

This PR adds a new RollingMerkleTree type for efficiently computing merkle roots. The leaves
of the tree are iteratively pushed onto the tree, and any complete substrees are evaluated and
stored so that subsequent pushes may consume them when their sibling tree is complete. As a
result, this requires O(log n) additional memory be allocated. The current BuildMerkleTreeStore
allocates an array of up to 4 times the number of leaves, which is used to store internal nodes and the root.

For 2000 leaves, the benchmarks show that this method is about 7% slower (~90 microseconds), but uses 99.04% less memory and 99.85% fewer allocations. These allocations reduce gc pressure, especially during IBD when the large slices of hashes currently used are almost immediately discarded.

BenchmarkMerkle/rolling-1000-8              2000            694382 ns/op             858 B/op          3 allocs/op
BenchmarkMerkle/rolling-2000-8              1000           1367763 ns/op             923 B/op          3 allocs/op
BenchmarkMerkle/rolling-4000-8               500           2900103 ns/op            1210 B/op          6 allocs/op
BenchmarkMerkle/rolling-8000-8               300           5799806 ns/op            2034 B/op         10 allocs/op
BenchmarkMerkle/rolling-16000-8              100          11572403 ns/op            1578 B/op          3 allocs/op
BenchmarkMerkle/rolling-32000-8               50          22949069 ns/op            1720 B/op          5 allocs/op
BenchmarkMerkle/old-1000-8                  2000            655435 ns/op           48416 B/op       1002 allocs/op
BenchmarkMerkle/old-2000-8                  1000           1277192 ns/op           96800 B/op       2002 allocs/op
BenchmarkMerkle/old-4000-8                   500           2533821 ns/op          193568 B/op       4002 allocs/op
BenchmarkMerkle/old-8000-8                   300           5029310 ns/op          387104 B/op       8002 allocs/op              
BenchmarkMerkle/old-16000-8                  200          10135631 ns/op          774176 B/op      16002 allocs/op                     
BenchmarkMerkle/old-32000-8                  100          20442194 ns/op         1548340 B/op      32002 allocs/op

alternative to #1377

[Roasbeef]

Wouldn't this be better served as a slice? So we just index in rather than having the overhead of a map. We'd also gain memory locality for all the items as well.

[Roasbeef]

👍

[jakesylvestre]

@jcvernaleo (as per #1530)

• Low priority
• Enhancement
• Duplicate (of blockchain: add new rolling merkle root computation to reduce allocations during IBD #1377, not sure if we have a tag for this)

[Rjected]

@cfromknecht mind rebasing so the actions can run?

[Roasbeef]

Superseded by #1979

[moza88]

I noticed BuildMerkleTreeStore is still in the codebase even though it's replaced with RollingMerkleTree and CalcMerkleRoot. Should we remove it from blockchain/merkle.go and blockchain/merkle_test.go?