Improve Wal Replay

[joe-elliott]

What this PR does:
For simplicity the current WAL replay code loads a file, iterates through all records and replays them into an ingester instance. This creates two issues:

• We are doubling our disk needs for a brief time period while the WAL replays and rebuilds itself in a second file. In our internal ops cluster we have seen this lead to failed WAL replays.
• There is a lot of overhead to reading an entire multi GB file and rewriting it back to disk.

This proposed fix iterates through the WAL as is and simply rebuilds the index in memory for the AppendBlock to use directly instead of rebuilding a new file.

Which issue(s) this PR fixes:
Fixes #409

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]