Support incoming and outgoing duplicates

[hannahhoward]

Goals

Support github.com/ipfs/specs/pulls/412

Implementation

This adds two primary pieces of functionality:

1. Converting outgoing http to default to dups=y, and verifying incoming dups
2. Supporting parsing of IPIP-412 parameters and supporting dups=y on the HTTP server

The key adds are:

• VerifiedCar now supports expecting duplicates on the incoming stream, and it also supports optionally writing duplicates on the outgoing stream
• DuplicatesAddedCar is bit of an automagic class to take an outgoing car with no dups and add dups, by running a verification with dups loaded from the DeferredStorageCar (i.e. the temp blockstore)

There are still some critical race condition issues here relating to io.Pipe, and I need to figure out how to better mitigate them

However, I think this is the baseline architecture.

Also needs further integration tests but wanted to validate the general approach

[codecov-commenter]

Codecov Report

Merging #249 (c7fd62b) into main (fecb692) will increase coverage by 0.88%.
The diff coverage is 84.61%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #249      +/-   ##
==========================================
+ Coverage   69.87%   70.75%   +0.88%     
==========================================
  Files          68       71       +3     
  Lines        5918     6162     +244     
==========================================
+ Hits         4135     4360     +225     
- Misses       1541     1559      +18     
- Partials      242      243       +1     

Impacted Files	Coverage Δ
cmd/lassie/fetch.go	0.00% <0.00%> (ø)
pkg/internal/testutil/correctedmemstore.go	50.00% <50.00%> (ø)
pkg/server/http/ipfs.go	59.48% <72.72%> (+1.97%)	⬆️
pkg/internal/testutil/toblocks.go	76.31% <76.31%> (ø)
pkg/verifiedcar/verifiedcar.go	81.42% <82.00%> (+1.23%)	⬆️
pkg/storage/duplicateaddercar.go	94.17% <94.17%> (ø)
pkg/internal/itest/testpeer/generator.go	67.25% <100.00%> (+4.15%)	⬆️
pkg/internal/testutil/gen.go	93.33% <100.00%> (+0.39%)	⬆️
pkg/retriever/httpretriever.go	84.95% <100.00%> (+2.34%)	⬆️
pkg/storage/cachingtempstore.go	74.54% <100.00%> (+1.81%)	⬆️
... and 3 more

... and 4 files with indirect coverage changes

[hannahhoward]

I've now refactored some of the work to make it more flexible and added integration tests. I feel generally happy with where things are at though I am still working. I'm marking ready for review.

It may be that we need to seperate out CAR verifier into two classes based on the new lower level interface, but also there is going to be some duped code if we do that.

[rvagg]

wew, this really does need refactoring out to a separate "accept" header decoder func

[hannahhoward]

I actually went ahead and refactored all of the ipfs handler file, but then decided this should be a seperate PR cause... lots more changes.

[rvagg]

could also reject on unknown params, but it's not as clear case as with novel orders, this could be something to bring up in 412 - do we want to proactively reject unknown options so upgrade paths are clearer?

[hannahhoward]

yea, for now, not known

[rvagg]

:shakes-fist-at-sky:

[rvagg]

I think this is sane. I've only done an initial pass, I'm going to have to reflect on the approach but I think my biggest concern is around increasing the complexity of the already complex CAR layering: temporary, w/ preload support, deferred, streaming. I don't know if there's an easy way to reduce that complexity so maybe just more documentation around it would help.