Consolidating booster-http endpoints

[jacobheun]

The goal of this is to clarify and create consistency with the initial endpoints supported for booster-http now that we've done a couple iterations of functionality. This is a followup of #946 and #888.

Endpoint Overview

A user has a GIF stored on IPFS, it’s 1.9MB and totals 9 blocks. Their GIF has been aggregated with other content in a car file via an Aggregator and stored on Filecoin.

$ ipfs dag stat {cid}
Size: 1907628, NumBlocks: 9

The user, and relevant stakeholders (SP’s) may retrieve the data via the following methods:

Method 1. The user wants to get their gif back via the gateway as is, they can curl the CID without headers. This is a trusted return as you cannot verify the data until you have all of it. This is also subject to IPLD codec deserialization and could fail if the codec is unknown. In which case, the user may wish to get the CAR.

$ curl http://{booster-http.url}/ipfs/{cid} --output cool.gif

Method 2. If the user wants their data back in a CAR and handle IPLD deserialization themselves, they can add the car format in the header. This is particularly useful if Boost doesn't know the IPLD codec of the data:

$ curl -H "Accept:application/vnd.ipld.car;" http://{booster-http.url}/ipfs/{cid} --output cool.gif.car

Method 3. If the user wants their data back block by block, they can retrieve and parse the first block of the GIF with the raw ipld format. This would be useful if the user wants to iterate over the blocks, but is likely a low priority use case for Filecoin. It could be supported to match the IPFS Trustless gateway spec though.

$ curl -H "Accept:application/vnd.ipld.raw;" http://{booster-http.url}/ipfs/{cid} --output {cid}.block

Method 4. In Filecoin we also have Pieces, which is a padded car file of the original deal. Users may wish to retrieve the whole piece byte stream to reduce risk of incompatible commp, such as when replicating deals.

• Update booster-http to version 0.2.0 #970

$ curl http://{booster-http.url}/piece/{cid} --output {cid}.piece

If the user doesn’t want the piece, and just wants their unpadded car file, they can access it via the /ipfs endpoint mentioned in method 2 above.

Quick Summary

This would leave us with 2 endpoints, /ipfs and /piece.

• /ipfs/{cid} ⇒ Results in the deserliazed ipld dag, ie your GIF.
• /ipfs/{cid}?format=car⇒ A car stream of the serialized ipld structure. Can also be used to get the Deal Root Payload
• /ipfs/{cid}?format=raw ⇒ Results in the raw block of the cid requested and ONLY that block.
• /piece/{cid}?format=piece ⇒ The full, padded Piece. (cid.piece) Format is always piece and does not need to be specified.

Note: As shown in the above, abbreviated urls, the query param format can be used instead of the content type headers. This matches behavior of the IPFS gateway spec.

[jacobheun]

After some discussions I've updated method 4 & 5 to clarify the output format of piece retrieval.

The main diff here from current functionality is that in the new version you won't be able to use a PieceCid to request the underlying payload car. You will also not be able to use a payloadCid to return the piece. If the former needed (payload by pieceCid), we can add that, and I think the most likely place would be to extend method 2 with a pieceCid query param and dropping the cid path.

[jacobheun]

Removing /piece?payloadCID={cid}&format=piece as it's nondeterministic and not particularly useful.

I will also need to add some clarity about SP's being able to enable/restrict /ipfs as that may not be useful to everyone.

[LexLuthr]

After some discussions I've updated method 4 & 5 to clarify the output format of piece retrieval.

The main diff here from current functionality is that you cannot currently use a PieceCid to request the underlying payload car. If this is needed, we can add that, and I think the most likely place would be to extend method 2 with a pieceCid query param and dropping the cid path.

@jacobheun I am not sure if this is correct or maybe I am misreading it.

We have /piece?pieceCid=bagaSomePieceCID&format=car which allows us to download a car file using a Piece CID.

[jacobheun]

We have /piece?pieceCid=bagaSomePieceCID&format=car which allows us to download a car file using a Piece CID.

Correct, but the idea is to get rid of that and consolidate to just returning the full piece. The language is a bit off there, I'll correct it, but basically we're saying you can only ever get the full, padded piece going forward unless there's a clear case for the payload. Spade, for example, will only ever use the full piece, and, will request it via its Piece cid.

[LexLuthr]

@jacobheun I think we are good to clone this issue. We can still use this as reference for /ipfs implementations in the future.

[jacobheun]

@LexLuthr let's leave it for now. I want to add some additional context around access controls for /ipfs endpoints. We can spin that out in a separate issue for implementation and close this then.