Skip to content
This repository has been archived by the owner on Jun 20, 2024. It is now read-only.

meta: GRAPH_BACKEND fixes and latency improvements #88

Closed
lidel opened this issue Apr 10, 2023 · 4 comments
Closed

meta: GRAPH_BACKEND fixes and latency improvements #88

lidel opened this issue Apr 10, 2023 · 4 comments
Assignees

Comments

@lidel
Copy link
Collaborator

lidel commented Apr 10, 2023

This is a meta issue for tracking things we can do to improve correctness/latency.

Spec conformance

### Tasks
- [x]   [IPIP-402](https://github.com/ipfs/specs/pull/402) conformance  tests for `dag-scope=block|entity|all`  →  [gateway-conformance#75 added tests](https://github.com/ipfs/gateway-conformance/pull/56)
- [x]   [IPIP-402](https://github.com/ipfs/specs/pull/402) conformance tests for `entity-bytes=from:to` →  [gateway-conformance tests](https://github.com/ipfs/gateway-conformance/pull/56) 
- [x]   [IPIP-402](https://github.com/ipfs/specs/pull/402) conformance tests for `entity-bytes` with missing block scenario to confirm `/ipfs/cid?format=car&bytes=from:to` works fast for big UnixFS file, and does not block until the entire thing is prefetched to some L1 →  [gateway-conformance#85 added tests](https://github.com/ipfs/gateway-conformance/pull/85)
- [x] Run gateway-conformance on Saturn CI: https://github.com/filecoin-saturn/L1-node/issues/447
- [x]  Saturn L1 pass `dag-scope` tests  [PR](https://github.com/filecoin-saturn/L1-node/pull/449)
- [ ] **(TODO)** Saturn L1 DOES NOT pass `entity-bytes` range request tests  [CI fails here](https://github.com/filecoin-saturn/L1-node/pull/449)

Perf. Improvements

### Tasks
- [ ] https://github.com/ipfs/specs/pull/402
- [ ] https://github.com/ipfs/specs/pull/412
- [x] **(PRECONDITION)** @aschmahmann IPIP-402 implementation in Boxo (https://github.com/ipfs/boxo/issues/283) so can verify tests and performance improvement ideas without doing a Rhea bifrost-gateway deployment.
- [x] **(OPTIMIZATION)** @aschmahmann https://github.com/ipfs/bifrost-gateway/pull/160
- [x] **(OPTIMIZATION)** @aschmahmann  implement https://github.com/ipfs/bifrost-gateway/issues/62
- [x] **(FIX)** ensure we use HTTP/2 when possible (#98  or https://github.com/ipfs/bifrost-gateway/pull/95)
- [x] **(FIX)** @lidel correctness degrading over time (https://github.com/ipfs/bifrost-gateway/issues/92)
- [x] **(OPTIONAL LATENCY FIX)** @lidel  / @hacdias / @aschmahmann   30m timeout for CARs is not enough, implement timeout-since-last-bytes-received that drops request/response processing if no new bytes were sent by L1 for [tbd] amount of time – stub in https://github.com/ipfs/bifrost-gateway/issues/91
- [ ] **(OPTIONAL CORRECTNESS FIX)** @lidel /  @aarshkshah1992   (content routing optimization, may improve correctness) when fetching path  remainders or single blocks, pass content path affinity to L1 and forward to Lassie in Ipfs-Path-Affinity HTTP header as a hint to augument content routing in situations where internal blocks are not announced on DHT/IPNI
- [ ] **(OPTIONAL LATENCY FIX)** (perf optimization) make caboose do more [RAPIDE-like retrieval](https://www.notion.so/MTB-WG-Meeting-6-RAPIDE-4ceba2dfb2fc4ceaa49f1cd566ca104f) and split DAG into parallel fetches?

Note: these things aren't blocking for enabling GRAPH_BACKEND=true. The blocker is #92

@BigLep
Copy link

BigLep commented May 9, 2023

2023-05-09 conversation: @BigLep to create the issue for implementing IPIP in Boxo. Needed for:

  • Conformance Tests
  • CAR gateway implementation vetting
  • Boost (since use Boxo gateway)

@lidel
Copy link
Collaborator Author

lidel commented May 15, 2023

@BigLep
Copy link

BigLep commented Jun 22, 2023

2023-06-22 maintainer conversation: @aschmahmann is going to update the content of this issue to describe what's remaining that we're going to do.

@BigLep
Copy link

BigLep commented Oct 3, 2023

2023-10-03 conversation: we did all the mandatory things.

@BigLep BigLep closed this as completed Oct 3, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants