feat: subtree support

[daviddias]

WIP

• Update README: update readme #168
• CID Support Needs to learn how to take in CIDs, not just multihashes #173
• Subtrees support Support for cat'ing and get'ing subtrees. #174
  • export('/ipfs/QmHash', ipldResolver)
  • export('/ipfs/QmHash/a', ipldResolver)
  • export('/ipfs/QmHash/b.txt', ipldResolver)
  • export('/ipfs/QmHash/a/b/c.txt', ipldResolver)
  • export('QmHash/a/b/c.txt', ipldResolver)
  • export('QmHash/a/b/c.txt', ipldResolver)
  • It also needs to support a subtree that starts on a non protobuf node, but the path leads to the start of a file that is a protobuf.

[pgte]

@diasdavid to support your last point (resolving from non-pb nodes), I had to implement a resolver for "unknown" file types.

Could you please take a look at https://github.com/ipfs/js-ipfs-unixfs-engine/blob/b446b0d219f902173c0ef389392a7e95c0fa1b59/src/exporter/unknown.js and see if there would be a better way of doing this?

[daviddias]

@pgte what is the logical difference between:

• /QmHashDAGPB/a/b/c.txt
• /QmHashDAGCBOR/a/b/c.txt

In terms of resolving? It should agnostically use the IPLD Resolver to traverse to the file and export it, correct?

Can you also make sure with tests the expected behaviour in js-ipfs is respected, namely:

(it might be already, I haven't checked)

[pgte]

@diasdavid the problem I found by trying to use the ipldResolver to try to resolve the whole path is that it hangs indefinitely when provided a path on the second argument of .get. I presume because it isn't able to resolve the whole path through IPLD, since IPLD doesn't speak "UnixFS".

I was hoping that the IPLD resolver would quit at the time it entered UnixFS territory and just return me the remainder path, but that hasn't happened..

Bug or am I using this wrong?

[pgte]

@diasdavid for your reference, here is the test I'm running:
https://github.com/ipfs/js-ipfs-unixfs-engine/blob/c060b53f1c0510179c5644a91ee37db8c087ca21/test/test-export-subtree.js#L62-L83

[pgte]

Having said that, with the exception of sharded dirs (where the link names do not correspond to dir entries), it should map pretty well though.. Any thoughts?

[daviddias]

I found by trying to use the ipldResolver to try to resolve the whole path is that it hangs indefinitely when provided a path on the second argument of .get. I presume because it isn't able to resolve the whole path through IPLD, since IPLD doesn't speak "UnixFS

Got it! Yeah, that is definitely the issue, it doesn't speak UnixFS so that paths don't map directly. Makes sense to have that unknown exporter. Could use a better name then, subTreeExporter?

Having said that, with the exception of sharded dirs

Now I'm unclear, you already got support for shared dirs, right? Using the UnknownExporter

#175 (comment)

Did you verify if you reproduce exactly the above with js-ipfs now?

[pgte]

@diasdavid paths into shared dirs is supported in this branch, but in theory, not natively by resolving through IPLD (because the link names are prefixed with the shard bucket number).

[pgte]

@diasdavid regarding the IPLD resolver not being able to resolve up until the file protobuf, I think we can live with it for now, since I'm resolving "by hand". I'll open an issue to solve this later.

[pgte]

@diasdavid other than that, I think this feature is ready to be merged.

[daviddias]

@diasdavid paths into shared dirs is supported in this branch, but in theory, not natively by resolving through IPLD (because the link names are prefixed with the shard bucket number).

That's the desired behavior

What about #175 (comment), did you npm link in js-ipfs and checked?

[pgte]

Looks like it works:

[daviddias]

@pgte exactly, not the different between #175 (comment) and #175 (comment). go-ipfs doesn't preserve the whole path when fetching a subtree, just the file name.

[pgte]

@diasdavid subtree path preserved, just like in go, given by 8dcdac6.

[daviddias]

awesome! :D