WIP: The DAG API

[daviddias]

WIP, just adding notes and discussion here so that all parts are connected and nothing gets dropped

• Review and agree on the DAG API spec
  • dag.put
  • dag.get
  • dag.resolve
  • dag.rm
• Write down a series of examples to show it is capabilities
  • Resolving through formats
  • more? Any real case that we can grab directly and show the 'magic'?
• Agree on https://github.com/ipfs/interface-ipfs-core/issues/81 (really important) for internals
• Implementation
  • js-ipfs WIP: The DAG API #99
  • js-ipfs-api
  • http-api in go

[daviddias]

I want to bring to the DAG API a way to manipulate IPLD objects through paths (aka Patch) and also, have a way to have workbenches where a lot of manipulations happen on a DAG, where checkpoints can be commited, reverted and `flushed.

More on the patch idea and discussion here

On the workbenches proposal, what I'm thinking is something along the lines of:

const wb = ipfs.dag.bench()
wb.put..
wb.put..
wb.patch..
const c1 = wb.commit.. // commits a root by pinCaching all of its nodes
...
const c2 = wb.commit.. // 
...
wb.revert(c1) // revert to commit c1
.. 
wb.flush(rootOfTheDag)

This is something similar to what unixfs-engine is doing, what @whyrusleeping has done with DAG Builder and what @wanderer has done with https://github.com/wanderer/merkle-trie. Would love to get everyone's input, it is something super useful for DAG manipulation that a lot of other devs will have to reinvent if not available through the API. This will also enable IPFS to make smarter choices when it comes to providing (delayed providing), in memory caching (with watermarks) and so on.

First steps:

• Enumerate the needs/concerns DAGBuilder, unixfs-engine and merkle-trie have, see what are those in common.
• Agree on the verbs for this API

[wanderer]

what will put and patch do differently?

[wanderer]

also i wouldn't recommend the commit and revert mechanism, it can get really messy when working with it. I'm leaning to a copy method.

trie.put([path], value1)
trie2 = trie.copy()
trie.put([path], value2)
// trie2 doesn't have the second val
trie2.flush()

It is also nice because you can lazily copy the trie so its memory efficient

[daviddias]

@wanderer

put - store a dag node (just this)
patch - given a CID, a path and a value, sets that value on the key represented by that path and bubbles up the changes to the root node (the CID passed), returning the new CID root

[wanderer]

so if I have a vertex v1 and I want to create an edge e1 from it to v2, would I use v1.put(e1, v2)? or patch?

[daviddias]

So far, with what is written (as in: please feel free to improve the API even if it has to be redone from scratch, just make sure it still works within the constraints/needs we have)

// using put
v1.linkToOtherNode = { '/' : cid(v2) }
ipfs.dag.put(v1)

// using patch
ipfs.dag.patch(v1, 'linkToOtherNode', { '/': cid(v2) })

[wanderer]

@diasdavid ok i see, i keep coming at this from the point of view that the graph operation are methods on the vertices.

Would the ipfs.dag.bench only be writing operations? or would it also have a get and a fetching equivalent to patch? Would it include more advanced graph operation such as finding the leafs in the graph? and graph traversals?

[daviddias]

@wanderer yeah, the idea o a bench is to create a workspace where nodes can be created and retrieved, creating a new DAG, but with the different that things only get flushed to disk under an explicit order, instead of per operation.

Would it include more advanced graph operation such as finding the leafs in the graph? and graph traversals?

We can expand the DAG API to have those features, yes. I believe @nicola has some ideas here

[ianopolous]

We also do this kind of patching when building our merkle-btree which is the core, network visible, data structure in Peergos.

[wanderer]

@diasdavid so i was digging around js-cid and when you do something like v1.linkToOtherNode = { '/' : cid(v2) } the cid needs to hash v2. We really want to avoid double hashing to be preformant. Is there a way to modify the api so that we only need to store references in the ipfs.dag and only do hashing when flush is called?

[daviddias]

I'm going to break this PR into 3 PR

• DAG API basics (get, put, rm) feat: DAG API basics (get, put, rm) #112
• DAG resolve API feat: dag.resolve API #113
• DAG workbench API feat: dag.workbench API #114