Replace the Node type (which is a map) with an interface ?

[mildred]

Currently, the Node type is defined as :

type Node map[string]Node

This makes all IPLD users very dependent on the implementation of the Node object, and forces IPLD to parse everything on the JSON/CBOR data to put it in the map. The alternative is to provide only a handle on the data and decode it on demand, and let the Node consumer store just the things he needs.

Say an application just need some small keys on many Node objects which otherwise are also containing huge chunks of data. We can imagine a mode of operation where the huge data is left on the disk while the file is merely traversed to look for the small keys the application needs. The big data shouldn't need to be kept on memory if we don't use it.

To allow this, I'd propose to make Node an interface equipped with a walk function (that would walk the JSON-like data structure, not local paths as the current walk function does) :

type Node interface {
    func Walk(WalkFun walker) error;
}

And then:

const TokenInt = 1 << iota;
const TokenFloat;
const TokenString;
const TokenObjectKey;
const TokenArrayIndex;
const SkipObject error;
const SkipValue error;
type WalkFun fun(err error, path []interface{}, tokenType int, token interface{}) error;

An example of decoding the following JSON:

{
  a: 1,
  b [ "c", "d" ]
}

Would result in the walk function being called this way:

WalkFun(nil, []string{}, TokenObjectKey, "a");
WalkFun(nil, []string{"a"}, TokenInt, 1);
WalkFun(nil, []string{}, TokenObjectKey, "b");
WalkFun(nil, []string{"b"}, TokenArrayIndex, 0);
WalkFun(nil, []string{"b", 0}, TokenString, "c");
WalkFun(nil, []string{"b"}, TokenArrayIndex, 1);
WalkFun(nil, []string{"b", 1}, TokenString, "d");

This would be a very low level block that will be used to implement higher level functions, like path walking within an object, or decoding to application specific formats.

To provide compatibility with ipfs/merkledag, we could add to the interface a few functions:

type NodeMdagCompatible {
    Node;
    func Links() []Link;
    func Data() []byte;
}

The Links and Data function would traverse the object using the walk function to provide a list of links and the data bytes required for ipfs/merkledag users.

[jbenet]

• yes i think it makes sense to move to an interface for the future.
• i originally wanted a map so that subobjects counted as ipld nodes :/ but may just not work here.
• 👍 on argument re handing out an implementation of a Node that keeps data on disk
• 👍 on mdag compatibility type.
• i would want a Links(Node) func that walks the node and returns the map[string]Link. (ideally some node impls could cache this)
• Not yet sure about this walk func-- why must that be in the interface? easiest to allow most other impls?
• I still want a walk function based on fs paths (this is walking the json datastruct, but in a different way)
• could create functions (using the walk func) to turn any node impl into the map type, json, or other things.
• could have an impl tuned for creating/editing new dags (hold ptrs in links, defer hashing until finalizing dag or there is mem pressure, etc)

[mildred]

• i would want a Links(Node) func that walks the node and returns the map[string]Link. (ideally some node impls could cache this)

No problem for that

• Not yet sure about this walk func-- why must that be in the interface? easiest to allow most other impls?

The new Node interface should allow reading a node in one single pass. A walk function seems a good way to do that. What interface would you propose that would be compatible to JSON, CBOR, ..., that reads the data sequentially (allow the data structure not to fit entirely in memory) and can be used to convert to mostly anything else.

The idea is that the functions of this new Node interface would be used to convert the actual Node (no matter how it is implemented, JSON, CBOR ...) into another data structure that could be a map[string]interface{} or a merkledag.Node in its curent form, or a map[string]Link, or anything suitable for the application in use.

• I still want a walk function based on fs paths (this is walking the json datastruct, but in a different way)

I wasn't planning on removing it. As this is higher level, it would be implemented on top, or alongside the other functions.

• could create functions (using the walk func) to turn any node impl into the map type, json, or other things.
• could have an impl tuned for creating/editing new dags (hold ptrs in links, defer hashing until finalizing dag or there is mem pressure, etc)

This needs to be possible of course. A simple solution would be to use the map[string]interface{} representation for node creation. But that's not the only possibility. I think we should drive this interface by the use cases we find in go-ipfs in any case.

[mildred]

I am working on my fork (the break branch) on an implementation of that (and everything else). There, I created an implementation of the reader interface for JSON and for CBOR. Works great!

[jbenet]

Fantastic! :) 👍