Support agoric follow -o jsonlines style vstorage browsing in the agd api

[arirubinstein]

To assist with metrics processing on all vstorage nodes, implement an alternative route for vstorage to allow for post-processing of cap-encoded data.

E.g. today using agoric follow -o jsonlines

agoric --sdk follow :published.vaultFactory.manager0.metrics -B https://devnet.agoric.net/network-config --proof none -o jsonlines | jq .
{
  "liquidatingCollateral": {
    "brand": "[Alleged: SEVERED: IbcATOM brand {}]",
    "value": "0"
  },
  "liquidatingDebt": {
    "brand": "[Alleged: SEVERED: IST brand {}]",
    "value": "0"
  },
  "numActiveVaults": 8,
  "numLiquidatingVaults": 0,
  "numLiquidationsAborted": 0,
  "numLiquidationsCompleted": 0,
  "retainedCollateral": {
    "brand": "[Alleged: SEVERED: IbcATOM brand {}]",
    "value": "0"
  },
  "totalCollateral": {
    "brand": "[Alleged: SEVERED: IbcATOM brand {}]",
    "value": "4120000000"
  },
  "totalCollateralSold": {
    "brand": "[Alleged: SEVERED: IbcATOM brand {}]",
    "value": "0"
  },
  "totalDebt": {
    "brand": "[Alleged: SEVERED: IST brand {}]",
    "value": "20881726389"
  },
  "totalOverageReceived": {
    "brand": "[Alleged: SEVERED: IST brand {}]",
    "value": "0"
  },
  "totalProceedsReceived": {
    "brand": "[Alleged: SEVERED: IST brand {}]",
    "value": "0"
  },
  "totalShortfallReceived": {
    "brand": "[Alleged: SEVERED: IST brand {}]",
    "value": "0"
  }
}

expected future

https://devnet.api.agoric.net/agoric/custom/vstorage/data... plus post/query params for the vstorage path, to return json in the encoded value response similar to above

[gibson042]

Probably related: #5968

[dckc]

@gibson042 do you have details on the format you aim to produce? "[Alleged: SEVERED: IST brand {}]" is missing the boardID, which seems like to lead clients to using Alleged names for things other than debugging.

Is this design space any different / smaller than endojs/endo#1478 ?

[gibson042]

The primary consumers of this interface will be CLI and shell scripts, and its nature suggests (if not requires) opinionated lossy transformation of source data (answering your last question in the affirmative—lossiness differentiates the design space). But different consumers will have different needs, and the interface also needs configurability via some kind of request parameters to control each kind of JSON-incompatible data (slot/bigint/symbol/undefined/etc.) and also for how to represent the sequence of structured items (JSON Lines/JSON array/CSV/etc. at the top level, deep JSON/flat JSON/etc. at the record level).

I'll consider the issue resolved when the initial/primary use case (metrics consumption) is satisfied and the path to supporting others with different rendering options is clear, even if the first pass doesn't actually support them yet.

[gibson042]

Well, this has become quite a journey... putting aside the other issues discovered during this work and focusing on the specific desire here, it seems to be the case that returning arbitrary HTTP content is not supported—everything is at minimum wrapped in JSON-RPC like { "jsonrpc": "2.0", …, "response": { …, "value": "SGVsbG8sIFdvcmxkIQo=" } }, so the burden of extracting a JSON string (either correctly or degenerately by e.g. sed -nr 's/.*(-?[0-9]+).*/\1/p; t done; d; :done q') and then decoding it as base64 is unavoidable. And further, the resulting value can only be arbitrary content such as the requested JSON Lines format from LegacyQuerierHandler, which is removed as of cosmos-sdk v0.47.0. What remains after that removal is protobuf-based gRPC, which means the content itself must be an object rather than a raw string (which is protobuf wire format through path "/agoric.vstorage.Query/$method" or a JSON equivalent through path "custom/vstorage/…"), thereby imposing yet more processing requirements on clients to unwrap something like { "value": "actually-desired data" }.

What we can do is translate the passable/capdata encodings, so that will be the focus from this point.

[gibson042]

We might also want to request more flexibility from CometBFT and/or cosmos-sdk, but that is obviously out of scope for the short term.

[gibson042]

The primary consumers of this interface will be CLI and shell scripts, and its nature suggests (if not requires) opinionated lossy transformation of source data (answering your last question in the affirmative—lossiness differentiates the design space). But different consumers will have different needs, and the interface also needs configurability via some kind of request parameters to control each kind of JSON-incompatible data (slot/bigint/symbol/undefined/etc.) and also for how to represent the sequence of structured items (JSON Lines/JSON array/CSV/etc. at the top level, deep JSON/flat JSON/etc. at the record level).

Proposed configuration

Top-level format

• JSON
• JSON lines (with loss of StreamCell data block height)
• [future] CSV/TSV or other DSV

StreamCell item

(subject to compatibility with top-level format)

• arbitrarily deep JSON
• flat JSON with kebab keys as from get-flattened-publication.sh (e.g., "totalDebt-value")

Bigint value

• [nonconfigurable] a string consisting of decimal digits with optional leading negation, e.g. 1234 or -40

Remotable value

• an { id, allegedName } object as from get-flattened-publication.sh
• as options from agoric follow
  • bracketed string-tagged like [Alleged: IST brand {}]
  • [subject to compatibility with top-level format] Justin expression like slotToVal("board0257","Alleged: IST brand")

Other JSON-incompatible value

• [nonconfigurable] error