Bundle hashing

[kriskowal]

What is the Problem Being Solved?

The current mechanism for checking integrity of a Zoe installation or generally any bundle is source inspection. Bundles are transparent and can be inspected directly. The new endoZipBase64 format is also transparent but requires considerable machinery to inspect.

Generally, integrity checks would be more robust if they were framed as verification of the bundle’s consistent hash.

Description of the Design

Endo now provides a mechanism for getting the SHA-512 of an Endo application, providing the same answer either on-disk or in-archive. For the existing nestedEvaluate and getExports bundle formats, we produce a SHA-512 from the UTF-8 encoded source.

We can now implement a new package hash-bundle which like bundle-source and import-bundle operates on all of our supported bundle formats to produce a consistent hash.

We then can expose a getHash method on Zoe installations so anyone with a handle on an installation can inspect its hash without the need to obtain the content of the hash.

Security Considerations

The hash function must be wide enough and strong enough to produce a unique value for any contract submitted to the chain and be resilient against attacks that might exploit it in order to submit imposters for other contracts.

Further Design Considerations

This design should flush with @warner’s proposal for “Blob Capabilities” that would allow us to feed bundles to the Agoric chain through a shared content address store and refer to bundles only by their consistent hash in messages to the chain.

This then would allow us to decompose Endo Zip archives into individual files addressed by SHA-512 for the purposes of reducing transmission of duplicate blobs between nodes of the content address store. (cc @dtribble)

Test Plan

This change will need tests that verify hash consistency for consistent bundles, crossed by all of the bundle formats, crossed by Node.js and XSnap vat environments. #3273 removes consistency checks from Zoe tests and this change must include replacements for those tests.

[kriskowal]

This change depends on #3512 Expose SHA-512 to XSnap vats.

[warner]

refs #3269 for the module/package -at-a-time install-to-chain API, plus the manifest-ish thing that sits on top of those to identify the complete bundle

[kriskowal]

Archive hashing described in greater detail here endojs/endo#794

[warner]

Just finished chatting with @kriskowal , it sounds like we'll wind up with four source formats:

• getExports
• nestedEvaluate
• an endo/zip/base64 thing in which Zoe is given the entire zipfile, or maybe an object that contains all the component files in a bit table
• an endo/zip thing in which Zoe is given only the compartment-map.json hash, and relies upon the controller.installBundle() #3269 blobstore for getting the pieces

We're adding the third thing now, and the fourth thing later. While we're in the third step, Zoe will still be passing around large strings, and the only support swingset needs to provide is a way to do SHA512 hashes from within vat code (and @kriskowal has a plan for making that visible as a global, although we should add that to V8-based vats too, for consistency).

We'll be adding a method like getHash to the Far "installation" object. When invoked on an Installation created from any of the first three formats, Zoe will need to do the SHA512 hashing (and the details depend upon the exact format: for the endo archive, it is allowed to extract and hash the compartment-map.json file). All hashes will include the module format, so a caller can distinguish between e.g. a getExports and a nestedEvaluate that happen to use the same source: property. Also the hash must include the sourceMap property value. All three types continue to support a getSource or getBundle, which returns the large original bundle string/object.

When we get to step 4, and introduce blobcaps, then getHash can do less work, and getSource/getBundle throws, because the Installation was never given the full bundle in the firts place. Finally, when we get rid of the first three types, clients will only compare installations by their hashes, and they'll run a local hash-bundle tool to figure out what hash to expect.

[Tartuffo]

@kriskowal This is in the "Up Next" pipeline, but does not have a MN-1 label. If it is needed for MN-1, please label, otherwise move from "Up Next" to "Product Backlog".

[Tartuffo]

@kriskowal This does not have an area label that is covered by our weekly tech / planning meetings. Can you assign the proper label? We cover: agd, agoric-cosmos, amm, core economy, cosmic-swingset, endo, getrun, governance, installation-bundling, metering, run-protocol, staking, swingset, swingset-runner, token economy, wallet, zoe contract,

[Tartuffo]

@kriskowal Is your work for this covered in another issue? If not, you might want to create a separate issue.

[kriskowal]

My work for this is covered entirely by this issue.