feat(swingset): implement Meters for crank computation charges

This introduces user-visible Meter objects, and allows new dynamic vats to be
bound to a single Meter, both exposed through the vatAdmin facet. Meters are
long-term reservoirs of execution credits, denominated in "computrons". Each
bound vat will deduct credits from its meter until exhausted, at which point
the vat will be terminated. This limits the long-term CPU consumption of a
vat, in addition to the fixed per-crank computron limit applied to any
metered vat.

Meters can be refilled and queried through their API. Each meter also has a
Notifier, and a configurable notification threshold: the notifier will be
updated if/when the remaining credits drop below the threshold. This should
allow a supervisor in userspace enough time to refill the meter before the
associated vat(s) are terminated.

See `docs/metering.md` for documentation.

Some notes:
* The vatAdmin facet now offers `createMeter()`, which returns a `Meter`
  object with methods to manipulate its `remaining` and `threshold` values, as
  well as a `getNotifier` to subscribe to threshold-passing events.
  * It also offers `createUnlimitedMeter()`, which never deducts.
* The vatAdmin `createVat()` call now takes a `meter: Meter` option instead
  of `metered: boolean`. If a Meter is provided, two things happen:
  * Each delivery to that (XS) vat is subject to a per-crank compute limit.
  * Each delivery deducts the compute usage from the Meter.
* When a Meter's `remaining` drops below its `threshold`, the notifier is
  triggered with the current `remaining` value. The actual Meter's value might
  have changed by the time the subscriber hears about the update.
* When a vat's Meter reaches zero, the vat is terminated, just as if it had
  violated the per-crank limit.
  * Currently the termination message (used to reject the control facet's
    `.done()` Promise) is different for the per-crank limit vs the Meter limit,
    but this may change.
* Meter deductions and threshold notifications are stashed in a new
  'postAbortActions' record, to make sure they happen even if the crank is
  aborted and all other state changes are unwound.
* The vatManager `managerOptions` still use `metered: boolean`, because the
  vat manager doesn't know about Meters: it only need to know whether to apply
  the per-crank limits or not.

closes #3308

packages/SwingSet/docs/metering.md

@@ -0,0 +1,96 @@
+# Metering CPU Usage
+
+The Halting Problem is unsolvable: no amount of static analysis or human auditing can pre-determine how many steps an arbitrary Turing-complete program will take before it finishes, or if it will ever finish. To prevent the code in one vat from preventing execution of code in other vats (or the kernel itself), SwingSet provides a mechanism to limit the amount of computation that each vat can perform. Any vat which exceeds its limit is terminated, and any messages it sent before the limit was reached are cancelled.
+
+Two limits can be imposed. The first is a per-crank limit. Each message delivered to a vat results in a sequence of "turns" known as a "crank". A crank is also triggered when the vat receives notification of a kernel-side promise being resolved or rejected. Cranks run until the vat stops adding work to the resolved-promise queue, and there is nothing left to do until the next message or notification arrives. A per-crank limit imports a ceiling on the amount of computation that can be done during each crank, but does not say anything about the number of cranks that can be run.
+
+The second limit spans multiple cranks and is managed by the "Meter": a variable-sized reservoir of execution credits. Each vat can be associated with a single Meter, and the remaining capacity of the Meter is reduced at the end of each crank by whatever amount the vat consumed during that crank. The Meter can be refilled by sending it a message, but if any crank causes the Meter's remaining value to drop below zero, the vat is terminated.
+
+## The Computron
+
+SwingSet measures computation with a unit named the "computron": the smallest unit of indivisible computation. The number of computrons used by a given piece of code depends upon its inputs, the state it can access, and the history of its previous activity, but it does *not* depend upon the activity of other vats, other processes on the same host computer, wall-clock time, or type of CPU being used (32-bit vs 64-bit, Intel vs ARM). The metering usage is meant to be consistent across any SwingSet using the same version of the kernel and vat code, which receives the same sequence of vat inputs (the transcript), making it safe to use in a consensus machine.
+
+Metering is provided by low-level code in the JavaScript engine, which is counting basic operations like "read a property from an object" and "add two numbers". This is larger than a CPU cycle. The exact mapping depends upon intricate details of the engine, and is likely to change if/when the JS engine is upgraded. SwingSet kernels that participate in a consensus machine must be careful to synchronize upgrades to prevent divergence of metering results.
+
+TODO: add examples of computron usage numbers
+
+Computrons have a loose relationship to wallclock time, but are generally correlated, so tracking the cumulative computrons spent during SwingSet cranks can provide a rough measure of how much time is being spent, which can be useful to e.g. limit blocks to a reasonable amount of execution time.
+
+The SwingSet Meter APIs accept and deliver computron values in BigInts.
+
+## Meter Objects
+
+The kernel manages `Meter` objects. Each one has a `remaining` capacity and a notification `threshold`. The Meter has a `Notifier` which can inform interested parties when the capacity drops below the threshold, so they can refill it before any associated vats are in danger of being terminated due to an underflow.
+
+Vats can create a Meter object by invoking the `createMeter` method on the `vatAdmin` object. This is the same object used to create new dynamic vats. `createMeter` takes two arguments, both denominated in computrons:
+
+* `remaining`: sets the initial capacity of the Meter
+* `threshold`: set the notification threshold
+
+If you want to impose a per-crank limit, but not a cumulative limit, you can use `createUnlimitedMeter` to make a Meter that never deducts (`remaining` is always the special string `'unlimited'`) and never notifies.
+
+```js
+const remaining = 100_000_000n; // 100M computrons
+const threshold = 20_000_000n: // notify below 20M
+const meter = await E(vatAdmin).createMeter(remaining, threshold);
+const umeter = await E(vatAdmin).createUnlimitedMeter();
+```
+
+The holder of a Meter object can manipulate the meter with the following API:
+
+* `meter.addRemaining(delta)`: increment the capacity by some amount
+* `meter.setThreshold(threshold)`: replace the notification threshold
+* `meter.get() -> { remaining, threshold }`: read the remaining capacity and current notification threshold
+* `meter.getNotifier() -> Notifier`: access the Notifier object
+
+```js
+await E(meter).get(); // -> { remaining: 100_000_000n, threshold: 20_000_000n }
+await E(meter).setThreshold(50n);
+await E(meter).get(); // -> { remaining: 100_000_000n, threshold: 50n }
+await E(meter).addRemaining(999n);
+await E(meter).get(); // -> { remaining: 100_000_999n, threshold: 50n }
+```
+
+## Notification
+
+The meter's `remaining` value will be deducted over time. When it crosses below `threshold`, the Notifier is updated. This is an instance of `@agoric/notifier`:
+
+```js
+const notifier = await E(meter).getNotifier();
+const initial = await E(notifier).getUpdateSince();
+const p1 = E(notifier).getUpdateSince(initial);
+p1.then(remaining => console.log(`meter down to ${remaining}, must refill`));
+```
+
+Note that the notification will occur only once for each transition from "above threshold" to "below threshold". So even if the vat continues to operate (and keeps deducting from the Meter), the notification will not be repeated.
+
+The notification may be triggered again if the meter is refilled above the current threshold, or if the threshold is reduced below the current remaining capacity.
+
+## Per-Crank Limits
+
+The per-crank limit is currently hardcoded to 100M computrons, defined by `DEFAULT_CRANK_METERING_LIMIT` in `packages/xsnap/src/xsnap.js`. This has experimentally been determined to be sufficient for loading large contract bundles, which is the single largest operation we've observed so far.
+
+This per-crank limit is intended to maintain fairness even among vats with a large Meter capacity: just because the Meter allows the vat to spend 17 hours of CPU time, we don't want it to spend it all at once. It also provides a safety mechanism when the vat is using an "unlimited" meter, which allows the vat to use as make cranks as it wants, but each crank is limited.
+
+## Assigning Meters to Vats
+
+Each vat can be associated with a single Meter. A Meter can be attached to multiple vats (although that may make it difficult to assign responsibility for the consumption it measures). To attach a Meter, include it in the options bag to the `vatAdmin`'s `createVat` or `createVatByName` methods:
+
+```js
+const control = await E(vatAdmin).createVat(bundle, { meter });
+```
+
+The default (omitting a `meter` option) leaves the vat unmetered.
+
+Assigning a Meter to a vat activates the per-crank limit. To achieve a per-crank limit without a Meter object (which must be refilled occasionally to keep the vat from being terminated), use an unlimited meter:
+
+```js
+const meter = await E(vatAdmin).createUnlimitedMeter();
+const control = await E(vatAdmin).createVat(bundle, { meter });
+```
+
+## runPolicy
+
+TODO: The host application can limit the number of cranks processed in a single call to `controller.run()` by providing a `runPolicy` object. This policy object is informed about each crank and the number of computrons it consumed. By comparing the cumulative computrons against an experimentally (and externally) determined threshold, the `runLimit` object can tell the kernel to stop processing before the run-queue is drained. For a busy kernel, with an ever-increasing amount of work to do, this can limit the size of a commitment domain (e.g. the "block" in a blockchain / consensus machine).
+
+This is a work in process, please follow issue #3460 for progress.

packages/SwingSet/src/kernel/kernel.js

@@ -1,6 +1,7 @@
 // @ts-check
 import { assert, details as X } from '@agoric/assert';
 import { importBundle } from '@agoric/import-bundle';
+import { stringify } from '@agoric/marshal';
 import { assertKnownOptions } from '../assertOptions.js';
 import { makeVatManagerFactory } from './vatManager/factory.js';
 import { makeVatWarehouse } from './vatManager/vat-warehouse.js';
@@ -374,12 +375,43 @@ export default function buildKernel(
   }
 
   let terminationTrigger;
+  let postAbortActions;
 
   function resetDeliveryTriggers() {
     terminationTrigger = undefined;
+    postAbortActions = {
+      meterDeductions: [], // list of { meterID, compute }
+      meterNotifications: [], // list of meterID
+    };
   }
   resetDeliveryTriggers();
 
+  function notifyMeterThreshold(meterID) {
+    // tell vatAdmin that a meter has dropped below its notifyThreshold
+    const { remaining } = kernelKeeper.getMeter(meterID);
+    const args = { body: stringify(harden([meterID, remaining])), slots: [] };
+    assert.typeof(vatAdminRootKref, 'string', 'vatAdminRootKref missing');
+    queueToKref(vatAdminRootKref, 'meterCrossedThreshold', args, 'logFailure');
+  }
+
+  function deductMeter(meterID, compute, firstTime) {
+    assert.typeof(compute, 'bigint');
+    const res = kernelKeeper.deductMeter(meterID, compute);
+    // we also recode deduction and any notification in postAbortActions,
+    // which are executed if the delivery is being rewound for any reason
+    // (syscall error, res.underflow), so their side-effects survive
+    if (firstTime) {
+      postAbortActions.meterDeductions.push({ meterID, compute });
+    }
+    if (res.notify) {
+      notifyMeterThreshold(meterID);
+      if (firstTime) {
+        postAbortActions.meterNotifications.push(meterID);
+      }
+    }
+    return res.underflow;
+  }
+
   // this is called for syscall.exit (shouldAbortCrank=false), and for any
   // vat-fatal errors (shouldAbortCrank=true)
   function setTerminationTrigger(vatID, shouldAbortCrank, shouldReject, info) {
@@ -391,12 +423,13 @@ export default function buildKernel(
     }
   }
 
-  async function deliverAndLogToVat(vatID, kd, vd) {
+  async function deliverAndLogToVat(vatID, kd, vd, useMeter) {
     // eslint-disable-next-line no-use-before-define
     assert(vatWarehouse.lookup(vatID));
     const vatKeeper = kernelKeeper.provideVatKeeper(vatID);
     const crankNum = kernelKeeper.getCrankNumber();
     const deliveryNum = vatKeeper.nextDeliveryNum(); // increments
+    const { meterID } = vatKeeper.getOptions();
     /** @typedef { any } FinishFunction TODO: static types for slog? */
     /** @type { FinishFunction } */
     const finish = kernelSlog.delivery(vatID, crankNum, deliveryNum, kd, vd);
@@ -412,6 +445,21 @@ export default function buildKernel(
         // probably a metering fault, or a bug in the vat's dispatch()
         console.log(`delivery problem, terminating vat ${vatID}`, problem);
         setTerminationTrigger(vatID, true, true, makeError(problem));
+        return;
+      }
+      if (deliveryResult[0] === 'ok' && useMeter && meterID) {
+        const metering = deliveryResult[2];
+        assert(metering);
+        const consumed = metering.compute;
+        assert.typeof(consumed, 'number');
+        const used = BigInt(consumed);
+        const underflow = deductMeter(meterID, used, true);
+        if (underflow) {
+          console.log(`meter ${meterID} underflow, terminating vat ${vatID}`);
+          const err = makeError('meter underflow, vat terminated');
+          setTerminationTrigger(vatID, true, true, err);
+          return;
+        }
       }
     } catch (e) {
       // log so we get a stack trace
@@ -433,7 +481,7 @@ export default function buildKernel(
       const kd = harden(['message', target, msg]);
       // eslint-disable-next-line no-use-before-define
       const vd = vatWarehouse.kernelDeliveryToVatDelivery(vatID, kd);
-      await deliverAndLogToVat(vatID, kd, vd);
+      await deliverAndLogToVat(vatID, kd, vd, true);
     }
   }
 
@@ -546,7 +594,7 @@ export default function buildKernel(
       // eslint-disable-next-line no-use-before-define
       const vd = vatWarehouse.kernelDeliveryToVatDelivery(vatID, kd);
       vatKeeper.deleteCListEntriesForKernelSlots(targets);
-      await deliverAndLogToVat(vatID, kd, vd);
+      await deliverAndLogToVat(vatID, kd, vd, true);
     }
   }
 
@@ -570,7 +618,7 @@ export default function buildKernel(
     }
     // eslint-disable-next-line no-use-before-define
     const vd = vatWarehouse.kernelDeliveryToVatDelivery(vatID, kd);
-    await deliverAndLogToVat(vatID, kd, vd);
+    await deliverAndLogToVat(vatID, kd, vd, false);
   }
 
   async function processCreateVat(message) {
@@ -682,9 +730,18 @@ export default function buildKernel(
           // errors unwind any changes the vat made
           abortCrank();
           didAbort = true;
+          // but metering deductions or underflow notifications must survive
+          const { meterDeductions, meterNotifications } = postAbortActions;
+          for (const { meterID, compute } of meterDeductions) {
+            deductMeter(meterID, compute, false);
+          }
+          for (const meterID of meterNotifications) {
+            // reads meter.remaining, so must happen after deductMeter
+            notifyMeterThreshold(meterID);
+          }
         }
-        // state changes reflecting the termination must survive, so these
-        // happen after a possible abortCrank()
+        // state changes reflecting the termination must also survive, so
+        // these happen after a possible abortCrank()
         terminateVat(vatID, shouldReject, info);
         kernelSlog.terminateVat(vatID, shouldReject, info);
         kdebug(`vat terminated: ${JSON.stringify(info)}`);
@@ -908,6 +965,13 @@ export default function buildKernel(
         return vatID;
       },
       terminate: (vatID, reason) => terminateVat(vatID, true, reason),
+      meterCreate: (remaining, threshold) =>
+        kernelKeeper.allocateMeter(remaining, threshold),
+      meterAddRemaining: (meterID, delta) =>
+        kernelKeeper.addMeterRemaining(meterID, delta),
+      meterSetThreshold: (meterID, threshold) =>
+        kernelKeeper.setMeterThreshold(meterID, threshold),
+      meterGet: meterID => kernelKeeper.getMeter(meterID),
     };
 
     // instantiate all devices

packages/SwingSet/src/kernel/loadVat.js

@@ -87,7 +87,7 @@ export function makeVatLoader(stuff) {
 
   const allowedDynamicOptions = [
     'description',
-    'metered',
+    'meterID',
     'managerType', // TODO: not sure we want vats to be able to control this
     'vatParameters',
     'enableSetup',
@@ -133,13 +133,14 @@ export function makeVatLoader(stuff) {
    *
    * @param {number} options.virtualObjectCacheSize
    *
-   * @param {boolean} [options.metered] if true,
-   *        subjects the new dynamic vat to a meter that limits
-   *        the amount of computation and allocation that can occur during any
-   *        given crank. Stack frames are limited as well. The meter is refilled
-   *        between cranks, but if the meter ever underflows, the vat is
-   *        terminated. If false, the vat is unmetered.  Defaults to false for
-   *        dynamic vats; static vats may not be metered.
+   * @param {string} [options.meterID] If a meterID is provided, the new
+   *        dynamic vat is limited to a fixed amount of computation and
+   *        allocation that can occur during any given crank. Peak stack
+   *        frames are limited as well. In addition, the given meter's
+   *        "remaining" value will be reduced by the amount of computation
+   *        used by each crank. The meter will eventually underflow unless it
+   *        is topped up, at which point the vat is terminated. If undefined,
+   *        the vat is unmetered. Static vats cannot be metered.
    *
    * @param {Record<string, unknown>} [options.vatParameters] provides
    *        the contents of the second argument to
@@ -199,7 +200,7 @@ export function makeVatLoader(stuff) {
       isDynamic ? allowedDynamicOptions : allowedStaticOptions,
     );
     const {
-      metered = false,
+      meterID,
       vatParameters = {},
       managerType,
       enableSetup = false,
@@ -231,7 +232,7 @@ export function makeVatLoader(stuff) {
     const managerOptions = {
       managerType,
       bundle: vatSourceBundle,
-      metered,
+      metered: !!meterID,
       enableDisavow,
       enableSetup,
       enablePipelining,

packages/SwingSet/src/kernel/state/kernelKeeper.js

@@ -54,6 +54,7 @@ const enableKernelGC = true;
 // v$NN.t.endPosition = $NN
 // v$NN.vs.$key = string
 // v$NN.lastSnapshot = JSON({ snapshotID, startPos })
+// v$NN.meter = m$NN
 
 // m$NN.remaining = $NN // remaining capacity (in computrons) or 'unlimited'
 // m$NN.threshold = $NN // notify when .remaining first drops below this

packages/SwingSet/src/kernel/state/vatKeeper.js

@@ -102,7 +102,7 @@ export function makeVatKeeper(
   }
 
   function getOptions() {
-    const options = JSON.parse(kvStore.get(`${vatID}.options`));
+    const options = JSON.parse(kvStore.get(`${vatID}.options`) || '{}');
     return harden(options);
   }
 

packages/SwingSet/src/kernel/vatAdmin/vatAdmin-src.js

@@ -18,12 +18,36 @@
  */
 
 import { Far } from '@agoric/marshal';
+import { Nat } from '@agoric/nat';
 
 export function buildRootDeviceNode({ endowments, serialize }) {
-  const { pushCreateVatEvent, terminate: kernelTerminateVatFn } = endowments;
+  const {
+    pushCreateVatEvent,
+    terminate: kernelTerminateVatFn,
+    meterCreate,
+    meterAddRemaining,
+    meterSetThreshold,
+    meterGet,
+  } = endowments;
 
   // The Root Device Node.
   return Far('root', {
+    createMeter(remaining, threshold) {
+      return meterCreate(Nat(remaining), Nat(threshold));
+    },
+    createUnlimitedMeter() {
+      return meterCreate('unlimited', 0n);
+    },
+    addMeterRemaining(meterID, delta) {
+      meterAddRemaining(meterID, Nat(delta));
+    },
+    setMeterThreshold(meterID, threshold) {
+      meterSetThreshold(meterID, Nat(threshold));
+    },
+    getMeter(meterID) {
+      return meterGet(meterID);
+    },
+
     // Called by the wrapper vat to create a new vat. Gets a new ID from the
     // kernel's vat creator fn. Remember that the root object will arrive
     // separately. Clean up the outgoing and incoming arguments.