Add cosmos-sdk vat snapshot/transcript retention configuration #10032

gibson042 · 2024-09-05T19:10:40Z

TODO:

swing-store should keep historical artifacts outside of SQLite DB #9389

Description

Adds consensus-independent vat-snapshot-retention ("debug" vs. "operational") and vat-transcript-retention ("archival" vs. "operational" vs. "default") cosmos-sdk swingset configuration (values chosen to correspond with artifactMode) for propagation in AG_COSMOS_INIT. The former defaults to "operational" and the latter defaults to "default", which infers a value from cosmos-sdk pruning to allow simple configuration of archiving nodes.

It also updates the semantics of TranscriptStore keepTranscripts: false configuration to remove items from only the previously-current span rather than from all previous spans when rolling over (to avoid expensive database churn). Removal of older items can be accomplished by reloading from an export that does not include them.

Security Considerations

I don't think this changes any relevant security posture.

Scaling Considerations

This will reduce the SQLite disk usage for any node that is not explicitly configured to retain snapshots and/or transcripts. The latter in particular is expected to have significant benefits for mainnet (as noted in #9174, about 116 GB ÷ 147 GB ≈ 79% of the database on 2024-03-29 was vat transcript items).

Documentation Considerations

The new fields are documented in our default TOML template, and captured in a JSDoc type on the JavaScript side.

Testing Considerations

This PR extends coverage TranscriptStore to include keepTranscripts true vs. false, but I don't see a good way to cover Go→JS propagation other than manually (which I have done). It should be possible to add testing for the use and validation of resolvedConfig in AG_COSMOS_INIT handling, but IMO that is best saved for after completion of split-brain (to avoid issues with same-process Go–JS entanglement).

Upgrade Considerations

This is all kernel code that can be used at any node restart (i.e., because the configuration is consensus-independent, it doesn't even need to wait for a chain software upgrade). But we should mention the new cosmos-sdk configuration in release notes, because it won't be added to existing app.toml files already in use.

gibson042 · 2024-09-05T19:18:15Z

golang/cosmos/x/swingset/config.go

 slogfile = "{{ .Swingset.SlogFile }}"

 # The maximum number of vats that the SwingSet kernel will bring online. A lower number
 # requires less memory but may have a negative performance impact if vats need to
 # be frequently paged out to remain under this limit.
-max_vats_online = {{ .Swingset.MaxVatsOnline }}
+max-vats-online = {{ .Swingset.MaxVatsOnline }}


@siarhei-agoric @mhofman This aligns with existing cosmos-sdk conventions; cf. https://github.com/cosmos/cosmos-sdk/blob/68cb1635a0a0c7b3bd923ec9a51d0530e47d8040/server/config/config.toml.tpl

Good catch!

Oh it's tendermint that uses underscore !

packages/cosmic-swingset/src/chain-main.js

gibson042 · 2024-09-05T19:27:01Z

packages/swing-store/src/transcriptStore.js

+  // Transcripts are broken up into "spans", delimited by heap snapshots.
+  // For every vatID, there will be exactly one current span with isCurrent=1,
+  // and zero or more non-current (historical) spans with isCurrent=null.
+  // If we take a heap snapshot after the first hundred deliveries and again
+  // after the second hundred (i.e., after zero-indexed deliveries 99 and 199),
+  // and have not yet performed a delivery after the second snapshot, we'll have
+  // two historical spans (one with startPos=0 and endPos=100, the second with
+  // startPos=100 and endPos=200) and a single empty current span with
+  // startPos=200 and endPos=200.  After we perform the next delivery, the
+  // single current span will still have startPos=200 but will now have
+  // endPos=201.


@warner I updated this to avoid mixing zero-indexed and one-indexed values; please look it over for correctness.

Yeah, I think that's correct. Notes:

"exactly one current span with isCurrent=1" admits the thought that maybe some "current spans" don't have isCurrent=1. Maybe say "exactly one current span (i.e. isCurrent=1)" ? Ditto for the second clause.

Yep, my previous wording was confused, if the first span included a delivery=100, then it should have had endPos=101, because we increment the span record's .endPos just after adding the item.

Do we actually ever have empty spans? I though the first entry of a span was to include a "load snapshot" entry. Maybe the first span can be empty, but I think that has a load entry as well?

I believe that means pos are not strictly equivalent to deliveries.

gibson042 · 2024-09-05T19:28:43Z

packages/swing-store/test/state.test.js

This file is best reviewed while ignoring whitespace changes.

packages/cosmic-swingset/src/chain-main.js

siarhei-agoric

Looks good to me.

mhofman · 2024-09-05T20:44:30Z

vat-transcript-retention ("archival" vs. "operational" vs. "default") cosmos-sdk swingset configuration (values chosen to correspond with artifactMode)

Should we add a "replay" option for consistency? I suppose that without a plan for deletion, that could be implemented the same as "archival" for now.

mhofman · 2024-09-05T20:47:54Z

I don't see a good way to cover Go→JS propagation other than manually (which I have done).

One test would be to bootstrap a node with "operational" mode, and verify that you cannot export the swing-store with "replay" or "archival" mode (previous to the config change you could)

warner

Looks good. I don't know Go well enough to approve that side (please wait for @mhofman 's r+), but the JS side looks ok.

golang/cosmos/x/swingset/config.go

packages/cosmic-swingset/src/chain-main.js

warner · 2024-09-05T22:37:05Z

packages/swing-store/src/transcriptStore.js

+  // Transcripts are broken up into "spans", delimited by heap snapshots.
+  // For every vatID, there will be exactly one current span with isCurrent=1,
+  // and zero or more non-current (historical) spans with isCurrent=null.
+  // If we take a heap snapshot after the first hundred deliveries and again
+  // after the second hundred (i.e., after zero-indexed deliveries 99 and 199),
+  // and have not yet performed a delivery after the second snapshot, we'll have
+  // two historical spans (one with startPos=0 and endPos=100, the second with
+  // startPos=100 and endPos=200) and a single empty current span with
+  // startPos=200 and endPos=200.  After we perform the next delivery, the
+  // single current span will still have startPos=200 but will now have
+  // endPos=201.


Yeah, I think that's correct. Notes:

"exactly one current span with isCurrent=1" admits the thought that maybe some "current spans" don't have isCurrent=1. Maybe say "exactly one current span (i.e. isCurrent=1)" ? Ditto for the second clause.

Yep, my previous wording was confused, if the first span included a delivery=100, then it should have had endPos=101, because we increment the span record's .endPos just after adding the item.

mhofman

Overall this is good. Some minor feedback, feel free to merge once addressed.

mhofman · 2024-09-05T22:53:37Z

golang/cosmos/x/swingset/config.go

 # If relative, it is interpreted against the application home directory
 # (e.g., ~/.agoric).
+# May be overridden by a SLOGFILE environment variable, which if relative is


overridden? I thought the config took precedence if present?

Nope, CLI option > environment variable > config file: https://github.com/spf13/viper?tab=readme-ov-file#why-viper

Viper uses the following precedence order. Each item takes precedence over the item below it:

explicit call to Set

flag

env

config

key/value store

default

packages/cosmic-swingset/src/chain-main.js

mhofman · 2024-09-05T23:06:40Z

golang/cosmos/x/swingset/config.go

+# * "default": determined by ` + "`pruning`" + ` ("archival" if ` + "`pruning`" + ` is
+#   "nothing", otherwise "operational")


Note to self: this is sufficient as even though a pruning strategy of "nothing" internally sets KeepRecent to 0, that value is in fact not a valid value otherwise (2 is the minimum enforced)

mhofman · 2024-09-05T23:08:53Z

golang/cosmos/x/swingset/config.go

+	// See CustomAppConfig in ../../daemon/cmd/root.go.
+	type ExtendedConfig struct {
+		serverconfig.Config `mapstructure:",squash"`
+		Swingset            SwingsetConfig `mapstructure:"swingset"`


Oh yikes I missed the missing mapstructure in the last PR review, how did it even work, does it default to a lowercasing of the struct property?

Yeah, I think so.

packages/cosmic-swingset/src/chain-main.js

mhofman · 2024-09-05T23:54:32Z

packages/swing-store/test/state.test.js

-  });
-  const { transcriptStore } = kernelStorage;
-  const { commit, close } = hostStorage;
+const testTranscriptStore = test.macro({


Wondering if we should, and if there's an easy way, test that older historical spans are not removed when disabling keepTranscripts

We'd need to import data that already includes a historical span, which is possible but not trivial. Given that, it didn't seem worth it because I think we don't actually care that they are left around.

golang/cosmos/x/swingset/config.go

mhofman · 2024-09-06T00:00:49Z

golang/cosmos/x/swingset/config.go

+	//   last snapshot of their vat)
+	// * "default": determined by `pruning` ("archival" if `pruning` is
+	//   "nothing", otherwise "operational")
+	VatTranscriptRetention string `mapstructure:"vat-transcript-retention" json:"vatTranscriptRetention,omitempty"`


I'm not sure omitempty makes sense here. I'd argue this ends up required at the JS/golang interface.

Eh, I'd rather not send an empty string only for this field if that ever comes up. If it truly were required, we'd enforce that on the JS side.

I don't see how it could ever be empty given the resolution logic

mhofman · 2024-09-06T00:05:59Z

packages/cosmic-swingset/src/chain-main.js

+      swingsetConfig;
+    const keepSnapshots =
+      vatSnapshotRetention === 'debug' ||
+      (!vatSnapshotRetention && ['1', 'true'].includes(XSNAP_KEEP_SNAPSHOTS));


I'm not sure how I feel about the env variable "overriding" an explicit "operational" retention in the config. I think that's fine, but maybe a comment here making the intention explicit would help.

That actually wouldn't happen, because of the !vatSnapshotRetention guard. But at any rate, I've updated both keepSnapshots and keepTranscripts to more clearly apply fallback logic only when the config value is falsy.

cloudflare-workers-and-pages · 2024-09-06T02:38:02Z

Deploying agoric-sdk with Cloudflare Pages

Latest commit:	`8b3a6d4`
Status:	✅ Deploy successful!
Preview URL:	https://5a300033.agoric-sdk.pages.dev
Branch Preview URL:	https://gibson-9174-transcript-span.agoric-sdk.pages.dev

View logs

The preferred separator is "-" rather than "_", cf. https://github.com/cosmos/cosmos-sdk/blob/a57b25418a594ae023274673f07b72611ccd2744/server/config/config.toml.tpl

…ration to AG_COSMOS_INIT Ref #9174 Ref #9386

…script span Fixes #9387

…to AG_COSMOS_INIT Fixes #9386

…ination

…lid value error message Ref #10032

…lid value error message (#10081) Ref #10032 ## Description ```diff -value for vat-transcript-retention must be in ["archival" "operational"] +value for vat-transcript-retention must be in ["archival" "operational" "default"] ``` ### Security Considerations n/a ### Scaling Considerations n/a ### Documentation Considerations n/a ### Testing Considerations n/a ### Upgrade Considerations n/a

gibson042 requested review from warner, mhofman and siarhei-agoric September 5, 2024 19:29

gibson042 commented Sep 5, 2024

View reviewed changes

siarhei-agoric reviewed Sep 5, 2024

View reviewed changes

packages/cosmic-swingset/src/chain-main.js Show resolved Hide resolved

siarhei-agoric approved these changes Sep 5, 2024

View reviewed changes

warner approved these changes Sep 5, 2024

View reviewed changes

mhofman approved these changes Sep 6, 2024

View reviewed changes

warner mentioned this pull request Sep 6, 2024

swingstore mode to move/archive historical transcript spans to separate files, outside the DB #10036

Closed

gibson042 force-pushed the gibson-9174-transcript-span-retention branch 2 times, most recently from 9bd39ae to f61c1c1 Compare September 6, 2024 02:43

gibson042 added the automerge:rebase Automatically rebase updates, then merge label Sep 6, 2024

gibson042 added 7 commits September 6, 2024 14:10

chore(cosmic-swingset): Validate AG_COSMOS_INIT resolvedConfig

3cc8ce7

chore(golang/cosmos): Align [swingset] TOML with cosmos-sdk

ea99c6d

The preferred separator is "-" rather than "_", cf. https://github.com/cosmos/cosmos-sdk/blob/a57b25418a594ae023274673f07b72611ccd2744/server/config/config.toml.tpl

feat: Add consensus-independent vat transcript span retention configu…

3cf6b57

…ration to AG_COSMOS_INIT Ref #9174 Ref #9386

feat(swing-store): Limit item deletion to the previously-current tran…

766c1bb

…script span Fixes #9387

feat: Add consensus-independent vat snapshot retention configuration …

a5311b5

…to AG_COSMOS_INIT Fixes #9386

docs(swing-store): Update for changes

b01c033

refactor(cosmic-swingset): Clarify keep{Snapshots,Transcripts} determ…

8b3a6d4

…ination

gibson042 force-pushed the gibson-9174-transcript-span-retention branch from f61c1c1 to 8b3a6d4 Compare September 6, 2024 14:11

mergify bot merged commit d6f50e3 into master Sep 6, 2024
80 checks passed

mergify bot deleted the gibson-9174-transcript-span-retention branch September 6, 2024 14:45

gibson042 added a commit that referenced this pull request Sep 12, 2024

chore(x/swingset): Include "default" in vat-transcript-retention inva…

2e0ff9c

…lid value error message Ref #10032

gibson042 mentioned this pull request Sep 12, 2024

chore(x/swingset): Include "default" in vat-transcript-retention invalid value error message #10081

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add cosmos-sdk vat snapshot/transcript retention configuration #10032

Add cosmos-sdk vat snapshot/transcript retention configuration #10032

gibson042 commented Sep 5, 2024

gibson042 Sep 5, 2024

siarhei-agoric Sep 5, 2024

mhofman Sep 5, 2024 •

edited

Loading

gibson042 Sep 5, 2024

warner Sep 5, 2024

mhofman Sep 5, 2024

gibson042 Sep 5, 2024

siarhei-agoric left a comment

mhofman commented Sep 5, 2024

mhofman commented Sep 5, 2024

warner left a comment

warner Sep 5, 2024

mhofman left a comment

mhofman Sep 5, 2024

gibson042 Sep 6, 2024

mhofman Sep 5, 2024

mhofman Sep 5, 2024

gibson042 Sep 6, 2024

mhofman Sep 5, 2024

gibson042 Sep 6, 2024 •

edited

Loading

mhofman Sep 6, 2024

gibson042 Sep 6, 2024

mhofman Sep 6, 2024

mhofman Sep 6, 2024

gibson042 Sep 6, 2024

cloudflare-workers-and-pages bot commented Sep 6, 2024 •

edited

Loading

		# * "default": determined by ` + "`pruning`" + ` ("archival" if ` + "`pruning`" + ` is
		# "nothing", otherwise "operational")

Add cosmos-sdk vat snapshot/transcript retention configuration #10032

Add cosmos-sdk vat snapshot/transcript retention configuration #10032

Conversation

gibson042 commented Sep 5, 2024

Description

Security Considerations

Scaling Considerations

Documentation Considerations

Testing Considerations

Upgrade Considerations

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mhofman Sep 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

siarhei-agoric left a comment

Choose a reason for hiding this comment

mhofman commented Sep 5, 2024

mhofman commented Sep 5, 2024

warner left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mhofman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gibson042 Sep 6, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cloudflare-workers-and-pages bot commented Sep 6, 2024 • edited Loading

Deploying agoric-sdk with Cloudflare Pages

mhofman Sep 5, 2024 •

edited

Loading

gibson042 Sep 6, 2024 •

edited

Loading

cloudflare-workers-and-pages bot commented Sep 6, 2024 •

edited

Loading