feat(swingset): slogfile visualization: PlantUML, causeway (WIP) #3624

dckc · 2021-08-07T07:06:11Z

This runs to completion in the case of chain-1.slog.gz that I got from @warner , but causeway complains about the output:

Uncaught Error: conflicting Sent for message: kp1167
    at makeMessageGraph (makeMessageGraph.js:78)
    at makeCausewayModel (makeCausewayModel.js:25)
    at makeModel (causalityGrid.js:195)
    at causalityGrid.js:203
    at Object.collect (causalityGrid.js:24)
    at XMLHttpRequest.request.onreadystatechange (causalityGrid.js:100)

I ran zcat chain-1.slog.gz | node slog-to-causeway.mjs >chain1.log and put chain-1.log in (my checked out copy) of https://github.com/cocoonfx/causeway/tree/master/src/js/com/teleometry/causeway/purchase_example/workers/log

Then I made this tweak to point at it:

diff --git a/src/js/com/teleometry/causalityGrid.js b/src/js/com/teleometry/causalityGrid.js
index 2e033f2..e350439 100644
--- a/src/js/com/teleometry/causalityGrid.js
+++ b/src/js/com/teleometry/causalityGrid.js
@@ -218,3 +218,3 @@ var showTracelog;
     //cg_logfiles = [ 'causeway/purchase_example/workers/log/V8-workers.log' ];
-    cg_logfiles = [ 'causeway/purchase_example/workers/log/FF-workers.log' ];
+    cg_logfiles = [ 'causeway/purchase_example/workers/log/chain1.log' ];
     //cg_logfiles = [ 'causeway/purchase_example/workers/log/JSC-workers.log' ];
@@ -237,3 +237 @@ var showTracelog;
 })();

and then visited causalityGrid.html in a vs-code live server.

cc @kriskowal @michaelfig @FUDCo @katelynsills

dckc · 2021-08-27T21:25:06Z

The slog-to-diagram script on this branch does an alternative rendering.

For example, see the sequence diagram for #3760

dckc · 2021-09-23T21:44:23Z

While working with @Chris-Hibbert on governance, I find myself wishing for this Causeway integration often. Just after @erights asked if something similar to recent work on the XS console (#3856) could be used to get stack traces in the REPL, I found myself looking at a rather un-helpful [TypeError: ?: no function] with no stack while working on the governance demo (#3869). p.s. that probably results from calling a method on a promise where I should have used E(..).foo()

Today I talked with @warner about adding message sequence numbers in kernel delivery objects to facilitate correlating send syscalls with deliveries (and resolve syscalls with notifications). (cf. #6501)

We puzzled over FF-workers.log some, along with mainBuyer.js but didn't reach a full understanding.

I didn't manage to work Causeway's Trace Log Format docs into the discussion.

Setting this aside again, for now...

dckc · 2021-10-04T16:42:08Z

Somewhat related: #2885 around deep stacks

more directly on target: #1864

dckc · 2021-10-17T03:41:46Z

progress!

Some causal links aren't quite hooked up right, but lots of stuff is working:

notes to self:

~/projects/agoric/agoric-sdk/packages/treasury
22:28 connolly@jambox$ SLOGFILE=$PWD/t2.slog yarn test test/swingsetTests/governance/test-governance.js 

connolly@jambox:~/projects/agoric/testnet-load-generator$ node slog-to-causeway.mjs <t2.slog >t2.log

connolly@jambox:~/projects/agoric/causeway/src/js/com/teleometry/causeway/purchase_example/workers/log$ ln -s ~/projects/agoric/testnet-load-generator/t2.log

dckc · 2021-10-26T16:55:25Z

represent promise resolution just like message delivery

After more study of examples, I realized they don't use anything fancy to represent promise resolution. Just .Sent and .Got just like message delivery.

Now it all seems to work pretty well.

vatAdmin device is a bit magic

The newVatCallback and meterCrossedThreshold methods in the vatAdmin vat are delivered as usual but not sent by send syscalls. slog-to-causeway.mjs treats them as causally unrelated.

dckc · 2021-10-26T18:01:31Z

A couple examples are working pretty well:

a10 from test-message-patterns in swingset: https://causeway-test-treasury.netlify.app/src/js/com/teleometry/causalitygrid#msg-pat (@warner @FUDCo )
treasury governance: https://causeway-test-treasury.netlify.app/src/js/com/teleometry/causalitygrid#test-treasury (@Chris-Hibbert )

The @@ error dialog is safe to ignore.

Chris-Hibbert

Causeway is an interesting tool. It would be nice to be able to make use of it. Some questions from glancing at the code....

Chris-Hibbert · 2021-10-26T17:51:46Z

packages/SwingSet/misc-tools/slog-to-causeway.mjs

+
+import './types.js';
+
+const { freeze } = Object;


how is freeze different from harden?

freeze is part of the JS standard library. I was avoiding dependencies on the Agoric platform while building this tool.

Substantively: freeze isn't recursive.

Chris-Hibbert · 2021-10-26T17:52:49Z

packages/SwingSet/misc-tools/slog-to-causeway.mjs

+const { freeze } = Object;
+
+/**
+ * @typedef { |


What does @typedef { | Something } mean?

oh, it's only used on line 17, so it expands to

@typedef { org.ref_send.log. org.ref_send.log.Something } LogClassT

Should the first entry end in a bare dot? Nope, from the def'n on line 20, it looks like the empty case shouldn't be included. Does @typedef drop it, or is this a mistake?

The leading | in @typedef { | Something } is a little like the trailing , in const x = [1,2,3,]. It's ignored.

But line breaking rules in typescript-in-jsdoc are weird, and the declaration breaks without it.

Chris-Hibbert · 2021-10-26T18:04:48Z

packages/SwingSet/misc-tools/slog-to-causeway.mjs

+        class: [LogClass.Got, LogClass.Event],
+        anchor,
+        message,
+        trace: { calls: [{ name: text, source: '@@' }] },


What does '@@' do? It looks like you expect the source location to be substituted, but I couldn't find any JS doc saying that's the case.

'@@' means "I don't have a source file name, but the causeway code ignores this whole node unless it has something here, so I'll stick some arbitrary marker for now."

Perhaps '<unknown-source>'. I’ve used this pattern elsewhere to improve error messages like JSON parse errors for a file with a name that defaults to '<unknown-something>'.

dckc · 2022-03-10T04:33:45Z

@arirubinstein for reference, https://github.com/Agoric/agoric-sdk/blob/slog-to-causeway/packages/SwingSet/misc-tools/slog-to-diagram.mjs is the tool that I used to make that .puml sequence diagram out of bootstrap and starting the RUN protocol.

Usage is as in the Makefile in https://gist.github.com/dckc/c300327762d1fa128580c1399e0ae41f

to generate the input slog file, use SLOGFILE=/tmp/f1 agoric start and / or SOLO_SLOGFILE=/tmp/f2 agoric start.

- reserve only appropriate names in agoricNames name hubs

- install gov contracts - test that boot-psm puts an instance in agoricNames

for PSM-only config

- sort by time - number by elapsed time within a block

- show promise resolution state - add @startuml , @enduml

- handle lack of block time markers, e.g. in testing

- make some fields optional - use SentIf / condition for promise resolution problem: conflicting Sent for message Rkp41, Rkp42 two resolve syscalls with the same deliveryNum

this provokes causeway to show stuff

- don't add { text} on .Got so that vat,turn is shown - add ko, kp, body.length detail to .Send

- log about unknonwn tags - handle / skip missing dInfo / dr[2] - handle / skip negative t

- a bit more static typing

dckc · 2024-08-23T21:12:00Z

inter-vat message visualization service

@toliaqat as discussed, it would be nice to wrap this half-baked code (or anything that does a similar job) as a service of sorts:

Use datadog to select a time interval of interest
extracts CSV
drop CSV on this service
service produces .svg diagram

slog.svg is an example I generated today.

Makefile shows how I did it. It belongs in agoric-sdk/packages/SwingSet/misc-tools. When I invoke it, it uses yq to get a .slog from the CSV, etc.:

agoric-sdk/packages/SwingSet/misc-tools$ make
yq -p csv -o json -I=0 '.[] | .Message' extract-2024-08-23T16_12_20.185Z.csv | tac | \
        jq -c 'select((.time >= 1724425557.382605 - 2) and (.time <= 1724425559.071834))' >requested-interval.slog
node slog-to-diagram.mjs <requested-interval.slog >requested-interval.puml
syscall tag unknown: vatstoreGet
syscall tag unknown: subscribe
syscall tag unknown: vatstoreSet
type unknown: console
type unknown: notify
syscall tag unknown: vatstoreDelete
syscall tag unknown: vatstoreGetNextKey
syscall tag unknown: invoke
??? t < 0 0
no source for { ref: 'Rkp13858960' }
# https://plantuml.com/command-line
java -jar plantuml-1.2022.7.jar -tsvg requested-interval.puml

agoric-sdk/packages/SwingSet/misc-tools$ ls -ltr extract-2024-08-23T16_12_20.185Z.csv requested-interval.* *.svg
-rw-rw-r-- 1 connolly connolly 21276242 Aug 23 11:12 extract-2024-08-23T16_12_20.185Z.csv
-rw-rw-r-- 1 connolly connolly  1311000 Aug 23 15:58 requested-interval.slog
-rw-rw-r-- 1 connolly connolly     3990 Aug 23 15:58 requested-interval.puml
-rw-rw-r-- 1 connolly connolly    49601 Aug 23 15:58 slog.svg

The slog-to-diagram.mjs code is kinda scruffy. Any messages such as syscall tag unknown: vatstoreGet that it spits out should probably be reported to the user. A few cases of no source for { ref: 'Rkp13858960' } are probably OK, but a zillion of them suggests that something is going wrong.

Option: diagram key

The participants in the state diagram are vats.

Arrows represent messages between vats.

A solid arrow labelled:

0.1 kp13858945 <- ko1009519.fromBridge(866)

That means: at 0.1 seconds into the block, a message with method fromBridge and 866 characters of arguments was sent to kernel object ko1009519, allocating a promise kp13858945 for the result.

An arrow with kp13858945.fulfilled() means the promise was fulfilled. An arrow with kp13858945.rejected() means it was rejected.

An arrow whose tail is the left edge of the diagram means the script couldn't figure out which vat it came from. (Some messages come from the kernel.)

Yellow boxes indicate large computron counts.

Option: vat key

@warner has some tools that make vat maps:

   v5 :   timer
   v7 :   board
   v9 :   zoe
  v10 :   bridge
  v15 :   zcf-b1-2371d-econCommitteeCharter
  v37 :   zcf-b1-94e20-walletFactory
  v41 :   zcf-b1-9f877-vaultFactory.governor

Option: slog file input

The user might have a slogfile rather than a datadog extract.

Option: start with offer id

The precipitating event in this case was proposing a DebtLimit change. Like all smart contract interactions, the relevant transaction has an OfferSpec including an id. The user could provide the id, and the service would find where that id shows up in datadog logs and extract from there for the following 10 or 20 seconds.

Option: start with any search string

Maybe the search term doesn't have to be an offer id. It could be a distinctive error message, or whatever.

Option: start with time interval

Perhaps the user knows the relevant time interval.

dckc mentioned this pull request Sep 27, 2021

why so many awaits? timer, ... agoric-labs/gov-demo1#1

Open

dckc force-pushed the slog-to-causeway branch from 33b99c4 to a923083 Compare October 14, 2021 00:46

dckc force-pushed the slog-to-causeway branch from 797ced8 to 31edf64 Compare October 26, 2021 16:49

Chris-Hibbert reviewed Oct 26, 2021

View reviewed changes

dckc force-pushed the slog-to-causeway branch from 3c128ee to 720f534 Compare March 10, 2022 04:24

dckc added logging devex developer experience labels May 15, 2022

dckc force-pushed the slog-to-causeway branch from f351c56 to d993a17 Compare May 16, 2022 01:08

dckc mentioned this pull request Aug 17, 2022

Office Hours: Agoric Dev Agoric/documentation#576

Closed

dckc added 14 commits August 24, 2022 20:33

wip(startPSM): instasllGovContracts

ec62b75

chore(inter-protocol): update vat installation in startPSM

7b8f867

build(inter-protocol): build bundle-psm.js

2833838

chore(inter-protocol): install PSM contract with gov contracts

28a8654

chore(inter-protocol): export PSM related manifests

6d07021

feat(vats): boot-psm provides kernel access to bootstrap space

cd89d55

- reserve only appropriate names in agoricNames name hubs

feat(vats): boot-psm starts the PSM contract

db92ef2

- install gov contracts - test that boot-psm puts an instance in agoricNames

chore(vats): attenuate calls from boot-psm using manifest

9777300

feat(vats): PSM-only SwingSet config

23b7a9d

test(cosmic-swingset): scenario2-run-chain-psm Makefile rule

0e9aede

feat(vats): configurable top-level agoricNames

9d32116

chore(vats): publish appropriate top-level agoricNames in boot-psm

e3ef2af

chore(vats): startWalletFactory without provisioning mailbox access

26416ef

for PSM-only config

chore(vats): keep smartWalletStartResult in bootstrap space

ac01a32

dckc added 27 commits August 24, 2022 21:18

docs: braindump TODOs

a5bd2bd

feat(swingset): slog to PlantUML

2afa785

feat: promise resolutions in slog-to-diagram

16acf82

- sort by time - number by elapsed time within a block

feat: show compute in slog-to-diagram

ca98c65

feat: highlight expensive compute cranks in slog-to-diagram

64fbfe9

- show promise resolution state - add @startuml , @enduml

feat: show active vat; dotted resolve arrows

6ed5777

- handle lack of block time markers, e.g. in testing

feat: vat names on sequence diagrams

34197fe

chore: another run at slog-to-causeway (WIP)

a7db80c

- make some fields optional - use SentIf / condition for promise resolution problem: conflicting Sent for message Rkp41, Rkp42 two resolve syscalls with the same deliveryNum

fix: SentIf events need message

386dba7

chore: maybe colons aren't allowed?

d3a025d

feat: provide _some_ trace calls in all cases

ffe4793

this provokes causeway to show stuff

chore: omit trace on promise resolution events, for now

0c03bc5

style: provide @yields

e7b0793

chore: console / comment TODO; types

bb81b1f

fix: represent promise resolution just like message delivery

1ef9a70

- don't add { text} on .Got so that vat,turn is shown - add ko, kp, body.length detail to .Send

chore: revert vat label to vNN:name rather than vNN_name

671eb8c

chore: move causeway tools under misc-tools

0c8385a

style(slog-to-diagram): lint

9bce8d6

fix(slog-to-diagram): more robust against missing elements

92622ed

- log about unknonwn tags - handle / skip missing dInfo / dr[2] - handle / skip negative t

refactor(slog-to-diagram): factor out puml-specific part

8fb8130

- a bit more static typing

feat(slog-to-diagram): mermaid output (WIP)

1be2500

fix(slog-to-diagram): stray )

18b3a4b

fixup: more refactor

011a6da

chore(slog-to-diagram): clean up activate / deactivate

91593c3

chore(slog-to-causeway): ignore new types

03321e1

chore(SwingSet): update slog-to-diagram w.r.t. methargs etc.

ca10e4f

feat(slog-to-diagram): put crank num on labels

2ee9664

dckc force-pushed the slog-to-causeway branch from 49272a6 to 2ee9664 Compare August 25, 2022 05:08

dckc changed the title ~~feat(swingset): convert slogfile to causeway format (WIP)~~ feat(swingset): slogfile visualization: PlantUML, causeway (WIP) Jul 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(swingset): slogfile visualization: PlantUML, causeway (WIP) #3624

feat(swingset): slogfile visualization: PlantUML, causeway (WIP) #3624

dckc commented Aug 7, 2021 •

edited

Loading

dckc commented Aug 27, 2021

dckc commented Sep 23, 2021 •

edited

Loading

dckc commented Oct 4, 2021 •

edited

Loading

dckc commented Oct 17, 2021 •

edited

Loading

dckc commented Oct 26, 2021

dckc commented Oct 26, 2021

Chris-Hibbert left a comment

Chris-Hibbert Oct 26, 2021

dckc Oct 26, 2021

Chris-Hibbert Oct 26, 2021

dckc Oct 26, 2021

Chris-Hibbert Oct 26, 2021

dckc Oct 26, 2021

kriskowal Mar 10, 2022

dckc commented Mar 10, 2022

dckc commented Aug 23, 2024 •

edited

Loading

feat(swingset): slogfile visualization: PlantUML, causeway (WIP) #3624

Are you sure you want to change the base?

feat(swingset): slogfile visualization: PlantUML, causeway (WIP) #3624

Conversation

dckc commented Aug 7, 2021 • edited Loading

dckc commented Aug 27, 2021

dckc commented Sep 23, 2021 • edited Loading

dckc commented Oct 4, 2021 • edited Loading

dckc commented Oct 17, 2021 • edited Loading

dckc commented Oct 26, 2021

represent promise resolution just like message delivery

vatAdmin device is a bit magic

dckc commented Oct 26, 2021

Chris-Hibbert left a comment

Choose a reason for hiding this comment

Chris-Hibbert Oct 26, 2021

Choose a reason for hiding this comment

dckc Oct 26, 2021

Choose a reason for hiding this comment

Chris-Hibbert Oct 26, 2021

Choose a reason for hiding this comment

dckc Oct 26, 2021

Choose a reason for hiding this comment

Chris-Hibbert Oct 26, 2021

Choose a reason for hiding this comment

dckc Oct 26, 2021

Choose a reason for hiding this comment

kriskowal Mar 10, 2022

Choose a reason for hiding this comment

dckc commented Mar 10, 2022

dckc commented Aug 23, 2024 • edited Loading

inter-vat message visualization service

Option: diagram key

Option: vat key

Option: slog file input

Option: start with offer id

Option: start with any search string

Option: start with time interval

dckc commented Aug 7, 2021 •

edited

Loading

dckc commented Sep 23, 2021 •

edited

Loading

dckc commented Oct 4, 2021 •

edited

Loading

dckc commented Oct 17, 2021 •

edited

Loading

dckc commented Aug 23, 2024 •

edited

Loading