Outline blocking dependencies on centralised tools (e.g. etherscan)

[SidestreamColdMelon]

Goal

Checklists do not enforce processes that are blocked in case a single service is down

Context

Recently, spell team experienced downtime of etherscan, which caused a multi-hour delay in the spell handover and later confusion among delegates on why contract is not verified on etherscan. The delay was resolved by unanimous agreement within the spell team to proceed with an alternative verifier service and later still use etherscan to verify contract and resolve the confusion. Another reason to not depend on a single service is of course security: it's much easier to compromise a single crucial service documented in the process, than try to attack multiple independent services at the same time.

In order to prepare to such events, we should 1) evaluate existing dependencies 2) evaluate potential circumventions 3) proceed with removing dependencies one-by-one.

Todo

• List existing dependencies found in the checklists / known processes
• List potential circumventions (general or applicable to each specific dependency)
• Create first specific issue to remove dependency on etherscan

[SidestreamColdMelon]

Existing dependencies found in the checklists / known processes

• https://docs.google.com/spreadsheets/

  • Get list of planned instructions from the Exec Sheet
    

    pe-checklists/spell/spell-reviewer-mainnet-checklist.md

    Lines 6 to 7 in 1aa58ee

    	* [ ] Exec Sheet for the specified date is found in the ["Executive Vote Implementation Process" google sheet](https://docs.google.com/spreadsheets/d/1w_z5WpqxzwreCcaveB2Ye1PP5B8QAHDglzyxKHG3CHw)
    	_Insert URL to the specific sheet here_

• http://github.com

  • Open pull request at https://github.com/makerdao/spells-mainnet/
  • Get latest version of checklists at https://github.com/makerdao/pe-checklists/
  • Get Exec Doc from https://github.com/makerdao/community/
    

    pe-checklists/spell/spell-reviewer-mainnet-checklist.md

    Line 320 in 1aa58ee

    * [ ] Exec Doc for the specified date is found in the [`makerdao/community` GitHub repo](https://github.com/makerdao/community/tree/master/governance/votes)

  • Check CI tests at https://github.com/makerdao/spells-mainnet/
    

    pe-checklists/spell/spell-reviewer-mainnet-checklist.md

    Lines 302 to 303 in 1aa58ee

    	* [ ] Check all CI tests are passing as at the latest commit
    	_Insert most recent commit hash where CI was passing_

  • References to other github repos in the checklists
    • DSS Exec Library https://github.com/makerdao/dss-exec-lib/, 40 occurrences, e.g.:
      

      pe-checklists/spell/spell-reviewer-mainnet-checklist.md

      Line 38 in 1aa58ee

      * [ ] git submodule hash of `dss-exec-lib` (run `git submodule status`) matches the [latest release version](https://github.com/makerdao/dss-exec-lib/releases) or newer

    • Multi Collateral Dai https://github.com/makerdao/dss/, 2 occurrences, e.g.:
      

      pe-checklists/spell/spell-reviewer-mainnet-checklist.md

      Line 57 in 1aa58ee

      * [ ] Precision units match with [Numerical Ranges](https://github.com/makerdao/dss/wiki/Numerical-Ranges#notation)

    • Mainnet spells https://github.com/makerdao/spells-mainnet/, 4 occurrences, e.g.:
      

      pe-checklists/spell/spell-reviewer-mainnet-checklist.md

      Line 66 in 1aa58ee

      * [ ] Match with [config](https://github.com/makerdao/spells-mainnet/blob/master/src/test/config.sol)

    • Debt Ceiling Instant Access Module https://github.com/makerdao/dss-auto-line/, 3 occurrences, e.g.:
      

      pe-checklists/spell/spell-reviewer-mainnet-checklist.md

      Line 180 in 1aa58ee

      * [ ] Collateral type (`ilk`) have [`AutoLine`](https://github.com/makerdao/dss-auto-line/tree/master) disabled previously or in the spell

    • Developer Guides https://github.com/makerdao/developerguides/, 1 occurrence:
      

      pe-checklists/spell/spell-reviewer-mainnet-checklist.md

      Line 199 in 1aa58ee

      * [ ] `val` makes sense in context of the [rate mechanism](https://github.com/makerdao/developerguides/blob/master/mcd/intro-rate-mechanism/intro-rate-mechanism.md)

    • Tinlake Manager https://github.com/centrifuge/tinlake-maker-lib/, 1 occurrence:
      

      pe-checklists/spell/spell-reviewer-mainnet-checklist.md

      Line 204 in 1aa58ee

      * [ ] IF `RWAXX_A_INPUT_CONDUIT` is an instance of [`TinlakeMgr`](https://github.com/centrifuge/tinlake-maker-lib/blob/master/src/mgr.sol) (it is a Centrifuge integration), additional `TinlakeMgr.tell()` call is present (in order to prevent further `TIN` redemptions in the Centrifuge pool)

    • DSS vest https://github.com/makerdao/dss-vest/, 1 occurrence:
      

      pe-checklists/spell/spell-reviewer-mainnet-checklist.md

      Line 248 in 1aa58ee

      * IF MKR stream ([DssVestTransferrable](https://github.com/makerdao/dss-vest/blob/master/src/DssVest.sol#L463)) is present

    • DSS interfaces https://github.com/makerdao/dss-interfaces/, 4 occurrences, e.g.:
      

      pe-checklists/spell/spell-crafter-mainnet-workflow.md

      Line 92 in 1aa58ee

      * [ ] Avoid `dss-interfaces` multi-import layout (see [issue #69](https://github.com/makerdao/dss-interfaces/issues/69))

    • JoinFab https://github.com/makerdao/JoinFab/, 1 occurrence:
      

      pe-checklists/spell/spell-crafter-mainnet-workflow.md

      Line 104 in 1aa58ee

      * [ ] Use [JoinFab](https://github.com/makerdao/JoinFab) to deploy

    • DSS deploy https://github.com/makerdao/dss-deploy/, 2 occurrences, e.g.:
      

      pe-checklists/spell/spell-crafter-mainnet-workflow.md

      Line 109 in 1aa58ee

      * [ ] Use [ClipFab](https://github.com/makerdao/dss-deploy) to deploy

    • DSS cron https://github.com/makerdao/dss-cron/, 1 occurrence:
      

      pe-checklists/spell/spell-crafter-mainnet-workflow.md

      Line 252 in 1aa58ee

      * [ ] Keeper Support for New Lerps (to call `tall` daily) via [dss-cron](https://github.com/makerdao/dss-cron)

  • Update Foundry (depends on github releases)
    

    pe-checklists/spell/spell-reviewer-mainnet-checklist.md

    Line 32 in 1aa58ee

    * [ ] Update Foundry by running `foundryup`

  • Fetch git submodules in spells-mainnet
    

    pe-checklists/spell/spell-reviewer-mainnet-checklist.md

    Line 33 in 1aa58ee

    * [ ] Reinstall libraries by running `rm -rf ./lib && git submodule update --init --recursive`

  • Publish spell reviews at https://github.com/makerdao/spells-mainnet/

• https://ipfs.io

  • Get pre-computed rates, 2 occurrences:
    https://github.com/makerdao/pe-checklists/blob/1aa58eeb9044ac332df4351b80e9b9755454972f/spell/spell-reviewer-mainnet-checklist.md?plain=1#L71C30-L71C45
    

    pe-checklists/spell/spell-reviewer-mainnet-checklist.md

    Line 103 in 1aa58ee

    * [ ] Double check that rate match [IPFS](https://ipfs.io/ipfs/QmVp4mhhbwWGTfbh2BzwQB9eiBrQBKiqcPRZCaAxNUaar6) document

• https://www.epochconverter.com

  • Convert timestamp
    https://github.com/makerdao/pe-checklists/blob/1aa58eeb9044ac332df4351b80e9b9755454972f/spell/spell-reviewer-mainnet-checklist.md?plain=1#L77C11-L78

• https://etherscan.io

  • Check Rely events
    

    pe-checklists/spell/spell-reviewer-mainnet-checklist.md

    Line 92 in 1aa58ee

    * [ ] Ensure that there are no other `Rely` events except for `PAUSE_PROXY` and `MCD_ESM` (using a block explorer like [etherscan](https://etherscan.io))

  • Check source code of a new contract
    

    pe-checklists/spell/spell-reviewer-mainnet-checklist.md

    Line 93 in 1aa58ee

    * [ ] Source code matches corresponding github source code (e.g. diffcheck via vscode `code --diff etherscan.sol github.sol`)

  • Check deployed spell (manually)
    

    pe-checklists/spell/spell-reviewer-mainnet-checklist.md

    Line 347 in 1aa58ee

    * [ ] Deployed spell is verified on etherscan

    
    

    pe-checklists/spell/spell-reviewer-mainnet-checklist.md

    Line 368 in 1aa58ee

    * [ ] Ensure Etherscan `Libraries Used` matches DssExecLib [Latest Release](https://github.com/makerdao/dss-exec-lib/releases/latest)

  • Checks deployed spell (using API)
    

    pe-checklists/spell/spell-reviewer-mainnet-checklist.md

    Lines 355 to 360 in 1aa58ee

    	* Automated checks via `make check-deployed-spell`
    	* [ ] Verified
    	* [ ] Valid license
    	* [ ] Version matches
    	* [ ] Optimizations are disabled
    	* [ ] dss-exec-lib library address used (under 'Libraries Used') matches the hardcoded local `DssExecLib.address` file

  • Get priority fee
    

    pe-checklists/spell/spell-crafter-mainnet-workflow.md

    Line 182 in 1aa58ee

    * [ ] Check [current gas priority fee](https://etherscan.io/gastracker) and set `ETH_PRIO_FEE` accordingly (e.g. 2 gwei)

• https://tenderly.co

  • Create testnet (using API)
    

    pe-checklists/spell/spell-crafter-mainnet-workflow.md

    Lines 200 to 203 in 1aa58ee

    	* Cast spell on a newly created Tenderly Testnet
    	* [ ] Create testnet and cast deployed spell there using `make cast-on-tenderly spell=0x...` command (or a CI trigger)
    	* [ ] Check that returned `public explorer url` is publicly accessible (e.g. using incognito browser mode)
    	* [ ] If `cast-on-tenderly` command is executed several times for the same spell, delete all testnets of the same name except the last one

  • Review testnet (manually)
    

    pe-checklists/spell/spell-reviewer-mainnet-checklist.md

    Lines 370 to 376 in 1aa58ee

    	* Tenderly Testnet checks
    	* [ ] A testnet with the name matching spell description is found at [maker dashboard](https://dashboard.tenderly.co/maker/virtual-networks)
    	* [ ] The testnet name is unique (previous testnets does not have the same name)
    	* [ ] Cast transaction is set to the correct "receiver" (matches deployed spell address)
    	* [ ] All actions are executed in the transaction trace
    	* [ ] No reverts are present that block execution
    	* [ ] No out-of-gas errors are present

• https://discord.com

  • Handover deployed spell
    

    pe-checklists/spell/spell-reviewer-mainnet-checklist.md

    Lines 393 to 394 in 1aa58ee

    	* [ ] Check that the spell address posted by the crafter in [`new-spells`](https://discord.com/channels/893112320329396265/897483518316265553) is correct
    	* [ ] Confirm the address in the [`new-spells`](https://discord.com/channels/893112320329396265/897483518316265553) channel (via a separate "reply to" message, restating the address to avoid edits)

• https://signal.org

  • Spell-related communication

[SidestreamColdMelon]

Potential circumventions

Most used services are not lock-in (e.g.: git, ipfs, xlsx, messaging) to circumvent their offline/compromised state we can just pre-define their alternatives. Less portable are spreadsheet comments, github PR reviews and discord channels as they have additional permissions attached to them. The most centralized services that does not have easily interchangeable alternatives are etherscan.io and tenderly.co.

1. Pre-define alternative services / communication channels. Applicable to:

   • Exec Sheet hosted on https://docs.google.com/spreadsheets/
     • Define a process in which instructions (e.g. raw xlsx file) can be sent over a group message or hosted on a different server if google docs are down
   • All repositories hosted on http://github.com
     • Mirror existing repositories into another git providers, use CI to keep them in sync
   • Source code verified on https://etherscan.io
     • Use multiple different services to verify the source code
   • Rates hosted on https://ipfs.io
     • Define different trusted ipfs gateways

2. Where possible, use local tools instead of services. Explicitly mark service-based checks as additional/non-blocking. Applicable to:

   • CI tests on http://github.com
     • Define this step as additional (as we're already running tests locally) or use 2 other CI providers
   • References to other repos at http://github.com
     • Write a script to pre-fetch all referenced repositories locally
   • Timestamp converter on https://www.epochconverter.com
     • Replace with local converter command / script which is part of the repo

3. Replace non-portable linked resources (PR comments / issues / releases / wikis and other information stored outside git) with git or ipfs. Applicable to:

   • Mentioned wiki page

     pe-checklists/spell/spell-reviewer-mainnet-checklist.md

     Line 57 in 1aa58ee

     * [ ] Precision units match with [Numerical Ranges](https://github.com/makerdao/dss/wiki/Numerical-Ranges#notation)

     • Find another source for the same information inside a different repo (as wikis are actually more easily editable)
   • Mentioned github issue

     pe-checklists/spell/spell-crafter-mainnet-workflow.md

     Line 92 in 1aa58ee

     * [ ] Avoid `dss-interfaces` multi-import layout (see [issue #69](https://github.com/makerdao/dss-interfaces/issues/69))

     • Describe the problem inside the checklist, otherwise make it clear that the link is optional
   • Mentioned release

     pe-checklists/spell/spell-reviewer-mainnet-checklist.md

     Line 38 in 1aa58ee

     * [ ] git submodule hash of `dss-exec-lib` (run `git submodule status`) matches the [latest release version](https://github.com/makerdao/dss-exec-lib/releases) or newer

     • Ensure alternative check is described (local tag) or alternative remote is listed

4. Use on-chain registry of the team + attestations for the most security-crucial operations. Applicable to:

   • Spell review and deployment confirmation at https://github.com/makerdao/spells-mainnet/
     • Use attestation service to store hash + address of a new spell
   • Spell handover on https://discord.com
     • Use 3 attestations by pre-specified users to fire a handover event

[SidestreamColdMelon]

First specific issue for etherscan is created: #31