Software upgrade documentation

docs/menu/index.md

@@ -12,6 +12,7 @@ headless: true
   - [Multisig guide]({{< relref "/docs/go-cyber/multisig_guide.md" >}})
   - [Run validator with Tendermint and Ledger]({{< relref "/docs/go-cyber/validator_launch_with_tendermintKMS_and_ledger.md" >}})
   - [Set up config.toml in the right way]({{< relref "/docs/go-cyber/setup_cyberd_configuration.md" >}})
+  - [Upgrade guide]({{< relref "/docs/go-cyber/upgrade_guide.md" >}})
 - Dev environment:
   - [Setup development environment]({{< relref "/docs/go-cyber/setup_dev_env.md" >}})
 - Specs:

docs/run_validator.md

@@ -49,47 +49,47 @@ As long as the current implementation of `cyber` is written in [Go](https://gola
 
 For `euler-6` Cyberd requires at least Go version 1.13+. Install it according to the official [guide](https://golang.org/doc/install):
 
-- Download the archive:
+1. Download the archive:
 
-```bash
-wget https://dl.google.com/go/go1.13.11.linux-amd64.tar.gz
-```
+    ```bash
+    wget https://dl.google.com/go/go1.13.11.linux-amd64.tar.gz
+    ```
 
-- Extract it into `/usr/local`, creating a Go tree in `/usr/local/go`:
+2. Extract it into `/usr/local`, creating a Go tree in `/usr/local/go`:
 
-```bash
-tar -C /usr/local -xzf go1.13.11.linux-amd64.tar.gz
-```
+    ```bash
+    tar -C /usr/local -xzf go1.13.11.linux-amd64.tar.gz
+    ```
 
-- Add `/usr/local/go/bin` to the PATH environment variable. You can do this by adding this line to your `/etc/profile` (for installation on the whole system) or `$HOME/.bashrc`:
+3. Add `/usr/local/go/bin` to the PATH environment variable. You can do this by adding this line to your `/etc/profile` (for installation on the whole system) or `$HOME/.bashrc`:
 
-```bash
-export PATH=$PATH:/usr/local/go/bin
-```
+    ```bash
+    export PATH=$PATH:/usr/local/go/bin
+    ```
 
-- Do `source` for the file with your `$PATH` variable or just log-out/log-in:
+4. Do `source` for the file with your `$PATH` variable or just log-out/log-in:
 
-```bash
-source /etc/profile
-```
+    ```bash
+    source /etc/profile
+    ```
 
-or
+    or
 
-```bash
-source $HOME/.bashrc
-```
+    ```bash
+    source $HOME/.bashrc
+    ```
 
-- To check your installation run
+5. To check your installation run
 
-```bash
-go version
-```
+    ```bash
+    go version
+    ```
 
-This will let you know if everything was installed correctly. As an output, you should see the following (version number may vary, of course):
+    This will let you know if everything was installed correctly. As an output, you should see the following (version number may vary, of course):
 
-```bash
-go version go1.13.11 linux/amd64
-```
+    ```bash
+    go version go1.13.11 linux/amd64
+    ```
 
 #### Installing Nvidia drivers
 
@@ -321,16 +321,19 @@ cd $DAEMON_HOME/config
 wget -O genesis.json https://ipfs.io/ipfs/QmZHpLc3H5RMXp3Z4LURNpKgNfXd3NZ8pZLYbjNFPL6T5n
 ```
 
-Setup private peers in `config.toml`. You can find them on our [forum](https://ai.cybercongress.ai/t/euler-6-testnet-faq/65).
-
-### Setup cyberd as a service (Ubuntu)
-
-Increase resource limits for [Tendermint](https://tendermint.com):
+Setup private peers in `config.toml`.
 
 ```bash
-ulimit -n 4096
+# Comma separated list of nodes to keep persistent connections to
+persistent_peers = "d0518ce9881a4b0c5872e5e9b7c4ea8d760dad3f@85.10.207.173:26656,0f7d8d5bb8e831a67d29d5950cff0f0ecafbab54@195.201.105.229:36656,30d949f592baf210dd2fc500c83f087f7ce95a84@86.57.254.202:36656"
 ```
 
+You can more of them on our [forum](https://ai.cybercongress.ai/t/euler-6-testnet-faq/65), the more the better for now.
+
+### Setup cyberd as a service (Ubuntu)
+
+#### Service setup for cyber node
+
 Make cyberd a system service. This will help you easily start/stop cyberd and run it in the background:
 
 ```bash
@@ -360,7 +363,11 @@ LimitNOFILE=4096
 WantedBy=multi-user.target
 ```
 
-If you need to enable search of the node add the flag `--allow-search=true` right after `--compute-rank-on-gpu=true`. If you need to run a rest-server alongside `cyberd` here is a service file for it (do `sudo nano /etc/systemd/system/cyberdcli-rest.service` and paste the following), just make sure you'll replace `ubuntu` to your user name and group:
+If you need to enable search of the node add the flag `--allow-search=true` right after `--compute-rank-on-gpu=true`.
+
+#### Run rest service with cyberdcli
+
+If you need to run a rest-server alongside `cyberd` here is a service file for it (do `sudo nano /etc/systemd/system/cyberdcli-rest.service` and paste the following), just make sure you'll replace `ubuntu` to your user name and group:
 
 ```bash
 [Unit]
@@ -378,16 +385,13 @@ RestartSec=30
 WantedBy=multi-user.target
 ```
 
-There's a possibility to build and run swagger-ui for your node to get a better experience with the rest-server. To get it up you'll have to install `static` library for Go: 
-
-```bash
-go get github.com/rakyll/statik
-```
+**Run swagger-ui on top of cyberdcli rest server**
 
-Then `cd` to the go-cyber repo and set static file for swagger-ui:
+There's a possibility to build and run swagger-ui for your node to get a better experience with the rest-server. To get it up you'll have to `cd` to the go-cyber repo, install `static` library for Go and set static file for swagger-ui:
 
 ```bash
 cd <path_to_go-cyber>/go-cyber/
+apt install golang-statik
 statik -src=cmd/cyberdcli/swagger-ui -dest=cmd/cyberdcli/lcd -f
 ```
 
@@ -400,7 +404,7 @@ cp build/cyberdcli /usr/local/bin/
 
 When all of the above steps are completed and cyberdcli-rest service has been started, you should have Swagger-ui available at `http://localhost:1317/swagger-ui/` .
 
-Run cyberd:
+#### Launch cyberd
 
 Reload `systemd` after the creation of the new service:
 

docs/upgrade_guide.md

@@ -0,0 +1,236 @@
+# How to upgrade cyber node with cosmosd
+
+Upgrade to newer version of binaries will be covered in this guide. Go-cyber using [cosmosd](https://github.com/regen-network/cosmosd) upgrade manager and [upgrade module](https://docs.cosmos.network/master/modules/upgrade/) for on-chain upgrades.
+
+If you set up your node using this [Run validator guide](https://github.com/cybercongress/go-cyber/blob/master/docs/run_validator.md) it should be pretty eady to prepare for upgrade.
+
+## Definition
+
+Current upgrade named Darwin, so all further referencies will be made to that name. Overal upgrade process is basically tied to changing binary file of the cyberd to new version, at the same time for all validator nodes. Governance proposal for upgrade will define name for upgrade and block in the chain, when this upgrade should happen. New binary file must be pre-compiled (or downloaded), and placed rigth inside `.cyberd` folder. Whole uprade process is as simple as theese steps:
+
+- newly compiled `cyberd` binary file is being placed into correct directory
+
+- when desired upgrade block comes `cosmosd` catches event and stops cyber node
+
+- `cosmosd` starting node from new binary file from the directory with upgrade name
+
+### Cosmosd explained
+
+[Cosmosd](https://github.com/regen-network/cosmosd) is a tool designed by [Regen-Network](https://github.com/regen-network) devs. It is designed for smooth and configurable management of upgrading binaries as a live chain is upgraded, as well as makes syncing a full node for genesis simplier. The upgrade manager will monitor the stdout of the daemon to look for messages from the upgrade module indicating a pending or required upgrade and act appropriately.
+
+Since the version 0.1.6 (Euler-6) upgrade module has been integrated to cyber, which add possibility to make new kind of proposal called `software-upgrade`. This proposal, if passed, set specific block or time when upgrade should happen as well as specifies unique upgrade name. By default cosmosd is launching deamon file that is located under `DAEMON_HOME/upgrade_manager/genesis/` folder. When upgrade time comes `cyberd` generates to it's stdout sting looks like following:
+
+```bash
+Jul 16 17:25:11 mars cosmosd[5791]: E[2020-07-16|17:25:11.325] UPGRADE "darwin" NEEDED at height: 2000:     module=main
+```
+
+Cosomosd catching that string at deamon stdout, stops cyberd and changes symbolic link current from
+
+```bash
+current -> /root/.cyberd_testnet/upgrade_manager/genesis
+```
+
+to new one according with upgrade name
+
+```bash
+current -> /root/.cyberd_testnet/upgrade_manager/upgrades/darwin
+```
+
+After link changed to new binary `cosmosd` will attempt to restart node using new daemon binary file.
+
+## Preparation
+
+There is a few checkpoints each validator must comlpy with to make upgrade succesfull:
+
+1. Correct version of cosmosd
+
+2. Correctly set `cyberd.service` file
+
+3. New `cyberd` binary file is compiled and placed correctly
+
+Let's dig into each point separately.
+
+### Cosmosd version check
+
+Cosmosd must be compiled from the latest commit of [cosmosd-repo](https://github.com/regen-network/cosmosd). Correct commit is `984175ff7c0a3815dcf2e99329cb3fbf7d7bb0b9`, so if you have cosmosd repo cloned on your node just do:
+
+```bash
+cd /path_to_cosmosd/cosmosd
+git log
+```
+
+and make sure current commit is same or newer than `984175ff7c0a3815dcf2e99329cb3fbf7d7bb0b9`. Released version 0.2.0 has a bug, which will fail upgrade. So if your cosmosd folder loks like this
+
+```bash
+cosmosd-0.2.0
+```
+
+you are on the wrong version, and will have to correct this.
+
+To get correct version do the following:
+
+```bash
+git clone https://github.com/regen-network/cosmosd.git
+cd cosmosd/
+go build
+service cyberd stop
+mv cosmosd $DAEMON_HOME/
+chmod +x $DAEMON_HOME/cosmosd
+service cyberd start
+```
+
+Attention - please, build cosmosd from latest master and double-check your version
+
+### Cyberd.service check
+
+Cyberd.service responsibe for cyberd operation must be 100% similar to current [doc](https://github.com/cybercongress/go-cyber/blob/master/docs/run_validator.md).
+
+To check it open
+
+```bash
+sudo nano /etc/systemd/system/cyberd.service
+```
+
+and verify that it identical to followng (of cource, except usernames and path to folders):
+
+```json
+[Unit]
+Description=Cyber Node
+After=network-online.target
+
+[Service]
+User=ubuntu
+WorkingDirectory=/home/ubuntu/.cyberd/
+ExecStart=/home/ubuntu/.cyberd/cosmosd start --compute-rank-on-gpu=true
+Environment=DAEMON_HOME=/home/ubuntu/.cyberd
+Environment=DAEMON_NAME=cyberd
+Environment=GAIA_HOME=/home/ubuntu/.cyberd
+Environment=DAEMON_RESTART_AFTER_UPGRADE=on
+Restart=always
+RestartSec=3m
+LimitNOFILE=4096
+
+[Install]
+WantedBy=multi-user.target
+```
+
+The most important part is
+
+```bash
+Environment=DAEMON_RESTART_AFTER_UPGRADE=on
+```
+
+Because it was missing in out earlier versions of run validator guide, and higly important for node start after upgrade.
+
+If service file was modified, systemd must be reloaded to apply changes:
+
+```bash
+systemctl daemon-reload
+```
+
+### Prepraing new cyberd binary
+
+Upgrade will be released with tag v0.1.6.3, so technically all that required to get it is pull changes inside cloned go-cyber repo:
+
+```bash
+cd /<path_to_go-cyber>/go-cyber/
+git reset --hard 
+git checkout master
+git pull
+```
+
+and then build all in the same way as during setup, cuda code first, `cyberd` binary next:
+
+```bash
+cd /<path_to_go-cyber>/go-cyber/x/rank/cuda/
+make
+cd /<path_to_go-cyber>/go-cyber/
+make build
+```
+
+After all above done, new `cyberd` and `cyberdcli` binaries must appear inside ~/go-cyber/build/.
+
+Next we need to create folders tree with upgrade name and place binaries to appropriate location:
+
+```bash
+mkdir $DAEMON_HOME/upgrade_manager/upgrades/
+mkdir $DAEMON_HOME/upgrade_manager/upgrades/darwin/
+mkdir $DAEMON_HOME/upgrade_manager/upgrades/darwin/bin/
+cp build/cyberd $DAEMON_HOME/upgrade_manager/upgrades/darwin/bin/
+cp build/cyberdcli /usr/local/bin/
+cp build/cyberd /usr/local/bin/
+chmod +x $DAEMON_HOME/upgrade_manager/upgrades/darwin/bin/cyberd
+```
+
+To check that everything builded correclty run:
+
+```bash
+$DAEMON_HOME/upgrade_manager/upgrades/darwin/bin/cyberd version
+```
+
+Version must be `v0.1.6.3`.
+
+And verify upgrade_manager folder from .cyberd directory looks like this:
+
+```bash
+.
+├── current -> /root/.cyberd/upgrade_manager/genesis
+├── genesis
+│   └── bin
+│       └── cyberd
+└── upgrades
+    └── darwin
+        └── bin
+            └── cyberd
+```
+
+And here thats it for preparation side. It everythng set correctly upgrade will happen automatically.
+
+After upgrade current link will be automatically changed to ```current -> /root/.cyberd/upgrade_manager/upgrades/darwin```
+
+## Upgrade itself
+
+When upgrade proposal will be submited and upgrade block will come, cosmosd will do following on all nodes across the chain automatically:
+
+- Stop cyber node
+
+- Change `current` link to` -> .cyberd/upgrade_manager/upgrades/darwin`
+
+- Start node
+
+As soon as there will be more than ~66.66% of voting power at succesfully upgraded validators, chain will continue block production.
+
+**Important**
+
+This manual would not cover docker setup, for that case you'll have to manually build new conatiner and start it over the node state, instead of old one, manually at/or after the time of the upgrade.
+
+## Manual upgrade
+
+If something goes wrong and cosmosd didn't start with new binary at upgrade block you need do following:
+
+1. Stop cyber node:
+
+    ```bash
+     service cyberd stop
+    ```
+
+2. Go to .cyberd directory and remove current link:
+
+    ```bash
+    cd /$DAEMON_HOME/upgrade_manager/
+    rm current
+    ```
+
+3. Create new symbolic link to directory with upgraded binary
+
+    ```bash
+    ln -s /$DAEMON_HOME/upgrade_manager/upgrades/darwin/ /$DAEMON_HOME/upgrade_manager/current
+    ```
+
+    that will point cosmosd to correct binary location.
+
+4. Start cyberd service
+
+    ```bash
+    service cyberd start
+    ```