information on tendermint pruning bug (#559)

contrib/kava-3/migration.md

@@ -1,9 +1,11 @@
 # Kava-3 Upgrade Instructions
 
-### Background
+## Background
 
 The version of Kava for kava-3 is __v0.8__.
 
+🚨 Please note that an issue with tendermint v0.33 has been found that affects the stability of nodes running with the default pruning strategy. Please see the [Pruning](#Pruning) section for full details and mitigation 🚨
+
 Many changes have occurred in both the Kava software and the cosmos-sdk software since the launch of kava-2. The primary changes in Kava are the addition of modules that comprise the [CDP system](https://docs.kava.io/). To review cosmos-sdk changes, see the [changelog](https://github.com/cosmos/cosmos-sdk/blob/v0.38.4/CHANGELOG.md) and note that kava-3 is launching with __v0.38.4__ of the cosmos-sdk.
 
 If you have technical questions or concerns, ask a developer or community member in the [Kava discord](https://discord.com/invite/kQzh3Uv).
@@ -14,7 +16,26 @@ As a validator, performing the upgrade procedure on your consensus nodes carries
 
 The riskiest thing a validator can do is discover that they made a mistake and repeat the upgrade procedure again during the network startup. If you discover a mistake in the process, the best thing to do is wait for the network to start before correcting it. If the network is halted and you have started with a different genesis file than the expected one, seek advice from a Kava developer before resetting your validator.
 
+### Pruning
+
+kava-3 uses tendermint version 0.33. Recent testing in Game of Zones and Kava has shown that nodes which are running with the default or custom pruning strategy have a [memory leak](https://github.com/tendermint/iavl/issues/256) that can cause nodes to crash and lead to irrecoverable data loss. Until a patch is released, the __ONLY__ pruning strategies that are safe to run are `everything` (an archival node) or `nothing` (only the most recent state is kept). It is __strongly__ recommended that validators use `pruning="everything"` for kava-3.
+
+The pruning config is set in $HOME/.kvd/config/app.toml. Example safe configurations are:
+
+```toml
+pruning = "nothing"
+```
+
+and
+
+```toml
+pruning = "everything"
+```
+
+It is expected that a patch to tendermint will be released in a non-breaking manner and that nodes will be able to update seamlessly after the launch of kava-3.
+
 ### Recovery
+
 Prior to exporting kava-2 state, validators are encouraged to take a full data snapshot at the export height before proceeding. Snapshotting depends heavily on infrastructure, but generally this can be done by backing up the .kvd and .kvcli directories.
 
 It is critically important to back-up the .kvd/data/priv_validator_state.json file after stopping your kvd process. This file is updated every block as your validator participates in consensus rounds. It is a critical file needed to prevent double-signing, in case the upgrade fails and the previous chain needs to be restarted.
@@ -25,7 +46,7 @@ In the event that the upgrade does not succeed, validators and operators must do
 
 Set your node to produce the final block of kava-2 at __13:00__ UTC June 10th, 2020. To restart your node with that stop time,
 
-```
+```sh
 kvd start --halt-time 1591794000
 ```
 
@@ -75,7 +96,6 @@ kvd start --halt-time 1591794000
   kvd start
 ```
 
-
 ### Coordination
 
 If the `kava-3` chain does not launch by June 10, 2020 at 16:00 UTC, the launch should be considered a failure. Validators should restore the state from `kava-2` and coordinate a relaunch. In the event of launch failure, coordination will occur in the [Kava discord](https://discord.com/invite/kQzh3Uv).