Syncing with Rclone

This document introduces another centralized fast state syncing method using rclone. Please use it with caution. This guide is mainly used for a newly started node to catch up with the blockchain faster. Otherwise, the blockchain syncing may take weeks from genesis block.
Rclone db snapshot is sync'ed with blockchain frequently. However, there maybe a potential race condition when the rclone may fail due to our nodes were updating the db files at the same time. In this case, just re-run the rclone command to re-sync again.
1. Installing Rclone
For installing Rclone, please follow the instructions at https://rclone.org.
TL;DR, on a Linux system, you may run the following command.
curl https://rclone.org/install.sh | sudo bash
make sure your rclone version is above v1.53.2 . you can check version by rclone version
2. Configuring Rclone
To check the location of the rclone.conffile run:
rclone config file
The rclone.conf file is usually located at ~/.config/rclone/rclone.conf . 
Now run the following command to create the rclone.conf file.
cat<<-EOF > ~/.config/rclone/rclone.conf
[mainnet]
type = s3
provider = DigitalOcean
env_auth = false
region = sfo2
acl = public-read
endpoint = sfo2.digitaloceanspaces.com
EOF
The above rclone config also work for the pangaea testnet network
3. Running Rclone
Below is the command to sync the shard you want. Replace <ShardID>with the shard number you want to sync. Each of the rclone snapshot is around 1.7 Gb as of 05/14/2020. Shard 0 is around 1.8 Gb as of 5/14/2020. 
It may take up to 10 minutes to download depending on your network connection.
Mainnet
Testnet
Mainnet
rclone -P sync mainnet:pub.harmony/mainnet.min/harmony_db_<ShardID> harmony_db_<ShardID>
Testnet
rclone -P sync mainnet:pub.harmony/testnet.min/harmony_db_<ShardID> harmony_db_<ShardID>
Nodes in shard 0 just need to sync harmony_db_0
Nodes in shard 1, 2, 3 need to sync both harmony_db_0, and harmony_db_<ShardID>
4. Cheat Sheet
shard0:
Mainnet
Testnet
Mainnet
rclone -P sync mainnet:pub.harmony/mainnet.min/harmony_db_0 harmony_db_0
Testnet
rclone -P sync mainnet:pub.harmony/testnet.min/harmony_db_0 harmony_db_0
Shard 1:
Mainnet
Testnet
Mainnet
rclone -P sync mainnet:pub.harmony/mainnet.min/harmony_db_0 harmony_db_0
rclone -P sync mainnet:pub.harmony/mainnet.min/harmony_db_1 harmony_db_1
Testnet
rclone -P sync mainnet:pub.harmony/testnet.min/harmony_db_0 harmony_db_0
rclone -P sync mainnet:pub.harmony/testnet.min/harmony_db_1 harmony_db_1
Shard 2:
Mainnet
Testnet
Mainnet
rclone -P sync mainnet:pub.harmony/mainnet.min/harmony_db_0 harmony_db_0
rclone -P sync mainnet:pub.harmony/mainnet.min/harmony_db_2 harmony_db_2
Testnet
rclone -P sync mainnet:pub.harmony/testnet.min/harmony_db_0 harmony_db_0
rclone -P sync mainnet:pub.harmony/testnet.min/harmony_db_2 harmony_db_2
Shard 3:
Mainnet
Testnet
Mainnet
rclone -P sync mainnet:pub.harmony/mainnet.min/harmony_db_0 harmony_db_0
rclone -P sync mainnet:pub.harmony/mainnet.min/harmony_db_3 harmony_db_3
Testnet
rclone -P sync mainnet:pub.harmony/testnet.min/harmony_db_0 harmony_db_0
rclone -P sync mainnet:pub.harmony/testnet.min/harmony_db_3 harmony_db_3
After the sync, you may use du -h harmony_db_* command to check the size of the downloaded snapshots.
-P will display a download progress & ETA.
Archival Node
As of 24th October 2020, the size for the shard 0 on mainnet is ~835Gb.
To sync your node in archival mode, just change the command below for the shard you want. Depending on the size, the syncing of the blockchain will take quite some time.
rclone -P sync mainnet:pub.harmony/mainnet.archival/harmony_db_<ShardID> harmony_db_<ShardID>
Example for shard 0:
rclone -P sync mainnet:pub.harmony/mainnet.archival/harmony_db_0 harmony_db_0
​
Setting up BLS Keys
Installing & Updating
Last updated 2 months ago