🏗️ Developers
🌏
Network
3. Syncing DB
Validator Nodes
This document introduces another centralized fast state syncing method using rclone. Please use it with caution. This guide is mainly used for a newly started node to catch up with the blockchain faster. Otherwise, the blockchain syncing may take weeks from genesis block.
Rclone db snapshot is sync'ed with blockchain frequently. However, there maybe a potential race condition when the rclone may fail due to our nodes were updating the db files at the same time. In this case, just re-run the rclone command to re-sync again.
1. Installing Rclone
TL;DR, on a Linux system, you may run the following command.
1
curl https://rclone.org/install.sh | sudo bash
Copied!
Make sure your rclone version is above v1.53.2 . you can check version by
rclone versionThere is a daily snapshot of the rclone DB happening at 1:42:30 AM UTC for 3h. It is not advisable to perform a backup during those time.
2. Configuring Rclone
To check the location of the
rclone.conffile run:1
rclone config file
Copied!
The
rclone.conf file is usually located at ~/.config/rclone/rclone.conf .Now run the following command to create the rclone.conf file.
1
cat<<-EOF > ~/.config/rclone/rclone.conf
2
[release]
3
type = s3
4
provider = AWS
5
env_auth = false
6
region = us-west-1
7
acl = public-read
8
server_side_encryption = AES256
9
storage_class = REDUCED_REDUNDANCY
10
EOF
Copied!
The above rclone config also work for the pangaea testnet network
3. Running Rclone
Below is the command to sync the shard you want. Replace
<ShardID>with the shard number you want to sync. Shard 0 is around 300 Gb as of November 2021.Mainnet
Testnet
1
rclone -P -L --checksum sync release:pub.harmony.one/mainnet.min/harmony_db_<ShardID> harmony_db_<ShardID> --multi-thread-streams 4 --transfers=32
Copied!
1
rclone -P -L --checksum sync release:pub.harmony.one/testnet.min/harmony_db_<ShardID> harmony_db_<ShardID> --multi-thread-streams 4 --transfers=32
Copied!
If you encounter the following error:
1
NOTICE: Config file "/root/.config/rclone/rclone.conf" not found - using defaults
Copied!
Add the option
--config=/home/harmony/.config/rclone/rclone.conf to the command line.Nodes in shard 0 just need to sync
harmony_db_0Nodes in shard 1, 2, 3 need to sync both
harmony_db_0, and harmony_db_<ShardID>4. Cheat Sheet
shard0:
Mainnet
Testnet
1
rclone -P -L --checksum sync release:pub.harmony.one/mainnet.min/harmony_db_0 harmony_db_0 --multi-thread-streams 4 --transfers=32
Copied!
1
rclone -P -L --checksum sync release:pub.harmony.one/testnet.min/harmony_db_0 harmony_db_0 --multi-thread-streams 4 --transfers=32
Copied!
Shard 1:
Mainnet
Testnet
1
rclone -P -L sync release:pub.harmony.one/mainnet.min/harmony_db_0 harmony_db_0 --multi-thread-streams 4 --transfers=8
2
rclone -P -L sync release:pub.harmony.one/mainnet.min/harmony_db_1 harmony_db_1 --multi-thread-streams 4 --transfers=8
Copied!
1
rclone -P -L --checksum sync release:pub.harmony.one/testnet.min/harmony_db_0 harmony_db_0 --multi-thread-streams 4 --transfers=32
2
rclone -P -L --checksum sync release:pub.harmony.one/testnet.min/harmony_db_1 harmony_db_1 --multi-thread-streams 4 --transfers=32
Copied!
Shard 2:
Mainnet
Testnet
1
rclone -P -L sync release:pub.harmony.one/mainnet.min/harmony_db_0 harmony_db_0 --multi-thread-streams 4 --transfers=8
2
rclone -P -L sync release:pub.harmony.one/mainnet.min/harmony_db_2 harmony_db_2 --multi-thread-streams 4 --transfers=8
Copied!
1
rclone -P -L --checksum sync release:pub.harmony.one/testnet.min/harmony_db_0 harmony_db_0 --multi-thread-streams 4 --transfers=32
2
rclone -P -L --checksum sync release:pub.harmony.one/testnet.min/harmony_db_2 harmony_db_2 --multi-thread-streams 4 --transfers=32
Copied!
Shard 3:
Mainnet
Testnet
1
rclone -P -L --checksum sync release:pub.harmony.one/mainnet.min/harmony_db_0 harmony_db_0 --multi-thread-streams 4 --transfers=32
2
rclone -P -L --checksum sync release:pub.harmony.one/mainnet.min/harmony_db_3 harmony_db_3 --multi-thread-streams 4 --transfers=32
Copied!
1
rclone -P -L sync release:pub.harmony.one/testnet.min/harmony_db_0 harmony_db_0 --multi-thread-streams 4 --transfers=8
2
rclone -P -L sync release:pub.harmony.one/testnet.min/harmony_db_3 harmony_db_3 --multi-thread-streams 4 --transfers=8
Copied!
After the sync, you may use
du -h harmony_db_* command to check the size of the downloaded snapshots.-P will display a download progress & ETA.Mainnet S0 Snapshot DB for validator
SnapDB is a new type of DB that doesn't contain any data before a given block (ie as of 4th March 2022, this is block 22816573), hence not suitable for RPC node
Mainnet validator in all shard 0/1/2/3 can use it with a size of 257G (as of 4th March 2022)
1
rclone -P -L --checksum sync release:pub.harmony.one/mainnet.snap/harmony_db_0 harmony_db_0 --multi-thread-streams 4 --transfers=32
Copied!
Non-Validating/Explorer Nodes
As of 17th December 2021, the size for the shard 0 on mainnet is ~16Tb. Note that the bucket is currently not fully synced.
To sync your node in archival mode, just change the command below for the shard you want. Depending on the size, the syncing of the blockchain will take quite some time.
1
rclone -P -L --checksum sync release:pub.harmony.one/mainnet.archival/harmony_db_<ShardID> harmony_db_<ShardID> --multi-thread-streams 4 --transfers=32
Copied!
Example for shard 0:
1
rclone -P -L --checksum sync release:pub.harmony.one/mainnet.archival/harmony_db_0 harmony_db_0 --multi-thread-streams 4 --transfers=32
Copied!
Last modified 2mo ago