3. Syncing DB
Last updated
Last updated
Using answers for these questions you can easily understand which snapshot type you will need
This document introduces another centralized fast state syncing method using rclone. Please use it with caution. This guide is mainly used for a newly started node to catch up with the blockchain faster. Otherwise, the blockchain syncing may take weeks from genesis block.
Rclone db snapshot is synced with blockchain frequently. However, there maybe a potential race condition when the rclone may fail due to our nodes were updating the db files at the same time. In this case, just re-run the rclone command to re-sync again.
For installing Rclone, please follow the instructions at https://rclone.org.
TL;DR: on a Linux system, you may run the following command:
Make sure your rclone version is above v1.53.2 . you can check version by rclone version
To check the location of the rclone.conffile run:
The rclone.conf file is usually located at ~/.config/rclone/rclone.conf .
Now run the following command to create the rclone.conf file:
The most important parts in bold:
Sync the source to the destination, changing the destination only. Doesn't transfer files that are identical on source and destination, testing by size and modification time or MD5SUM. Destination is updated to match source, including deleting files if necessary (except duplicate objects). Link to the full rclone documentation.
sync command was chosen because you need to have your blockchain data folder to be identical with the snapshot data
As user you need to carefully get the full path to the folder and as destination. If you have some important data in the destination, please move it somewhere else.
Usually data dir folder is `./` which means folder in which harmony is running
You can double check this via:
ls -l | grep 'harmony_db_'
Expected result id is name with shard number on the end, e.x. harmony_db_0.
Q: How to get the full path to the your blockchain data folder if it isn't default?
Check how do you start harmony via ps -ef | grep harmony and check for the --datadir option, if you don't have it go to the following step, else to the 3rd
Check the harmony.conf for DataDir option or check your grep DataDir harmony.conf
Go to the folder from the cmd option if have it or to folder from config if you don't have cmd option, do the following:
SnapDB is a type of DB that contain a snapshot of the state at a given block. It doesn't have any block history and hence not suitable for RPC node.
Below is the command to sync the Snap DB for the shard 0. It is around 100 Gb as of October 2023:
Important: Since this can cause data loss, test first with the --dry-run or the --interactive/-i flag.
Small explanation for flags used to save your time with rclone manual:
Since v4.3.12 shard 1 doesn't need to download anymore shard 0 DB. Instead, the node will download automatically the blocks required for the node in shard 1 to work.
Each node will simply need to rclone its own DB.
Important: Since this can cause data loss, test first with the --dry-run or the --interactive/-i flag.
Important: Since this can cause data loss, test first with the --dry-run or the --interactive/-i flag.
As of 1st October 2023, the size for the shard 0 on mainnet is ~23TiB. Note that the bucket is currently not fully synced.
Please contact us at devops@harmony.one if you need to access our archival db.
If you want to quickly test your validator/RPC setup via the testnet network, please use the setup from 2. Configuring Rcloneand the info below.
Important: Since this can cause data loss, test first with the --dry-run or the --interactive/-i flag.
Important: Since this can cause data loss, test first with the --dry-run or the --interactive/-i flag.
