Harmony
Support ForumGithubTelegramDiscord
  • Welcome
  • 🎶General
    • Introduction
      • What is Harmony?‌
      • Roadmap
      • Strategy & Architecture
      • Study Materials
      • FAQ
    • Technology
      • Key Features
      • Randomness
      • Sharding
      • Consensus
      • Effective Proof-of-Stake
      • Tokenomics
      • Transactions
    • LayerZero Bridge
      • Important Notice
      • Bridge tutorial
      • FAQ
        • What token will I get after the bridge
        • How to find a bridged token address
    • Ecosystem
      • Wallets
        • 1Wallet
        • Gnosis Safe
        • Hardware Wallets
          • Trezor
          • Ledger Nano
            • Ledger with Metamask
            • Download & Setup
            • Ledger with HMY CLI
            • Ledger with Staking Dashboard
              • Sign In With Ledger
              • Sending transactions via Ledger
              • Staking Transactions via Ledger
          • Safepal
            • Download & setup
            • Create/import account
            • Send transaction
        • Browser Extension Wallets
          • Metamask
            • Installing MetaMask
            • Adding Harmony
            • Create a New Account
            • Sending & Receiving
            • Adding Custom Harmony Tokens
            • Import an Account
          • Math Wallet
            • Download & Setup
            • Create/Import/Export Wallet
            • Sending Transactions
            • Staking
            • Collecting Rewards
            • Undelegating
          • Harmony Chrome Extension Wallet
        • Desktop Wallets
          • Guarda Wallet
          • Staking4All Wallet
        • Mobile Wallets
          • Metamask Wallet
          • Blits Wallet
          • Cobo Wallet
          • Frontier Wallet
          • Guarda Wallet
          • Infinity Wallet
          • ONTO Wallet
          • Sef Wallet
          • Trust Wallet
          • Trustee Wallet
        • Web Wallets
          • Guarda Wallet
        • HMY CLI (Harmony Command Line Interface)
          • Download & Setup
          • Create or Import Wallet
          • Sending Transactions
          • Staking Transactions
          • Querying Balances
          • Querying the Blockchain
          • List of Transaction Error Messages
          • Cookbook
          • Other CLI References
      • DApps
        • User Guide
        • DeFi
          • Sushi
          • Onsen
        • CLI 1Wallet User Guide
        • Media
          • Timeless
        • FAQ
        • NFTs
      • Partners
        • Exchanges
        • Fiat Gateways
        • DeFi Protocols
      • Integrations
      • Cross-Border Finance
      • DAOs
        • Components & Tools
          • Snapshot
        • Community DAO
        • Validator DAO
        • Developer DAO
    • Community
  • 🏗️Developers
    • Getting Started
      • Network & Faucets
      • List of RPC Providers
      • Remix IDE
      • Dev Environment Setup
      • Ethereum Compatibility
    • Deploying on Harmony
      • Using Remix
        • Ethereum Remix
        • Harmony Remix
      • Using Truffle
      • Using Hardhat
      • Using Web3
      • Using Harmony-JS
        • Setup
        • Compile & Deploy
        • Demo: Deploying an Ethereum Smart Contract on Harmony
      • Deploy HRC20
      • Smart Contract Verification
    • SDK
      • Web3.js
        • Using Web3.js to Send Transactions on Harmony
        • Find the last transaction
      • JavaScript SDK
      • Go CLI
      • Java SDK
      • Python SDK
      • Harmony Ethers.js Wrapper
    • API
      • Methods
        • Account Methods
          • hmy_getBalanceByBlockNumber
          • hmy_getTransactionCount
          • hmy_getBalance
        • Filter Methods
          • hmy_getFilterLogs
          • hmy_newFilter
          • hmy_newPendingTransactionFilter
          • hmy_newBlockFilter
          • hmy_getFilterChanges
          • hmy_getLogs
        • Transaction Related Methods
          • hmy_getStakingTransactionByBlockHashAndIndex
          • hmy_getStakingTransactionByBlockNumberAndIndex
          • hmy_getStakingTransactionByHash
          • hmy_getCurrentTransactionErrorSink
          • hmy_getPendingCrossLinks
          • hmy_getPendingCXReceipts
          • hmy_getCXReceiptByHash
          • hmy_pendingTransactions
          • hmy_sendRawStakingTransaction
          • hmy_getTransactionsHistory
          • hmy_sendRawTransaction
          • hmy_getTransactionReceipt
          • hmy_getBlockTransactionCountByHash
          • hmy_getBlockTransactionCountByNumber
          • hmy_getTransactionByHash
          • hmy_getTransactionByBlockNumberAndIndex
          • hmy_getTransactionByBlockHashAndIndex
          • hmy_getBlockByNumber
          • hmy_getBlockByHash
          • hmy_getBlocks
        • Contract Related Methods
          • hmy_estimateGas
          • hmy_getStorageAt
          • hmy_call
          • hmy_getCode
        • Protocol Related Methods
          • hmy_isLastBlock
          • hmy_epochLastBlock
          • hmy_latestHeader
          • hmy_getShardingStructure
          • hmy_blockNumber
          • hmy_syncing
          • hmy_gasPrice
          • net_peerCount
          • hmy_getEpoch
          • hmy_getLeader
        • Staking Related Methods
          • hmy_getCirculatingSupply
          • hmy_getTotalSupply
          • hmy_getStakingNetworkInfo
          • hmy_getAllValidatorInformation
          • hmy_getAllValidatorInformationByBlockNumber
          • hmy_getCurrentUtilityMetrics
          • hmy_getDelegationsByValidator
          • hmy_getDelegationsByDelegatorAndValidator
          • hmy_getDelegationsByDelegator
          • hmy_getValidatorMetrics
          • hmy_getMedianRawStakeSnapshot
          • hmy_getElectedValidatorAddresses
          • hmy_getAllValidatorAddresses
          • hmy_getCurrentStakingErrorSink
          • hmy_getValidatorInformation
          • hmy_getValidators
          • hmy_getSignedBlocks
          • hmy_isBlockSigner
          • hmy_getBlockSigners
        • Tracing Methods
          • trace_block
          • trace_transaction
      • Sample Code
    • Tools
      • Harmony VRF
      • The Graph
      • Envio
      • Ganache
        • Harmony Ganache
      • Harmony-React
      • Oracles
        • Band Protocol
      • Smart Contract Verification
    • Tutorials
      • Deploying HRC20
      • Deploying HRC721/NFT
      • The Graph - Subgraphs
        • Building & Deploying Subgraph (local node)
      • Using Band Oracle
      • Using Crosschain API
        • Scripts
        • Testing
        • Webserver
      • Using Web3.py & Pyhmy
      • Using IPFS & Filecoin
        • Using IPFS
        • Using NFT.storage
      • Indexing HRC20 with Envio
      • Building a Simple Metaverse Contract
      • Building a Simple Bridge with Ethereum
      • Staking for Multisig
    • DApp Examples
      • DApp Samples
      • Games
        • Harmony Puzzle
      • Cross-Chain
      • DeFi
      • Hackathons
        • DevPost
        • Hack the Horizon
      • Others
    • Wallets
      • Metamask
        • Interacting With Metamask
        • Using Metamask with Harmony Smart Contracts
        • Add or Switch to Harmony chain on Metamask
      • Harmony Chrome Extension Wallet
      • Math Wallet
      • WalletConnect
    • Harmony Stack and Projects
  • 🌏Network
    • Governance
      • Network Governance
        • Voting via Governance App
        • Voting via HMY CLI
      • HRC-20 Governance
      • FAQ
    • Validators
      • Terms & Concepts
        • Validator, BLS key, Instance
          • Shard Assignment
        • Slots Bidding and Election
        • Effective Proof-of-Stake
        • Block Reward
        • Epoch Transition
        • Slashing
        • Undelegation
      • Server Setup
        • Requirements
        • Cloud Guides
          • Digital Ocean
          • Vultr
          • AWS
          • Google Cloud
        • Raspberry Pi Guide
      • Node Setup
        • 1. HMY CLI Download
        • 2. Setting up BLS Keys
        • 3. Syncing DB
        • 4. Installing & Updating
          • Installing A Node
            • Using Node Binary
            • (Deprecated) Using AutoNode
              • Install & Run
              • Update
              • Monitor
              • BLS Key Management
              • Collect Rewards
              • Maintenance
              • Troubleshoot
              • Extra
            • (deprecated) Using Node.sh
          • Upgrading A Node
            • Using Node Binary
            • Using AutoNode (deprecated)
            • Using Node.sh (deprecated)
          • Checking A Node
      • Creating A Validator
      • Managing A Validator
        • Checking Validator Information
        • Changing Validator Information
        • Delegating To A Validator
        • Undelegating From A Validator
        • Check Your Delegations
        • Collecting Rewards
        • Adding A Validator Logo
      • Staking Dashboard Basics
      • Validator Information Terms
      • Validator Security Tips
      • Slashing and Uptime
      • Monitoring
        • Node Sync
        • Prometheus & Grafana
        • Network Status
      • Troubleshooting
        • Why am i not Elected?
        • Frequently Asked Questions (FAQ)
      • Tools
        • Telegram Bots
        • Dashboards
        • Reward Calculators
        • Text User Interface (TUI)
        • HMY Bidder
        • Mobile Apps
          • One Validator Dashboard
          • Termux
    • Delegators
      • Introduction
      • Staking
        • Via Browser
          • Staking Dashboard
          • Staking Transactions
          • Sending Transactions
        • Via Mobile
        • FAQ
      • Redelegation
      • Informational Videos
Powered by GitBook
On this page
  • Simple algorithm to distinguish between snapshot types
  • Validator Nodes
  • 1. Installing Rclone
  • 2. Configuring Rclone
  • 2.1 BEWARE: Understanding the `rclone sync`
  • 3. Shard 0 validator - Snap DB sync
  • 4. Full db sync for appropriate shard
  • 4.1 Shard0 RPC Explorer node (non archival)
  • 4.2 Shard 1 validator
  • Archival snapshot for the Non-Validating/Explorer Nodes
  • 5. Testnet snapshots
  • 5.1 Testnet shard 0
  • 5.2 Testnet shard 1

Was this helpful?

Export as PDF
  1. Network
  2. Validators
  3. Node Setup

3. Syncing DB

Previous2. Setting up BLS KeysNext4. Installing & Updating

Last updated 1 year ago

Was this helpful?

Simple algorithm to distinguish between snapshot types

Using answers for these questions you can easily understand which snapshot type you will need

Validator Nodes

This document introduces another centralized fast state syncing method using rclone. Please use it with caution. This guide is mainly used for a newly started node to catch up with the blockchain faster. Otherwise, the blockchain syncing may take weeks from genesis block.

Rclone db snapshot is synced with blockchain frequently. However, there maybe a potential race condition when the rclone may fail due to our nodes were updating the db files at the same time. In this case, just re-run the rclone command to re-sync again.

1. Installing Rclone

TL;DR: on a Linux system, you may run the following command:

curl https://rclone.org/install.sh | sudo bash

Make sure your rclone version is above v1.53.2 . you can check version by rclone version

2. Configuring Rclone

To check the location of the rclone.conffile run:

rclone config file

The rclone.conf file is usually located at ~/.config/rclone/rclone.conf .

Now run the following command to create the rclone.conf file:

cat<<-EOF > ~/.config/rclone/rclone.conf
[snap]
type = webdav
vendor = other
user = snap
pass = ufbTDtK0fENuutwuDOHae57xT8URsZVIcdotK30T5A
EOF

2.1 BEWARE: Understanding the `rclone sync`

The most important parts in bold:

sync command was chosen because you need to have your blockchain data folder to be identical with the snapshot data

As user you need to carefully get the full path to the folder and as destination. If you have some important data in the destination, please move it somewhere else.

Usually data dir folder is `./` which means folder in which harmony is running

You can double check this via:

ls -l | grep 'harmony_db_'

Expected result id is name with shard number on the end, e.x. harmony_db_0.

Q: How to get the full path to the your blockchain data folder if it isn't default?

  1. Check how do you start harmony via ps -ef | grep harmony and check for the --datadir option, if you don't have it go to the following step, else to the 3rd

  2. Check the harmony.conf for DataDir option or check your grep DataDir harmony.conf

  3. Go to the folder from the cmd option if have it or to folder from config if you don't have cmd option, do the following:

cd your_data_dir # folder from the cmd line or config
ls -l # it will give the shard folder name like harmony_db_0/
pwd # now you have full path to your data dir

3. Shard 0 validator - Snap DB sync

SnapDB is a type of DB that contain a snapshot of the state at a given block. It doesn't have any block history and hence not suitable for RPC node.

Below is the command to sync the Snap DB for the shard 0. It is around 100 Gb as of October 2023:

Important: Since this can cause data loss, test first with the --dry-run or the --interactive/-i flag.

YOUR_DATA_DIR='./' # This is the default installation, change it you use something else
rclone -P -L --webdav-url 'http://snapdb.s0.t.hmny.io/webdav'  --checksum sync \
  snap: "${YOUR_DATA_DIR}harmony_db_0" --multi-thread-streams 4 --transfers=32 --verbose

Small explanation for flags used to save your time with rclone manual:

# --webdav-url - source of the data and URL of http host to connect to
# YOUR_DATA_DIR/harmony_db_0 - your destination
# -P - flag to show the progress
# -L - Follow symlinks and copy the pointed to item
# --checksum - Normally rclone will look at modification time and size of 
#  files to see if they are equal.  If you set this flag then rclone will 
#  check the file hash and size to determine if files are equal
# snap: - the way how do you tell which config is used 
# --multi-thread-streams 4 - When using multi thread downloads this 
#  sets the maximum number of streams to use
# --transfers=32 - The number of file transfers to run in parallel.
#   It can sometimes be useful to set this to a smaller number if the remote is giving a lot of timeouts or bigger if you have lots of bandwidth and a fast remote.
# --verbose - With -v rclone will tell you about each file that is transferred and a small number of significant events.
# sync - make the origin and destination the same, after 1st run add the deltas

4. Full db sync for appropriate shard

Since v4.3.12 shard 1 doesn't need to download anymore shard 0 DB. Instead, the node will download automatically the blocks required for the node in shard 1 to work.

Each node will simply need to rclone its own DB.

4.1 Shard0 RPC Explorer node (non archival)

Important: Since this can cause data loss, test first with the --dry-run or the --interactive/-i flag.

YOUR_DATA_DIR='./' # This is the default installation, change it you use something else
rclone -P -L --webdav-url 'http://fulldb.s0.t.hmny.io/webdav' \
   --checksum sync snap: "${YOUR_DATA_DIR}harmony_db_0" --multi-thread-streams 4 --transfers=32 --verbose 

4.2 Shard 1 validator

Important: Since this can cause data loss, test first with the --dry-run or the --interactive/-i flag.

YOUR_DATA_DIR='./' # This is the default installation, change it you use something else
rclone -P -L --webdav-url 'http://fulldb.s1.t.hmny.io/webdav' \
   --checksum sync snap: "${YOUR_DATA_DIR}harmony_db_1" --multi-thread-streams 4 --transfers=32 --verbose 

Archival snapshot for the Non-Validating/Explorer Nodes

As of 1st October 2023, the size for the shard 0 on mainnet is ~23TiB. Note that the bucket is currently not fully synced.

5. Testnet snapshots

If you want to quickly test your validator/RPC setup via the testnet network, please use the setup from 2. Configuring Rcloneand the info below.

5.1 Testnet shard 0

Important: Since this can cause data loss, test first with the --dry-run or the --interactive/-i flag.

YOUR_DATA_DIR='./' # This is the default installation, change it you use something else
rclone -P -L --webdav-url 'http://fulldb.s0.b.hmny.io/webdav' \
   --checksum sync snap: "${YOUR_DATA_DIR}harmony_db_0" --multi-thread-streams 4 --transfers=32 --verbose 

5.2 Testnet shard 1

Important: Since this can cause data loss, test first with the --dry-run or the --interactive/-i flag.

YOUR_DATA_DIR='./' # This is the default installation, change it you use something else
rclone -P -L --webdav-url 'http://fulldb.s1.b.hmny.io/webdav' \
   --checksum sync snap: "${YOUR_DATA_DIR}harmony_db_1" --multi-thread-streams 4 --transfers=32 --verbose 

For installing Rclone, please follow the instructions at .

Sync the source to the destination, changing the destination only. Doesn't transfer files that are identical on source and destination, testing by size and modification time or MD5SUM. Destination is updated to match source, including deleting files if necessary (except duplicate objects). .

Please contact us at if you need to access our archival db.

🌏
https://rclone.org
Link to the full rclone documentation
devops@harmony.one