Validator Restoration

Validators (pseudo exchanges) receive trade orders from miners (traders). Due to new validators joining and unexpected downtime from individual validators, there will be times where some validators will be out of sync with the miners.

Problem statement

Like any distributed network, it’s crucial to synchronize the validator’s positions to maintain consensus.

• Failure to sync missing positions will cause miner scoring to diverge from the rest of the subnet validators.
• If scoring diverges across validators, miners won’t have their incentives correctly set and won’t be rewarded!

Phase 1 - Original solution

1. validator_checkpoint.json contains the data needed for a validator to be synched with the network.

   1. Generated by a trusted validator and uploaded to the Taoshi dashboard for real time visualization
   2. validator_checkpoint.json publicly downloadable for all

2. Validators monitor their own consensus score (vtrust attribute) and perform a restore if it gets too low

   1. Validator restore requires stopping the validator, running the restore script with a validator_checkpoint.json file, and restarting the validator
   2. Restoring deletes all existing orders and replacing them with checkpoint data (full nuke)

Phase 2 - (Old solution)

1. Discourage plagiarism and enforce the value of real time trade data that validators are selling via Request Network

   1. Remove public download of validator_checkpoint.json
   2. Dashboard visualization shows orders with a 24 hour lag
   3. Realtime validator_checkpoint.json must be requested from a team member (limit to ~1 per week)

2. Validators continue to monitor their own consensus scores (vtrust) and perform a manual restore if it gets too low.

Phase 3 - Google Cloud Auto Sync (Current Solution)

1. Toggle auto sync in launch command as a flag to have your validator automatically perform a synchronization once per day.

   1. Removes the need to reach out to a team member for the file
   2. This flag will be optional. Validators can run without auto sync

2. No longer requires a full nuke of existing orders

   1. Attempts to match existing orders with backup orders and always favors the existing one. This keeps the network distributed.
   2. A validator with 100% uptime would never have their data modified with auto sync enabled.

3. Reintroduce validator_checkpoint.json download by anyone with the public URL

   1. Restore file is 24 hour delayed. This is important because it lets us distribute the checkpoint file while discouraging plagiarism and enforcing the value of real time trade data that we are selling via Request Network
   2. This 24 hour delay is consistent with the Dashboard state
   3. This checkpoint file is generated by a single trusted validator.
   4. This is the same file that is used in the autosync logic

4. Manual restore using a realtime checkpoint is still available for validators that don’t want to enable auto sync. Performs a full nuke as before. (realtime validator_checkpoint.json distribution limited to ~1 per week)

Phase 4 - Peer to Peer Distributed Restore (Future enhancement ETA July 14th)

1. Enhance auto sync by constructing the checkpoint based off multiple high-stake validators

2. Google Cloud is no longer needed. Checkpoint creation is completely distributed and peer to peer.

3. Validators communicate to share positions to create their own “golden unified checkpoint“

   1. data could be sent on the BT network using synapses (preferred)
   2. data could be sent using the Request Network)