Crate pepper_sync 

Pepper Sync

Overview

Pepper-sync is a rust-based sync engine library for wallets operating on the zcash network and provides the following features:

• Non-linear scanning, allowing chain tip and targetted scanning.
• Spend-before-sync, combines the shard roots with high priority chain tip scanning to enable spending of notes as they are scanned.
• Speed, trial decryption and tree building are computed in parallel and a multi-task architecture maximizes throughput for fetching and scanning.
• Scan by shards, uses subtree metadata to create scan ranges that contain all note commitments to each shard to enable faster spending of decrypted outputs.
• Fixed memory batching, each scan worker receives a batch with a fixed number of outputs for stable memory usage.
• Pause/resume and stop, the sync engine can be paused to allow the wallet to perform time critical tasks that would require the acquisition of the wallet lock multiple times in quick succession. It can also be stopped before the wallet is fully synchronized.

Terminology

• Chain height - highest block height of best chain from the server.
• Chain tip - the range of blocks at the top of the blockchain; Starting from the lowest block which contains the last note commitment to the latest shard of each shielded protocol; Ending at the chain height.
• Wallet height - highest block height of blockchain known to the wallet.
• Fully scanned height - block height in which the wallet has completed scanning all blocks equal to and below this height.
• Shard range - the range of blocks that contain all note commitments to a fully completed shard for a given shielded protocol.
• Nullifier map - a map of all the nullifiers collected from each transaction’s shielded inputs/spends during scanning.
• Outpoint map - a map of all the outpoints collected from each transaction’s transparent inputs/spends during scanning.
• Coin - transparent output.

Initialization

1. Launch the mempool monitor for scanning transactions in the mempool.
2. Run transparent address discovery, creating scan targets for any previously unscanned transactions containing inputs or outputs related to transparent addresses generated from the wallet’s keys or stored in the wallet, up to the configured gap limit.
3. Fetch subtree metadata from the server for shard range scanning and add the initial frontiers to the wallet’s shard trees.
4. Create a new scan range from [wallet height + 1] to chain height. For first time sync, wallet height is [wallet birthday - 1].
5. Update wallet height to chain height.
6. Uses scan targets from transparent address discovery and targetted scanning to set “found note” priority ranges.
7. Finds the upper range bound of the latest orchard and sapling shard ranges and splits the scan range at the lowest height of the two, setting the upper scan range to priority “chain tip”. This ensures that both the sapling and orchard note commitments are scanned in the latest incomplete shard at the chain tip.
8. Set the first 10 blocks after the highest previously scanned blocks to “verify” priority to check for re-org.

Scanning

1. If the “batcher” task is idle, set the highest priority scan range to “scanning” priority and send it to the “batcher” task. If the scan priority is “historic”, it first splits an orchard shard range off the lower end. If all scan ranges in the wallet’s sync state are “scanned”, shutdown the sync process.
2. Batch the scan range: 2a. Stream compact blocks from server until a fixed threshold of outputs is reached. If the entire scan range is batched, the “batcher” task goes idle. 2b. Store the batch and wait until it is taken by an idle “scan worker” before returning to step 2a
3. Scan each batch: 3a. Check block hash and height continuity 3b. Collect all inputs in each compact block to populate the nullifier maps and outpoint maps for spend detection 3c. Trial decrypt all notes in each compact block 3d. Fetch and scan full transactions containing any successfully decrypted notes, creating wallet transactions 3e. Derive the nullifier and position of each successfully decrypted note 3f. Calculate the note commitment leaf and shard tree retention for each successfully decrypted note 3g. Create wallet blocks from compact blocks 3h. Send scan results back to main sync task for processing
4. Process scan results: 4a. Set surrounding shard ranges of all incoming notes to “found note” to prioritize their spendability 4b. Discover unified addresses not known to the wallet but in use on chain 4c. Add wallet transactions (containing wallet notes and coins) to the wallet 4d. Add nullifiers to the wallet’s nullifier map 4e. Add outpoints, to the wallet’s outpoint map 4f. Update the wallet’s shard trees 4g. Check output id of all coins in wallet against the outpoint map. If a spend is found, update the coin’s spend status and set the surrounding narrow range of blocks to “found note” priority 4h. Check derived nullifiers of all notes in wallet against the nullifier map. If a spend is found, update the note’s spend status and set the surrounding shard range of its corresponding shielded protocol to “found note” priority 4i. Add wallet blocks to the wallet. Only retaining blocks at scan ranges bounds, blocks containing relevant transactions to the wallet or blocks within the max verification window of the highest scanned block in the wallet 4j. Set the scan range containing the batch of scanned blocks to “scanned” priority 4k. Merge all adjacent scanned ranges together 4l. Clean wallet of data that is no longer relevant
5. Scan mempool transactions.
6. Repeat all steps.

Verification

After the sync process is initialized, it will be in a state of verification, only scanning ranges of “verify” priority to check for re-orgs. If a continuity check fails, it will set the 10 blocks below the current “verify” scan range to “verify” and truncate the wallet data. This will repeat until the continuity checks passes - proceeding to scan all other scan ranges - or the maximum verification window is exceeded, returning an error.

Configuration

Transparent Address Discovery

• Gap limit - number of unused addresses for each account
• Scopes - whether external, internal and/or refund (a.k.a ephemeral) addresses will be discovered and relevant transactions scanned.

Performance Level

• Low – Number of outputs per batch is quartered – Nullifier map only contains chain tip
• Medium – Nullifier map only contains chain tip
• High – Nullifier map has a large maximum size
• Maximum – Number of outputs per batch is quadrupled – Nullifier map has no maximum size

Sync Diagram

flowchart LR
  %% SYNC
  START([Start sync]) --> INIT["Initialization"]
  INIT --> S0
  S0{Batcher idle?}
  S0 -- No --> S6
  S0 -- Yes --> S1{All ranges in wallet state are 'scanned'?}
  S1 -- Yes --> SHUTDOWN([Shutdown sync])
  S1 -- No --> S2["Pick highest-priority scan range"]
  S2 --> S3{Priority is 'historic'?}
  S3 -- Yes --> S4["Split orchard shard range off lower end"] --> S5["Set range to 'scanning' and send to batcher"]
  S3 -- No --> S5
  S5 --> S6
  S6["Batcher: stream compact blocks until fixed output threshold; store batch; when entire scan range batched, set batcher idle"]
  subgraph WORKERS[Scan workers]
    direction LR
    W1["Scan Worker 1"]
    W2["Scan Worker 2"]
    Wn["Scan Worker N"]
  end
  S6 --> W1
  S6 --> W2
  S6 --> Wn
  W1 --> PROC_RES["Process scan results"]
  W2 --> PROC_RES
  Wn --> PROC_RES

  PROC_RES --> S7["Scan mempool transactions"]
  S7 --> S0

Initialization Diagram

flowchart LR
  %% INITIALIZATION
  START([Start initialization]) --> I1["Launch mempool monitor"]
  I1 --> I2["Transparent address discovery"]
  I2 --> I3["Fetch subtree metadata for shard-range scanning and add initial frontiers to wallet's shard trees"]
  I3 --> I4{First-time sync?}
  I4 -- Yes --> I4Y["Set wallet height = [wallet birthday - 1]"] --> I5
  I4 -- No  --> I4N["Keep existing wallet height"] --> I5
  I5["Create scan range: [wallet height + 1] -> chain height"] --> I6["Update wallet height -> chain height"]
  I6 --> I7["Set 'found note' priority ranges (transparent discovery + targeted scanning)"]
  I7 --> I8["Find latest orchard and sapling shard upper bounds; split at the lowest height; set upper split to 'chain tip' priority"]
  I8 --> I9["Set first 10 blocks after highest previously scanned -> 'verify' (re-org check)"]
  I9 --> END([Proceed to verification])

Verification Diagram

flowchart LR
  %% VERIFICATION
  SV([Start verification]) --> V1{Continuity check fails?}
  V1 -- Yes --> V2["Set 10 blocks below current 'verify' range -> 'verify' and truncate wallet data"] --> V3{Exceeded max verification window?}
  V3 -- Yes --> V4([Return error])
  V3 -- No  --> SV
  V1 -- No  --> V5([Proceed to scanning lower priority scan ranges])

Scan Worker Diagram

flowchart TD
  %% SCAN WORKER
  START([Receive scan task]) --> WS1["Check block hash and height continuity"]
  WS1 --> WS2["Collect inputs -> build nullifier and outpoint maps"]
  WS2 --> WS3["Trial decrypt all notes in each compact block"]
  WS3 --> WS4["Fetch and scan full transactions for successful decrypts -> create wallet transactions"]
  WS4 --> WS5["Derive nullifier and position for each decrypted note"]
  WS5 --> WS6["Compute note commitment leaf and shard tree retention"]
  WS6 --> WS8["Create wallet blocks from compact blocks"]
  WS8 --> WS7([Return scan results])

Process Scan Results Diagram

flowchart TD
  %% PROCESS SCAN RESULTS
  START([Start]) --> P1["Set surrounding shard ranges of incoming notes 
-> 'found note'"]
  P1 --> P2["Discover unified addresses in use on-chain"]
  P2 --> P3["Add wallet transactions (including notes and coins)"]
  P3 --> P4["Add nullifiers to wallet's nullifier map"]
  P4 --> P5["Add outpoints to wallet's outpoint map"]
  P5 --> P6["Update wallet's shard trees"]
  P6 --> P7["Check coins output ids vs outpoint map; if spent: mark coin and set surrounding narrow block range -> 'found note'"]
  P7 --> P8["Check notes derived nullifiers vs nullifier map; if spent: mark note and set surrounding shard range -> 'found note'"]
  P8 --> P9["Add wallet blocks; retain only: range bounds, blocks with relevant txs, and within max verification window of highest scanned block"]
  P9 --> P10["Mark scan range containing this batch -> 'scanned'"]
  P10 --> P11["Merge adjacent 'scanned' ranges"]
  P11 --> P12["Clean wallet data no longer relevant"]
  P12 --> END([End])

Re-exports

pub use sync::add_scan_targets;

pub use sync::reset_spends;

pub use sync::scan_pending_transaction;

pub use sync::sync;

pub use sync::sync_status;

Modules

config

Sync configuration.

error

Pepper sync error module

keys

Copied and modified from LRZ due to thread safety limitations and missing OVK

sync

Entrypoint for sync engine

wallet

Module for wallet structs and types generated by the sync engine from block chain data or to track the wallet’s sync status. The structs will be (or be transposed into) the fundamental wallet components for the wallet interfacing with this sync engine.