snap-fetch

A high-performance file downloader designed to fetch Solana snapshot files listed in the snapshot manifest at https://data.pipedev.network/snapshot-manifest.json.

Solana Validator Usage

Solana validators use these snapshot files to initialize or catch up to the current blockchain state:

Where to save the files:

Snapshots should be downloaded to your validator's snapshots directory:

[ledger-path]/snapshots/

Default ledger paths:

• Linux: ~/.local/share/solana/validator/ledger
• macOS: ~/Library/Application Support/solana/validator/ledger
• Custom: Many validators specify a custom ledger path with --ledger

How to use with a validator:

1. Download snapshots to your ledger's snapshots directory:

   snap-fetch -o /path/to/ledger/snapshots --skip-existing
   

2. Start your validator with the snapshot-fetch flag:

   solana-validator --ledger /path/to/ledger --snapshot-fetch ...other-options
   

The Solana validator software will automatically:

• Detect the snapshots in the directory
• Load the latest full snapshot
• Apply any newer incremental snapshots
• No manual unzipping or processing is required

Features

• Optimized for high-bandwidth connections (10-20 Gbps)
• Automatically probes the server and selects an optimal download strategy (no performance flags/knobs)
• Caches per-host tuning automatically for faster subsequent runs ($XDG_CACHE_HOME/snap-fetch/strategy-cache.json or ~/.cache/snap-fetch/strategy-cache.json)
• Uses byte range requests (when supported) to download files in parallel chunks
• Downloads multiple snapshot files concurrently (full + incrementals)
• Single-line progress bar (aggregated bytes + speed/ETA + file count)
• Automatic retries with exponential backoff
• Pre-sizes output files (best-effort fallocate on Linux for smaller files)
• Skip already downloaded files
• Downloads to *.part and renames on success (safe to re-run)
• Ctrl-C exits immediately (SIGINT -> exit code 130)
• URL substitution for downloading from alternative servers
• Automatic fallback for servers that don't implement Range requests
• Connection pooling and HTTP/2 optimizations
• Memory-efficient large file handling
• Buffered I/O for maximum disk performance
• Linux-specific optimizations (pwrite, fallocate when beneficial)
• Optional Linux io_uring-backed writes (cargo build --release --features io_uring)

Usage

snap-fetch [OPTIONS]

OPTIONS:
    -o, --output-dir <OUTPUT_DIR>       Output directory for downloaded files [default: downloads]
    --status-url <STATUS_URL>           Custom URL for the manifest JSON
    --skip-existing                     Skip files that already exist in the output directory
    --base-url <BASE_URL>               Substitute base URL for downloads (e.g., http://207.121.20.172:8080)
    --latest-only                       Only download the latest snapshot and newer incremental snapshots [default: true]
    --save-hash-name                    Save files using the hash from the URL instead of the original filename
    -q, --quiet                         Reduce output to warnings and errors only
    -v, --verbose                       Show verbose output including debug information
    --no-progress                       Disable progress bars
    -h, --help                          Print help
    -V, --version                       Print version

Examples

Download snapshots with fully automatic tuning:

snap-fetch

Save files to a specific directory:

snap-fetch -o /path/to/downloads

Skip files that already exist (with size validation):

snap-fetch --skip-existing

Note: --skip-existing now validates file sizes before skipping. It will re-download files that:

• Are partially downloaded (size mismatch)
• Are empty (0 bytes)
• Have different sizes than the server reports

Partial/in-progress downloads are written to *.part and are never treated as complete.

This ensures you never skip corrupted or incomplete downloads.

Use an alternative server to download files:

snap-fetch --base-url "http://207.121.20.172:8080"

Download all files, not just the latest snapshot:

snap-fetch --latest-only=false

Run with minimal output (quiet mode):

snap-fetch --quiet

Run in silent mode for automation:

snap-fetch --quiet --no-progress

Building

cargo build --release

The optimized binary will be in target/release/snap-fetch

Linux io_uring build (optional)

cargo build --release --features io_uring

Memory and Performance Optimizations

The downloader streams data and uses an auto-tuned range strategy when supported:

• Detects Range support and probes throughput to select parallelism + range size automatically
• Keeps disk writes efficient via pre-sizing and buffered I/O

Hardware Requirements

For optimal performance, the following hardware specifications are recommended:

Minimum Requirements:

• CPU: 4+ cores for parallel downloading operations
• RAM:
  • General use: At least 16GB for small to medium files
  • Validators: Minimum 128GB RAM required (more is better)
• Storage: SSD with at least 200GB free space for snapshots
• Network: 1 Gbps connection or better

Recommended for Validators:

• CPU: 8+ cores, modern processor (Intel Xeon, AMD Ryzen/EPYC)
• RAM:
  • Minimum 128GB RAM (required for handling large snapshot files)
  • Recommended: 256GB+ RAM for optimal performance
• Storage: NVMe SSD with 1TB+ free space (higher endurance drives recommended)
• Network: 10+ Gbps connection for fast snapshot downloads

Network Considerations:

• Fast, stable internet connection is essential
• Network bandwidth is typically the bottleneck rather than CPU
• Consider using a datacenter with good connectivity to Solana infrastructure

Storage Considerations:

• Write speed directly impacts download performance
• RAID configurations can improve throughput
• For validators, consider separate drives for ledger and snapshots

System Requirements

• Rust 1.70 or later
• Sufficient disk space for downloads
• Internet connection (optimal performance on 10-20 Gbps links)

Development

Running Tests

# Run all tests
cargo test

# Run only unit tests
cargo test --bins

# Run only integration tests  
cargo test --tests

# Run with coverage (requires cargo-tarpaulin)
cargo install cargo-tarpaulin
cargo tarpaulin --verbose --all-features --workspace --timeout 120