Skip to main content

Crate par2_rs

Crate par2_rs 

Source
Expand description

General-purpose PAR2 verification and repair engine.

A pure-Rust implementation of PAR2 (Parity Archive Volume Set v2.0): load a set, find out what is damaged, and repair it from the recovery data.

§Verifying a set

A PAR2 set is usually spread across several .par2 files. Packets from all of them aggregate into one Par2FileSet, and verification runs against that.

use par2_rs::{DiskFileAccess, Par2FileSet, Repairability, scan_packets_from_path, verify_all};

let packets = scan_packets_from_path(std::path::Path::new("release.par2"))?
    .into_iter()
    .map(|(packet, _offset)| packet)
    .collect();
let set = Par2FileSet::from_packets(packets)?;

let access = DiskFileAccess::new("/downloads/release".into(), &set);
let result = verify_all(&set, &access);

println!("{} recovery blocks available", result.recovery_blocks_available);
match result.repairable {
    Repairability::NotNeeded => println!("everything verified clean"),
    Repairability::Repairable { blocks_needed, .. } => {
        println!("repairable: {blocks_needed} blocks to rebuild")
    }
    Repairability::Insufficient { deficit, .. } => {
        println!("not enough recovery data: {deficit} blocks short")
    }
    other => println!("{other:?}"),
}

Verification is slice-level, using the CRC32 + MD5 pairs in IFSC packets, so damage is localised to the slices that are actually wrong rather than condemning the whole file. Sets carrying no IFSC data fall back to full-file MD5, and quick_check_16k identifies a candidate file cheaply before either.

§Verifying bytes that are not files

verify_all reads through the FileAccess trait, not the filesystem. DiskFileAccess is the ordinary implementation; supply your own and a set can be verified against bytes still arriving over a network, or assembled from somewhere that has no paths at all. MemoryFileAccess is useful in tests.

§Repair

Par2Repairer drives the whole sequence — scan, verify, solve, repair, then verify again. Repair is placement-aware: files that were renamed or moved are matched by content rather than by name, so a set still repairs after its files have been reorganised.

§Repairing across a whole download

Par2RepairSession is the retained form: one session accumulates evidence — per-slice verdicts, whole-file proofs — while the data is still arriving, so assessment is incremental and repair runs from what is already known instead of a fresh walk. Its sources may be files under a base directory, or bytes served through a FileAccess handle (Par2RepairSessionOptions::with_source_access) for sets that never became files — and where the .par2 volumes themselves never became files either, Par2RepairSessionOptions::from_set takes the parsed set directly. Repair output is always real files either way.

§Damaged PAR2 files

A malformed or truncated packet does not fail the set. The scanner skips forward to the next valid packet, because the recovery data that survived is usually still enough — which is the entire point of parity.

§Feature flags

  • crypto-aws-lc (default): AWS-LC-backed MD5. Needs a C toolchain to build aws-lc-sys, and is the configuration the published performance figures were measured with.

  • crypto-rust: the portable RustCrypto (md-5) MD5 backend, for builds that must not carry a C/assembly dependency and for wasm, where AWS-LC is unavailable. Select it with default-features = false, features = ["crypto-rust"]. Expect slower hashing; nothing else changes.

  • native-crypto: back-compat alias for crypto-aws-lc.

    Exactly one backend is active: on a native target AWS-LC wins whenever crypto-aws-lc is on, and enabling neither backend is a compile error.

  • metal / wgpu: GPU-accelerated repair through reedsolomon_rs, with repair fallback to CPU when no suitable device or driver is present. The metal feature also enables policy-driven creation on native Apple Silicon through CreationBackend. CreationBackend::Auto keeps creation work below 16 GiB (slice size × source-slice count × recovery- slice count) on CPU; on supported native Apple Silicon at or above that threshold it preflights Metal and falls back to CPU when unavailable.

§Benchmarks

Heavy PAR2 repair against par2cmdline-turbo 1.4.0, from the deterministic 43-case rarpar-bench corpus. Each figure is the geometric mean of reference wall time / rarpar wall time over par2-heavy-damage-28 and par2-heavy-damage-250, so 2.0x means half the time.

CPUArchInstruction setpar2 (heavy)
AMD EPYC 9R14 (Zen 4)x86-64GFNI + AVX-5121.8x
Intel Xeon Platinum 8488C (Sapphire Rapids)x86-64GFNI + AVX-5121.7x
Intel Core i5-1240P (Alder Lake)x86-64GFNI + AVX21.9x
AMD Ryzen 5 3600 (Zen 2)x86-64AVX21.5x
Intel Atom C3538 (Denverton)x86-64SSSE3 (no AVX)1.3x
Apple M5 Maxarm64NEON7.1x
Arm Cortex-A72arm64NEON1.2x
Arm Neoverse N1arm64NEON1.4x
Arm Neoverse V2arm64NEON1.5x

The Apple row is the CPU lane, and is measured against upstream’s published macOS arm64 reference binary, which is much slower than the same version’s Linux and Windows builds; that lifts every macOS PAR2 figure.

Per-case charts for every machine, the full methodology, and the versions these numbers were measured with are in rarpar benchmarks.

The format is specified in the Parity Volume Set Specification 2.0.

Re-exports§

pub use checksum::FileHashState;
pub use checksum::SliceChecksumState;
pub use create::BlockSizing;
pub use create::CreationBackend;
pub use create::CreationSource;
pub use create::ForwardKernel;
pub use create::Par2CreateOutcome;
pub use create::Par2CreatePlan;
pub use create::Par2Creator;
pub use create::Par2CreatorOptions;
pub use create::Par2MemoryPlan;
pub use create::RecoveryAmount;
pub use create::RecoveryVolumePlan;
pub use create::VolumeScheme;
pub use disk::DiskFileAccess;
pub use disk::MultiDirectoryFileAccess;
pub use disk::PlacementFileAccess;
pub use error::Par2Error;
pub use error::Result;
pub use evidence::CommittedFileEvidence;
pub use evidence::ContiguousAssemblyProof;
pub use evidence::FileStatFingerprint;
pub use matrix::Matrix;
pub use matrix::build_decode_matrix;
pub use packet::CreatorPacket;
pub use packet::DEFAULT_MAX_EXAMINED_PACKETS;
pub use packet::DEFAULT_MAX_RETAINED_METADATA_BYTES;
pub use packet::DEFAULT_MAX_RETAINED_PACKETS;
pub use packet::FileDescriptionPacket;
pub use packet::IfscPacket;
pub use packet::MAX_RECOVERY_EXPONENT;
pub use packet::MainPacket;
pub use packet::Packet;
pub use packet::PacketHeader;
pub use packet::PacketScanBudget;
pub use packet::PacketScanLimits;
pub use packet::PacketSink;
pub use packet::PacketType;
pub use packet::RECOVERY_EXPONENT_DOMAIN;
pub use packet::RecoverySliceData;
pub use packet::RecoverySlicePacket;
pub use packet::ScannedPacket;
pub use packet::parse_packet;
pub use packet::scan_packets;
pub use packet::scan_packets_bounded;
pub use packet::scan_packets_from_path;
pub use packet::scan_packets_from_path_bounded;
pub use packet::scan_packets_from_path_with_set_ids;
pub use packet::scan_packets_from_path_with_set_ids_limited;
pub use packet::scan_packets_with_limits;
pub use par2_set::FileDescription;
pub use par2_set::MergeResult;
pub use par2_set::Par2Diagnostic;
pub use par2_set::Par2FileSet;
pub use par2_set::Par2ParseResult;
pub use par2_set::RecoverySlice;
pub use path::translate_par2_name_to_local_path;
pub use path::translate_par2_name_to_relative;
pub use placement::PlacementEntry;
pub use placement::PlacementPlan;
pub use placement::apply_placement_plan;
pub use placement::scan_placement;
pub use rename::MatchType;
pub use rename::RenameSuggestion;
pub use rename::SplitFileGroup;
pub use rename::detect_split_files;
pub use rename::identify_par2_files;
pub use rename::scan_for_renames;
pub use repair::NativeRepairSolver;
pub use repair::RepairOptions;
pub use repair::RepairPlan;
pub use repair::RepairProblem;
pub use repair::RepairSolver;
pub use repair::SolverError;
pub use repair::execute_repair;
pub use repair::execute_repair_with_options;
pub use repair::execute_repair_with_solver;
pub use repair::plan_repair;
pub use repair::plan_repair_with_memory_limit;
pub use repair::prepare_recovery_buffers;
pub use repair::reconstruct_and_write;
pub use repair::xor_out_slice;
pub use repair_session::DEFAULT_RETAINED_STATE_LIMIT;
pub use repair_session::Par2RepairSession;
pub use repair_session::Par2RepairSessionDiagnostics;
pub use repair_session::Par2RepairSessionOptions;
pub use repair_session::Par2SessionError;
pub use repair_transform::TransformArm;
pub use repair_transform::TransformArmStats;
pub use repair_transform::set_transform_arm_override;
pub use repair_transform::transform_arm_override;
pub use repair_transform::transform_arm_stats;
pub use repairer::BlockLocation;
pub use repairer::BlockLocationKind;
pub use repairer::CarryDiagnostics;
pub use repairer::CarryRetryReason;
pub use repairer::ExternalCarryError;
pub use repairer::PacketDiagnostics;
pub use repairer::PacketInventory;
pub use repairer::Par2RepairOutcome;
pub use repairer::Par2RepairStatus;
pub use repairer::Par2Repairer;
pub use repairer::Par2RepairerOptions;
pub use repairer::ScanCarry;
pub use repairer::ScanDiagnostics;
pub use repairer::SourceBlock;
pub use repairer::SourceFileEntry;
pub use repairer::SourceLocation;
pub use session::FeedDisposition;
pub use session::FeedOutcome;
pub use session::InStreamCrc32Proof;
pub use session::InStreamCrc32ProofError;
pub use session::SettleRead;
pub use session::SliceEvidence;
pub use session::SliceEvidenceStrength;
pub use session::VerificationMemoryBudget;
pub use session::VerificationSession;
pub use session::VerificationSessionOptions;
pub use types::CancellationToken;
pub use types::ProgressCallback;
pub use types::ProgressPhase;
pub use types::ProgressStage;
pub use types::ProgressUpdate;
pub use types::FileId;
pub use types::RecoveryExponent;
pub use types::RecoverySetId;
pub use types::SliceChecksum;
pub use types::SliceIndex;
pub use verify::FileAccess;
pub use verify::FileStatus;
pub use verify::FileVerification;
pub use verify::MemoryFileAccess;
pub use verify::Repairability;
pub use verify::VerificationResult;
pub use verify::VerifyOptions;
pub use verify::quick_check_16k;
pub use verify::verify_all;
pub use verify::verify_all_with_options;
pub use verify::verify_full_hash;
pub use verify::verify_selected_file_ids;
pub use verify::verify_selected_file_ids_with_options;
pub use verify::verify_slices;
pub use verify::verify_slices_from_crcs;

Modules§

checksum
create
Validated PAR2 creation with deterministic packet allocation and transactional outputs. Output transactions detect ordinary replacement races, but assume no other process with equivalent filesystem permissions mutates their staging or backup paths.
disk
Filesystem-backed implementation of FileAccess.
error
evidence
gf
GF(2^16) field arithmetic for PAR2 Reed-Solomon coding.
gf_pmul
Element-wise GF(2^16) multiply: dst[i] = a[i] * b[i].
gf_simd
SIMD-accelerated GF(2^16) region operations.
matrix
Matrix operations over GF(2^16) for PAR2 Reed-Solomon repair.
matrix_tiled
Rank-k tiled Gauss-Jordan inversion for PAR2 repair-matrix solves.
md5_simd
Multi-buffer MD5: compute several independent MD5 digests at once by putting one message per SIMD lane.
packet
par2_set
path
PAR2 filename translation.
placement
Content-placement scan: match on-disk files to PAR2 file descriptions by hash.
rename
Obfuscated filename recovery and split-file detection.
repair
PAR2 repair orchestration using Reed-Solomon decoding over GF(2^16).
repair_session
Retained PAR2 repair orchestration.
repair_transform
The transform arm of PAR2 repair: syndromes by DFT, solve by the m×m inverse.
repairer
High-level PAR2 verifier/repairer.
session
Streaming verification session for incremental PAR2 verification during download.
types
verify

Structs§

CacheEvictionDeferral
RAII guard deferring page-cache eviction until the outermost scope drops.
FactorDst
A (factor, destination) pair for multi-region multiply-accumulate.

Functions§

gf_add
Addition in GF(2^16) is XOR.
gf_inv
Multiplicative inverse in GF(2^16).
gf_mul
Multiplication in GF(2^16) via log/antilog tables.
gf_pow
Exponentiation in GF(2^16): base^exp.
input_slice_constants
Compute the PAR2 input slice constant assignment sequence.
mul_acc_multi_region
Multiply each u16 word in src by multiple factors and XOR-accumulate into corresponding destination buffers.
mul_acc_region
Multiply each u16 word in src by factor in GF(2^16) and XOR-accumulate into dst.