cksfv.rs

A 10x faster drop-in reimplementation of cksfv using Rust and the crc32fast crate.

🗺️ Overview

cksfv is a FOSS tool developed by Bryan Call and maintained by Heikki Orsila to validate integrity of files using the CRC32 checksum. It can be used to check a list of files against a .sfv signature list, and to generate new lists.

This repository contains code for an implementation written from scratch in Rust, that provides the same CLI as the original program but much better performance thanks to a CRC32 implementation that takes advantage of modern CPUs.

🛠️ Features

Features from the original binary:

• Exact same Command-Line Interface for scripting compatibility
• Exact same behaviour with respect to argument parsing
• SFV listing generation compatible with the original cksfv
• SFV listing validation with -f, -g or -r
• SFV filtering in validation mode using files given as arguments
• Specific options:
  • -b flag to only print base filenames
  • -c flag to print everything to STDOUT
  • -C flag to change directory when processing files
  • -i flag to ignore case on filenames
  • -L flag to follow symlinks
  • -q flag to only print error messages
  • -s flag to replace backslashes

Additional features:

• Support for mmap syscall to avoid reading the file directly
• Multithreading for several files

⏱️ Benchmarks

Benchmark where conducted using a 1.6GiB file using the cksfv binary distributed with ArchLinux, or this program compiled with either stable or nightly Rust, running hyperfine, on an Intel i7-8550U CPU:

Benchmark #1: cksfv test.mkv
  Time (mean ± σ):      4.376 s ±  0.120 s    [User: 4.117 s, System: 0.251 s]
  Range (min … max):    4.237 s …  4.555 s    10 runs

Benchmark #2: cargo +stable run --release -- test.mkv
  Time (mean ± σ):     387.9 ms ±   9.3 ms    [User: 158.3 ms, System: 224.9 ms]
  Range (min … max):   380.0 ms … 402.3 ms    10 runs

Benchmark #3: cargo +nightly run --release -- test.mkv
  Time (mean ± σ):     387.9 ms ±  11.7 ms    [User: 160.9 ms, System: 226.6 ms]
  Range (min … max):   373.8 ms … 414.9 ms    10 runs

Benchmark #4: cargo +nightly run --all-features --release test.mkv
  Time (mean ± σ):     347.6 ms ±  12.3 ms    [User: 208.8 ms, System: 136.1 ms]
  Range (min … max):   330.7 ms … 368.3 ms    10 runs
 d

Summary
  'cargo +nightly run --all-features --release test.mkv' ran
    1.12 ± 0.05 times faster than 'cargo +stable run --release -- test.mkv'
    1.12 ± 0.05 times faster than 'cargo +nightly run --release -- test.mkv'
   12.59 ± 0.56 times faster than 'cksfv test.mkv'

Using the generated SFV listing to check the same file, we get the following results:

Benchmark #1: cksfv -f test.sfv
  Time (mean ± σ):      4.672 s ±  0.102 s    [User: 4.380 s, System: 0.269 s]
  Range (min … max):    4.526 s …  4.845 s    10 runs

Benchmark #2: cargo +stable run --release -- -f test.sfv
  Time (mean ± σ):     344.8 ms ±  12.0 ms    [User: 145.9 ms, System: 194.1 ms]
  Range (min … max):   325.0 ms … 355.7 ms    10 runs

Benchmark #3: cargo +nightly run --release -- -f test.sfv
  Time (mean ± σ):     356.5 ms ±   7.8 ms    [User: 144.4 ms, System: 210.2 ms]
  Range (min … max):   349.3 ms … 374.3 ms    10 runs

Benchmark #4: cargo +nightly run --all-features --release -- -f test.sfv
  Time (mean ± σ):     300.4 ms ±  12.4 ms    [User: 197.3 ms, System: 99.8 ms]
  Range (min … max):   290.7 ms … 324.6 ms    10 runs

Summary
  'cargo +nightly run --all-features --release -- -f test.sfv' ran
    1.15 ± 0.06 times faster than 'cargo +stable run --release -- -f test.sfv'
    1.19 ± 0.06 times faster than 'cargo +nightly run --release -- -f test.sfv'
   15.55 ± 0.72 times faster than 'cksfv -f test.sfv'

📜 License

This tool is provided under the open-source MIT license.