picca
picca (acronym for a Parallel Implementation of Common Checksum Algorithms) is a relatively small Rust program and wrapper library intented to speed up hashing files by reading multiple files at once.
Awesome Features!
What CAN picca do?
picca can:
- Hash any file using any one of 50 algorithms! Current options can be found in the source code here.
- Read checksum files in BSD and GNU Tagged formats, and verify the files listed.
- Hash each file up to the number of cores/threads your CPU has, or a custom amount.
- Be a potential replacement for the UNIX
cksumutility and the individual algorithmic checksum utilities (sha256sum,b2sum, etc.) - Ignore common errors, such as missing files, and produce quiet output.
What is picca NOT?
- A custom implementation of each hashing algorithm. The hash algorithims are sourced from RustCrypto's hashes libraries.
How do I use such an awesome program?
picca operates mostly (if not, the same) as the UNIX cksum program. Here is a non-exhaustive list of examples:
- On it's own, picca will hash whatever comes in from standard input using the SHA256 algorithm.
| - Using the
-aflag, you can change the algorithm (only with the single binary, see the section Installing the binaries for what that means).| - Instead of reading from standard input, you can specify file(s) as arguments to hash the contents of them instead.
- By default, picca will read and hash up to n files simultaneously in separate threads, where n is the amount of cores your CPU has. You can adjust the amount of threads using the
-tflag.# use only 4 threads- If less than n files are specified, picca will ignore the thread count you specify and spawn as many threads as there are files.
- You can also use picca to verify the hash of files using the
-cflag.- picca supports BSD-style and GNU-style checksum formats.
- For BSD-style checksums, if the hashes in your file are different than SHA256, you must specify the algorithm with
-a.
picca is also available for use in your Rust program as a crate! See the examples to learn how to use it.
Project hosting distribution
The picca project is hosted on two separate Forgejo instances, Codeberg and my own (hereafter refered to as "my forge" or "forge"), mainly because I enjoy self-hosting, but also as an experiment to see if projects like these can survive being hosted like this. Here is what lives on each:
Both Codeberg and My Forge
- The raw Git repository
- All changes that occur on either platform are synced to the other immediately
- Forgejo releases
Codeberg only
- Issue management and pull requests (to welcome community contributions)
- Project management (see the picca project)
My Forge only
Installing the Binaries
picca offers two categories of installs, refered to hereafter as "single" and "standalone" respectively:
- The singular
piccabinary that can hash with any of the supported algorithms (defaults to SHA256) - Individual binaries named after the only algorithm they can hash with, such as
md5sum,blake3sum, etc.
The single and standalone binaries can be installed in several different ways depending on what you prefer and/or what your operating system offers.
- Platform Agnostic
- Cargo (building and installing from source)
- Docker/Podman
- Linux
- Alpine 3.x
- Arch
- Debian 12, 13/Ubuntu 22, 24
- Enterprise Linux (Rocky/Red Hat) 8, 9, 10
- Gentoo
- Ubuntu 22, 24
- Individual Binaries
- macOS Binaries
- Windows Binaries
Cargo
This is essentially just compiling it yourself and putting it in the same $PATH location as Cargo so it can be invoked.
# clone the repository
# you can use forge.steck.dev instead of codeberg.org if you wish.
&&
# to install the single binary:
# to install all the standalone binaries
for; do
done
# to install only a certain standalone binary, replace <NAME> with the name of the binary:
Docker/Podman
Not really "installing" per se, but allows you to use it in an isolated environment.
A OCI image is available for use in an OCI-compatible container environment like Docker or Podman. The following examples use Docker.
# Run in a bare minimum container. -i is required to read from stdin, but I like to add --rm so I can skip the step of removing the container after.
# Mount file and scan it. -i is no longer needed since we won't read from stdin, but all arguments after the image name are passed to picca.
Adding to your image
FROM forge.steck.dev/bryson/picca:latest AS picca
COPY --from=picca /usr/local/bin/picca /usr/local/bin
Alpine Linux
Follow these instructions to set up the Alpine repository on your system.
Arch Linux
Follow these instructions to set up the Arch repository on your system.
Debian/Ubuntu Linux
Follow these instructions to set up the Apt repository on your system.
Enterprise Linux (Rocky, Red Hat)
Follow these instructions to set up the DNF repository on your system.
Gentoo Linux
The Gentoo package is hosted in a Git-based overlay on my forge.
# Use eselect to add the overlay
# Use emaint to sync the overlay
# Finally, emerge picca
Individual Binaries
If your distro doesn't appear above or has different requirements (architecture, libc, etc.), you can use one of the several release binaries and install them in a place you can find and use it.
# assuming that ~/bin is in $PATH:
|
macOS
macOS binaries aren't packaged in an installer format, you must install them in a place you can find it.
# assuming that ~/bin is in $PATH:
|
Windows
Windows binaries aren't packaged in an installer format, you must install them in a place you can find it.
# assuming that ~\bin is in $env:PATH:
Set-Location ~\Downloads
Invoke-WebRequest "https://codeberg.org/bryson/picca/releases/download/v0.15.2/picca-v0.15.2-x86_64-pc-windows-msvc.zip" -OutFile "picca-v0.15.2-x86_64-pc-windows-msvc.zip"
Expand-Archive ".\picca-v0.15.2-x86_64-pc-windows-msvc.zip"
Move-Item ".\picca-v0.15.2-x86_64-pc-windows-msvc\bin\picca.exe" ~\bin
Installing the Library
picca is also available for use as a crate in your Rust program. Simply add it to your dependencies using Cargo:
# you can also use the git repo if you wish.
Benchmarks
Some benchmarks are available for your use to see how picca stands up against your coreutils binaries. Simply run cargo bench to run the following benchmarks:
blake3- This will test an algorithm that is already multithreaded against picca which is adding the ability to read multiple files at once (UNIX only,b3must be available on your system).sha256- This tests a conventional, single threaded algorithm to allow picca to use more processing power that it leaves behind for reading the files.
Each benchmark creates 100, 300, and 500 random, 4MB files and runs the local binary and picca against those files to see if they provide any speed benefit. The more the files, the easier to see if it provides faster speeds.