Expand description
§md5-simd
MD5 checksums for object-storage workloads: streaming uploads, snapshot ETags, RustCrypto integration, and batches of independent messages.
The default build uses optimized x86_64 and little-endian AArch64 single-stream compressors with SIMD batch kernels. Other targets retain a portable Rust implementation. Single-stream hashing and batch hashing use separate dispatch paths.
MD5 is cryptographically broken. Use it only for legacy interoperability and non-adversarial checksums, never for passwords, signatures, or authenticity.
§Getting started
[dependencies]
md5-simd = "0.2"The library import is md5_simd. Default features are std, opt, digest, and
simd. These features can also be selected individually.
use md5_simd::{Md5, digest, hex_encode};
assert_eq!(hex_encode(&digest(b"abc")), "900150983cd24fb0d6963f7d28e17f72");
let mut hasher = Md5::new();
hasher.update(b"a");
assert_eq!(hasher.finalize_snapshot(), digest(b"a"));
hasher.update(b"bc");
assert_eq!(hasher.finalize(), digest(b"abc"));§Streaming and snapshots
Md5::finalize() consumes the hasher. finalize_snapshot() and digest_so_far()
return the current digest without consuming or changing it. clone().finalize()
remains supported. finalize_reset() returns the digest and resets the hasher
for another message.
With std, the hasher implements std::io::Write:
use md5_simd::{Md5, digest, digest_reader};
use std::io::Write;
let mut hasher = Md5::new();
hasher.write_all(b"hello ")?;
let prefix_hex = hasher.finalize_hex(); // Non-consuming lowercase hex.
assert_eq!(prefix_hex, md5_simd::digest_hex(b"hello "));
hasher.write_all(b"world")?;
assert_eq!(hasher.finalize(), digest(b"hello world"));
assert_eq!(digest_reader(&b"abc"[..])?, digest(b"abc"));Reader helpers hash until EOF, retry interrupted reads, and propagate other I/O
errors. etag_hex_streaming(reader, chunk) clamps its buffer size to 4 KiB–1 MiB.
| API | Output | Allocation |
|---|---|---|
digest, native byte finalizers | [u8; 16] | None |
hex_encode_digest | [u8; 32] | None |
hex_encode_into | Caller-provided byte slice | None |
hex_encode, digest_hex, etag_hex | Lowercase String | One output allocation |
Md5::finalize_hex (std) | Snapshot String | One output allocation |
hex_encode encodes the entire slice. hex_encode_into requires at least twice
as many output bytes as input bytes and preserves unused output bytes.
An MD5 hex string has the shape of an ordinary single-part ETag. This crate does not implement multipart ETag composition, HTTP quoting, Content-MD5 Base64 encoding, or storage-specific encryption semantics.
§RustCrypto Digest
The digest feature provides DigestMd5 and re-exports the Digest 0.11 traits.
Use DigestMd5 for generic Digest consumers and Md5 for the native snapshot API.
Traits from Digest 0.10 are a different API and are not interchangeable.
use md5_simd::{Digest, DigestMd5, digest};
let mut hasher = DigestMd5::new();
Digest::update(&mut hasher, b"abc");
assert_eq!(hasher.finalize_reset().as_slice(), &digest(b"abc"));
Digest::update(&mut hasher, b"next message");
assert_eq!(hasher.finalize().as_slice(), &digest(b"next message"));Both interfaces share the selected compressor and padding builder. Digest owns its partial-block buffer; complete blocks are compressed directly.
§Batch hashing
use md5_simd::{Md5Engine, digest};
let messages = [[0x5a; 1024]; 8];
let inputs: Vec<&[u8]> = messages.iter().map(|m| m.as_slice()).collect();
let mut outputs = [[0u8; 16]; 8];
Md5Engine::new().hash_many(&inputs, &mut outputs);
assert!(outputs.iter().all(|out| *out == digest(&messages[0])));hash_many preserves input order and processes contiguous equal-length runs.
Runs with at least four messages of at least 32 bytes can use SIMD; small runs,
short messages, and unsupported targets use the active single-stream backend.
Inputs need no special alignment. The caller supplies output storage, and the
byte batch API performs no heap allocation or thread spawning.
When an object scheduler has non-adjacent equal-size messages, the std
feature also provides Md5Engine::hash_many_grouped. It sorts internal indices
by message length, runs the same SIMD dispatcher, and scatters digests back to
the original order. This method allocates temporary indices and digest storage;
use it only when the expected SIMD work outweighs that scheduling cost.
| Target | Batch selection with simd |
|---|---|
x86_64 with std | Runtime detection: AVX2 (8 lanes), AVX-512F + AVX2 (16 lanes), or scalar |
| Little-endian aarch64 | NEON (4 lanes) or paired NEON (8 lanes) |
Other targets (including big-endian); x86_64 without std | Scalar fallback |
Any target with force-portable | Textbook scalar fallback |
On an AVX-512 host, runs of up to eight messages prefer AVX2. Larger runs can
process up to four vector groups per kernel call. runtime_lanes() and
simd_name() report available capability; a particular call may use fewer lanes
or fall back. SIMD accelerates independent messages, not one message’s sequential
MD5 chain.
Md5State is a copyable incremental state. Md5Engine::update_many and
finalize_many loop over independent scalar states; they are not incremental
SIMD APIs. Md5State::finalize() is a non-consuming snapshot.
§Measured AArch64 batch results
On Apple Silicon, same-host ABBA measurements show the intended batch shape:
| Workload | NEON result vs sequential md-5 |
|---|---|
| 8 independent × 1 MiB | 3.10× |
| 16 independent × 1 MiB | 4.22× |
| 7 independent × 1 MiB tail batch | 2.67× |
| 9 independent × 1 MiB with one scalar tail | 1.73× |
| grouped mixed lengths | 1.60× |
These are independent-message throughput results. A single MD5 message remains
serial and is measured separately; current AArch64 single-message oneshot and
streaming are approximately at RustCrypto md-5 parity. For non-adjacent object
sizes, hash_many_grouped can recover equal-length runs when its temporary sort
and output scatter cost is justified.
§Features and portability
| Feature | Default | Effect |
|---|---|---|
std | Yes | I/O helpers, native snapshot hex, batch hex, x86 runtime detection |
opt | Yes | x86_64 single-stream assembly; little-endian aarch64 single-aarch kernel |
digest | Yes | Optional RustCrypto Digest 0.11 dependency |
simd | Yes | SIMD batches; single-stream dispatch is unchanged |
force-portable | No | Overrides the selected single-stream and batch acceleration |
pair-interleave | No | Experimental scalar pairs when opt and force-portable are off |
zeroize | No | Best-effort clearing of native Md5 on drop |
# Portable scalar with I/O support:
md5-simd = { path = "../md5-simd", default-features = false, features = ["std"] }
# no_std library (uses alloc for String-returning helpers):
# md5-simd = { path = "../md5-simd", default-features = false }Byte hashing APIs do not allocate. no_std builds use alloc; applications using
allocating helpers need an allocator. zeroize does not cover Md5State copies,
Digest buffers, or all compiler-made copies. Call reset() before reusing a
manually cleared Md5.
The project uses edition 2024. No minimum supported Rust version is declared; the release checks use Rust 1.98.1. Older compiler versions are not guaranteed.
§Performance
Performance depends on CPU, message size, batch shape, features, and compiler. Performance notes record the measured aarch64 refactor comparison, validation limits, and reproduction commands. No result is a universal throughput guarantee.
Applications should validate their own checksum formatting, I/O error handling, and workload performance when adopting a new implementation.
§Development
cargo test --locked
cargo test --locked --features force-portable
cargo test --locked --no-default-features --features std
cargo test --locked --no-default-features
cargo fmt --all -- --check
cargo clippy --all-targets -- -D warnings
cargo clippy --all-targets --all-features -- -D warnings
cargo doc --no-deps
cargo package --locked
cargo run --release --example checksum_profileCI covers tests, feature combinations, documentation, and package verification. The Audit workflow checks locked dependencies on changes and weekly. Version tags run the release checks; registry upload requires an explicit manual dispatch.
Run both clippy profiles: --all-features enables force-portable and excludes
SIMD adapters. README examples run as rustdoc tests.
Architecture describes state ownership, dispatch, padding, and unsafe boundaries. See the changelog for release notes and the release guide for maintainer checks. Repository tooling and local files are excluded from the published package by an explicit manifest allowlist.
§License
Apache-2.0. The vendored scalar assembly and its glue retain their BSD-2-Clause and assembly provenance notices. See LICENSE and NOTICE.
§Acknowledgements
Thank you to fast-md5 and md5-many for their open-source work on MD5 performance and multi-buffer design. Third-party code attribution is retained in NOTICE and the relevant source headers.
Re-exports§
pub use digest;
Modules§
- hazmat
- Hazmat: raw compress entry points (custom framing). Hazmat: raw compress entry points.
Structs§
- Digest
Md5 - Streaming MD5 hasher compatible with RustCrypto
Digest(0.11). - Md5
- Streaming MD5 hasher.
- Md5Core
- Core MD5 state for the Digest trait wrapper.
- Md5Engine
- Batch MD5 engine. Construct once and reuse.
- Md5State
- Incremental state for one MD5 message in a multi-stream workload.
Constants§
- BLOCK_
SIZE - MD5 compresses 64-byte blocks.
- DIGEST_
LEN - MD5 produces a 16-byte digest.
- K
- Per-round additive constants
T[i](RFC 1321 §3.4). - S
- Left-rotate amounts for each of the 64 steps (RFC 1321 §3.4).
- STATE_
INIT - Standard MD5 initialization vector (RFC 1321 §3.3).
Traits§
- Digest
- Convenience wrapper trait covering functionality of cryptographic hash functions with fixed output size.
Functions§
- available_
backends - Single-stream backends compiled into this build.
- backend_
name - Active single-stream backend name.
- digest
- One-shot hashing (active backend).
- digest_
hex - Hash a message and return its 32-character lowercase hexadecimal digest.
- digest_
opt_ scalar - In-tree production-kernel one-shot (ignores
opt/force-portable). - digest_
portable - Textbook-oracle one-shot (CI / differential). Ignores accelerated backends.
- digest_
reader - Hash a reader until EOF using a 64 KiB stack buffer.
- etag_
hex - Alias for
digest_hexfor applications using MD5-shaped ETags. - etag_
hex_ streaming - Hash a reader to lowercase hexadecimal until EOF.
- hash_
pair - Software multi-buffer: pair-interleaved hashing for independent messages. Software multi-buffer: pair-interleaved hashing for independent messages. Software multi-buffer: pair-interleaved hashing for independent messages. Hash two messages; equal-length pairs use interleaved compress when active.
- hex_
encode - Encode every input byte as two lowercase hexadecimal characters.
- hex_
encode_ digest - Lowercase hex of a 16-byte digest into a fixed 32-byte array (no alloc).
- hex_
encode_ into - Write lowercase hex into
out; returns bytes written (2 * bytes.len()). - is_
portable_ only - True when the selected compressor is the textbook oracle.
- md5_
many - Hash many independent messages with the process-default engine.
- pair_
path_ active - Software multi-buffer: pair-interleaved hashing for independent messages.
Software multi-buffer: pair-interleaved hashing for independent messages.
Software multi-buffer: pair-interleaved hashing for independent messages.
Whether
hash_manyshould use the software pair path. - runtime_
lanes - Best usable batch width on this process (cached CPUID on x86_64).
- simd_
active - Whether SIMD multi-buffer can run on this process.
- simd_
lanes - Compile-time max SIMD width for this target (runtime CPUID may be lower).
- simd_
name - Best SIMD kernel label for reports.