Skip to main content

Crate md5_simd

Crate md5_simd 

Source
Expand description

§md5-simd

CI Audit Crates Documentation Dependency status Crates.io Total Downloads Crates.io License

MD5 checksums for object-storage workloads: streaming uploads, snapshot ETags, RustCrypto integration, and batches of independent messages.

The default build uses optimized x86_64 and little-endian AArch64 single-stream compressors with SIMD batch kernels. Other targets retain a portable Rust implementation. Single-stream hashing and batch hashing use separate dispatch paths.

MD5 is cryptographically broken. Use it only for legacy interoperability and non-adversarial checksums, never for passwords, signatures, or authenticity.

§Getting started

[dependencies]
md5-simd = "0.2"

The library import is md5_simd. Default features are std, opt, digest, and simd. These features can also be selected individually.

use md5_simd::{Md5, digest, hex_encode};

assert_eq!(hex_encode(&digest(b"abc")), "900150983cd24fb0d6963f7d28e17f72");
let mut hasher = Md5::new();
hasher.update(b"a");
assert_eq!(hasher.finalize_snapshot(), digest(b"a"));
hasher.update(b"bc");
assert_eq!(hasher.finalize(), digest(b"abc"));

§Streaming and snapshots

Md5::finalize() consumes the hasher. finalize_snapshot() and digest_so_far() return the current digest without consuming or changing it. clone().finalize() remains supported. finalize_reset() returns the digest and resets the hasher for another message.

With std, the hasher implements std::io::Write:

use md5_simd::{Md5, digest, digest_reader};
use std::io::Write;

let mut hasher = Md5::new();
hasher.write_all(b"hello ")?;
let prefix_hex = hasher.finalize_hex(); // Non-consuming lowercase hex.
assert_eq!(prefix_hex, md5_simd::digest_hex(b"hello "));
hasher.write_all(b"world")?;
assert_eq!(hasher.finalize(), digest(b"hello world"));
assert_eq!(digest_reader(&b"abc"[..])?, digest(b"abc"));

Reader helpers hash until EOF, retry interrupted reads, and propagate other I/O errors. etag_hex_streaming(reader, chunk) clamps its buffer size to 4 KiB–1 MiB.

APIOutputAllocation
digest, native byte finalizers[u8; 16]None
hex_encode_digest[u8; 32]None
hex_encode_intoCaller-provided byte sliceNone
hex_encode, digest_hex, etag_hexLowercase StringOne output allocation
Md5::finalize_hex (std)Snapshot StringOne output allocation

hex_encode encodes the entire slice. hex_encode_into requires at least twice as many output bytes as input bytes and preserves unused output bytes.

An MD5 hex string has the shape of an ordinary single-part ETag. This crate does not implement multipart ETag composition, HTTP quoting, Content-MD5 Base64 encoding, or storage-specific encryption semantics.

§RustCrypto Digest

The digest feature provides DigestMd5 and re-exports the Digest 0.11 traits. Use DigestMd5 for generic Digest consumers and Md5 for the native snapshot API. Traits from Digest 0.10 are a different API and are not interchangeable.

use md5_simd::{Digest, DigestMd5, digest};

let mut hasher = DigestMd5::new();
Digest::update(&mut hasher, b"abc");
assert_eq!(hasher.finalize_reset().as_slice(), &digest(b"abc"));
Digest::update(&mut hasher, b"next message");
assert_eq!(hasher.finalize().as_slice(), &digest(b"next message"));

Both interfaces share the selected compressor and padding builder. Digest owns its partial-block buffer; complete blocks are compressed directly.

§Batch hashing

use md5_simd::{Md5Engine, digest};

let messages = [[0x5a; 1024]; 8];
let inputs: Vec<&[u8]> = messages.iter().map(|m| m.as_slice()).collect();
let mut outputs = [[0u8; 16]; 8];
Md5Engine::new().hash_many(&inputs, &mut outputs);
assert!(outputs.iter().all(|out| *out == digest(&messages[0])));

hash_many preserves input order and processes contiguous equal-length runs. Runs with at least four messages of at least 32 bytes can use SIMD; small runs, short messages, and unsupported targets use the active single-stream backend. Inputs need no special alignment. The caller supplies output storage, and the byte batch API performs no heap allocation or thread spawning.

When an object scheduler has non-adjacent equal-size messages, the std feature also provides Md5Engine::hash_many_grouped. It sorts internal indices by message length, runs the same SIMD dispatcher, and scatters digests back to the original order. This method allocates temporary indices and digest storage; use it only when the expected SIMD work outweighs that scheduling cost.

TargetBatch selection with simd
x86_64 with stdRuntime detection: AVX2 (8 lanes), AVX-512F + AVX2 (16 lanes), or scalar
Little-endian aarch64NEON (4 lanes) or paired NEON (8 lanes)
Other targets (including big-endian); x86_64 without stdScalar fallback
Any target with force-portableTextbook scalar fallback

On an AVX-512 host, runs of up to eight messages prefer AVX2. Larger runs can process up to four vector groups per kernel call. runtime_lanes() and simd_name() report available capability; a particular call may use fewer lanes or fall back. SIMD accelerates independent messages, not one message’s sequential MD5 chain.

Md5State is a copyable incremental state. Md5Engine::update_many and finalize_many loop over independent scalar states; they are not incremental SIMD APIs. Md5State::finalize() is a non-consuming snapshot.

§Measured AArch64 batch results

On Apple Silicon, same-host ABBA measurements show the intended batch shape:

WorkloadNEON result vs sequential md-5
8 independent × 1 MiB3.10×
16 independent × 1 MiB4.22×
7 independent × 1 MiB tail batch2.67×
9 independent × 1 MiB with one scalar tail1.73×
grouped mixed lengths1.60×

These are independent-message throughput results. A single MD5 message remains serial and is measured separately; current AArch64 single-message oneshot and streaming are approximately at RustCrypto md-5 parity. For non-adjacent object sizes, hash_many_grouped can recover equal-length runs when its temporary sort and output scatter cost is justified.

§Features and portability

FeatureDefaultEffect
stdYesI/O helpers, native snapshot hex, batch hex, x86 runtime detection
optYesx86_64 single-stream assembly; little-endian aarch64 single-aarch kernel
digestYesOptional RustCrypto Digest 0.11 dependency
simdYesSIMD batches; single-stream dispatch is unchanged
force-portableNoOverrides the selected single-stream and batch acceleration
pair-interleaveNoExperimental scalar pairs when opt and force-portable are off
zeroizeNoBest-effort clearing of native Md5 on drop
# Portable scalar with I/O support:
md5-simd = { path = "../md5-simd", default-features = false, features = ["std"] }

# no_std library (uses alloc for String-returning helpers):
# md5-simd = { path = "../md5-simd", default-features = false }

Byte hashing APIs do not allocate. no_std builds use alloc; applications using allocating helpers need an allocator. zeroize does not cover Md5State copies, Digest buffers, or all compiler-made copies. Call reset() before reusing a manually cleared Md5.

The project uses edition 2024. No minimum supported Rust version is declared; the release checks use Rust 1.98.1. Older compiler versions are not guaranteed.

§Performance

Performance depends on CPU, message size, batch shape, features, and compiler. Performance notes record the measured aarch64 refactor comparison, validation limits, and reproduction commands. No result is a universal throughput guarantee.

Applications should validate their own checksum formatting, I/O error handling, and workload performance when adopting a new implementation.

§Development

cargo test --locked
cargo test --locked --features force-portable
cargo test --locked --no-default-features --features std
cargo test --locked --no-default-features
cargo fmt --all -- --check
cargo clippy --all-targets -- -D warnings
cargo clippy --all-targets --all-features -- -D warnings
cargo doc --no-deps
cargo package --locked
cargo run --release --example checksum_profile

CI covers tests, feature combinations, documentation, and package verification. The Audit workflow checks locked dependencies on changes and weekly. Version tags run the release checks; registry upload requires an explicit manual dispatch.

Run both clippy profiles: --all-features enables force-portable and excludes SIMD adapters. README examples run as rustdoc tests.

Architecture describes state ownership, dispatch, padding, and unsafe boundaries. See the changelog for release notes and the release guide for maintainer checks. Repository tooling and local files are excluded from the published package by an explicit manifest allowlist.

§License

Apache-2.0. The vendored scalar assembly and its glue retain their BSD-2-Clause and assembly provenance notices. See LICENSE and NOTICE.

§Acknowledgements

Thank you to fast-md5 and md5-many for their open-source work on MD5 performance and multi-buffer design. Third-party code attribution is retained in NOTICE and the relevant source headers.

Re-exports§

pub use digest;

Modules§

hazmat
Hazmat: raw compress entry points (custom framing). Hazmat: raw compress entry points.

Structs§

DigestMd5
Streaming MD5 hasher compatible with RustCrypto Digest (0.11).
Md5
Streaming MD5 hasher.
Md5Core
Core MD5 state for the Digest trait wrapper.
Md5Engine
Batch MD5 engine. Construct once and reuse.
Md5State
Incremental state for one MD5 message in a multi-stream workload.

Constants§

BLOCK_SIZE
MD5 compresses 64-byte blocks.
DIGEST_LEN
MD5 produces a 16-byte digest.
K
Per-round additive constants T[i] (RFC 1321 §3.4).
S
Left-rotate amounts for each of the 64 steps (RFC 1321 §3.4).
STATE_INIT
Standard MD5 initialization vector (RFC 1321 §3.3).

Traits§

Digest
Convenience wrapper trait covering functionality of cryptographic hash functions with fixed output size.

Functions§

available_backends
Single-stream backends compiled into this build.
backend_name
Active single-stream backend name.
digest
One-shot hashing (active backend).
digest_hex
Hash a message and return its 32-character lowercase hexadecimal digest.
digest_opt_scalar
In-tree production-kernel one-shot (ignores opt / force-portable).
digest_portable
Textbook-oracle one-shot (CI / differential). Ignores accelerated backends.
digest_reader
Hash a reader until EOF using a 64 KiB stack buffer.
etag_hex
Alias for digest_hex for applications using MD5-shaped ETags.
etag_hex_streaming
Hash a reader to lowercase hexadecimal until EOF.
hash_pair
Software multi-buffer: pair-interleaved hashing for independent messages. Software multi-buffer: pair-interleaved hashing for independent messages. Software multi-buffer: pair-interleaved hashing for independent messages. Hash two messages; equal-length pairs use interleaved compress when active.
hex_encode
Encode every input byte as two lowercase hexadecimal characters.
hex_encode_digest
Lowercase hex of a 16-byte digest into a fixed 32-byte array (no alloc).
hex_encode_into
Write lowercase hex into out; returns bytes written (2 * bytes.len()).
is_portable_only
True when the selected compressor is the textbook oracle.
md5_many
Hash many independent messages with the process-default engine.
pair_path_active
Software multi-buffer: pair-interleaved hashing for independent messages. Software multi-buffer: pair-interleaved hashing for independent messages. Software multi-buffer: pair-interleaved hashing for independent messages. Whether hash_many should use the software pair path.
runtime_lanes
Best usable batch width on this process (cached CPUID on x86_64).
simd_active
Whether SIMD multi-buffer can run on this process.
simd_lanes
Compile-time max SIMD width for this target (runtime CPUID may be lower).
simd_name
Best SIMD kernel label for reports.