md5-simd
MD5 checksums for object-storage workloads: streaming uploads, snapshot ETags, RustCrypto integration, and batches of independent messages.
The default build uses optimized x86_64 and little-endian AArch64 single-stream compressors with SIMD batch kernels. Other targets retain a portable Rust implementation. Single-stream hashing and batch hashing use separate dispatch paths.
MD5 is cryptographically broken. Use it only for legacy interoperability and non-adversarial checksums, never for passwords, signatures, or authenticity.
Getting started
[]
= "0.2"
The library import is md5_simd. Default features are std, opt, digest, and
simd. These features can also be selected individually.
use ;
assert_eq!;
let mut hasher = new;
hasher.update;
assert_eq!;
hasher.update;
assert_eq!;
Streaming and snapshots
Md5::finalize() consumes the hasher. finalize_snapshot() and digest_so_far()
return the current digest without consuming or changing it. clone().finalize()
remains supported. finalize_reset() returns the digest and resets the hasher
for another message.
With std, the hasher implements std::io::Write:
#
#
# Ok::
Reader helpers hash until EOF, retry interrupted reads, and propagate other I/O
errors. etag_hex_streaming(reader, chunk) clamps its buffer size to 4 KiB–1 MiB.
| API | Output | Allocation |
|---|---|---|
digest, native byte finalizers |
[u8; 16] |
None |
hex_encode_digest |
[u8; 32] |
None |
hex_encode_into |
Caller-provided byte slice | None |
hex_encode, digest_hex, etag_hex |
Lowercase String |
One output allocation |
Md5::finalize_hex (std) |
Snapshot String |
One output allocation |
hex_encode encodes the entire slice. hex_encode_into requires at least twice
as many output bytes as input bytes and preserves unused output bytes.
An MD5 hex string has the shape of an ordinary single-part ETag. This crate does not implement multipart ETag composition, HTTP quoting, Content-MD5 Base64 encoding, or storage-specific encryption semantics.
RustCrypto Digest
The digest feature provides DigestMd5 and re-exports the Digest 0.11 traits.
Use DigestMd5 for generic Digest consumers and Md5 for the native snapshot API.
Traits from Digest 0.10 are a different API and are not interchangeable.
#
#
Both interfaces share the selected compressor and padding builder. Digest owns its partial-block buffer; complete blocks are compressed directly.
Batch hashing
use ;
let messages = ;
let inputs: = messages.iter.map.collect;
let mut outputs = ;
new.hash_many;
assert!;
hash_many preserves input order and processes contiguous equal-length runs.
Runs with at least four messages of at least 32 bytes can use SIMD; small runs,
short messages, and unsupported targets use the active single-stream backend.
Inputs need no special alignment. The caller supplies output storage, and the
byte batch API performs no heap allocation or thread spawning.
When an object scheduler has non-adjacent equal-size messages, the std
feature also provides Md5Engine::hash_many_grouped. It sorts internal indices
by message length, runs the same SIMD dispatcher, and scatters digests back to
the original order. This method allocates temporary indices and digest storage;
use it only when the expected SIMD work outweighs that scheduling cost.
| Target | Batch selection with simd |
|---|---|
x86_64 with std |
Runtime detection: AVX2 (8 lanes), AVX-512F + AVX2 (16 lanes), or scalar |
| Little-endian aarch64 | NEON (4 lanes) or paired NEON (8 lanes) |
Other targets (including big-endian); x86_64 without std |
Scalar fallback |
Any target with force-portable |
Textbook scalar fallback |
On an AVX-512 host, runs of up to eight messages prefer AVX2. Larger runs can
process up to four vector groups per kernel call. runtime_lanes() and
simd_name() report available capability; a particular call may use fewer lanes
or fall back. SIMD accelerates independent messages, not one message's sequential
MD5 chain.
Md5State is a copyable incremental state. Md5Engine::update_many and
finalize_many loop over independent scalar states; they are not incremental
SIMD APIs. Md5State::finalize() is a non-consuming snapshot.
Measured AArch64 batch results
On Apple Silicon, same-host ABBA measurements show the intended batch shape:
| Workload | NEON result vs sequential md-5 |
|---|---|
| 8 independent × 1 MiB | 3.10× |
| 16 independent × 1 MiB | 4.22× |
| 7 independent × 1 MiB tail batch | 2.67× |
| 9 independent × 1 MiB with one scalar tail | 1.73× |
| grouped mixed lengths | 1.60× |
These are independent-message throughput results. A single MD5 message remains
serial and is measured separately; current AArch64 single-message oneshot and
streaming are approximately at RustCrypto md-5 parity. For non-adjacent object
sizes, hash_many_grouped can recover equal-length runs when its temporary sort
and output scatter cost is justified.
Features and portability
| Feature | Default | Effect |
|---|---|---|
std |
Yes | I/O helpers, native snapshot hex, batch hex, x86 runtime detection |
opt |
Yes | x86_64 single-stream assembly; little-endian aarch64 single-aarch kernel |
digest |
Yes | Optional RustCrypto Digest 0.11 dependency |
simd |
Yes | SIMD batches; single-stream dispatch is unchanged |
force-portable |
No | Overrides the selected single-stream and batch acceleration |
pair-interleave |
No | Experimental scalar pairs when opt and force-portable are off |
zeroize |
No | Best-effort clearing of native Md5 on drop |
# Portable scalar with I/O support:
= { = "../md5-simd", = false, = ["std"] }
# no_std library (uses alloc for String-returning helpers):
# md5-simd = { path = "../md5-simd", default-features = false }
Byte hashing APIs do not allocate. no_std builds use alloc; applications using
allocating helpers need an allocator. zeroize does not cover Md5State copies,
Digest buffers, or all compiler-made copies. Call reset() before reusing a
manually cleared Md5.
The project uses edition 2024. No minimum supported Rust version is declared; the release checks use Rust 1.98.1. Older compiler versions are not guaranteed.
Performance
Performance depends on CPU, message size, batch shape, features, and compiler. Performance notes record the measured aarch64 refactor comparison, validation limits, and reproduction commands. No result is a universal throughput guarantee.
Applications should validate their own checksum formatting, I/O error handling, and workload performance when adopting a new implementation.
Development
CI covers tests, feature combinations, documentation, and package verification. The Audit workflow checks locked dependencies on changes and weekly. Version tags run the release checks; registry upload requires an explicit manual dispatch.
Run both clippy profiles: --all-features enables force-portable and excludes
SIMD adapters. README examples run as rustdoc tests.
Architecture describes state ownership, dispatch, padding, and unsafe boundaries. See the changelog for release notes and the release guide for maintainer checks. Repository tooling and local files are excluded from the published package by an explicit manifest allowlist.
License
Apache-2.0. The vendored scalar assembly and its glue retain their BSD-2-Clause and assembly provenance notices. See LICENSE and NOTICE.
Acknowledgements
Thank you to fast-md5 and md5-many for their open-source work on MD5 performance and multi-buffer design. Third-party code attribution is retained in NOTICE and the relevant source headers.