English | 中文

whasher : Hardware-accelerated hashing and collections

• Overview
• Usage
  • Installation and target requirements
  • Hashing and standard collections
  • Streaming checksums
  • Concurrent collections
• Features
• Design
  • Direct and generic hashing
  • Streaming path
  • Collection path
  • Compatibility boundaries
• Technology stack
• Directory structure
• API reference
  • Hash functions
  • StreamHasher
  • Collection types and re-exports
  • Collection constructors
• Validation

Overview

whasher provides byte and integer hashing, streaming checksums, and hash collections backed by gxhash. It supports 64-bit and 128-bit output, explicit seeds, and generic values implementing Rust's Hash trait.

Standard maps and sets use GxBuildHasher. Concurrent maps and sets combine the same builder with papaya, keeping hash selection consistent across collection types.

These are non-cryptographic hashes. Do not use them for passwords, signatures, authentication, or tamper-proof integrity checks. Wider output does not guarantee collision freedom.

Usage

Installation and target requirements

cargo add whasher

Use Rust 2024 with toolchain support for slice::as_chunks (stabilized in Rust 1.88). The current backend requires AES and SSE2 on x86_64, or AES and NEON on aarch64. It has no portable software fallback for unsupported targets.

The repository workspace enables target-feature=+aes. Outside this workspace, configure the required target features explicitly, or build for the local processor:

RUSTFLAGS="-C target-cpu=native" cargo build

The processor must support the required instructions. Native builds may not run on deployment machines with different processor capabilities; select flags for the deployment target when distributing binaries.

Hashing and standard collections

This example follows the hashing and collection tests.

use whasher::{
  Entry, fast_hash, fast_hash_u64, fast_hash_with_seed, fast_hash128,
  hash_value, hash_value_with_seed, hash128, hash128_with_seed,
  new_hash_map, new_hash_set,
};

fn main() {
  let data = b"hello world";
  assert_eq!(fast_hash(data), fast_hash_with_seed(data, 0));
  assert_eq!(fast_hash_u64(42), fast_hash(&42u64.to_le_bytes()));
  assert_eq!(fast_hash128(data), hash128_with_seed(data, 0));
  assert_eq!(hash128(data, 7, 9), hash128_with_seed(data, 7 ^ 9u64.rotate_left(32)));
  assert_eq!(hash_value(&(1u64, 2u64)), hash_value_with_seed(&(1u64, 2u64), 0));

  let mut map = new_hash_map();
  map.insert("key", 100);
  if let Entry::Occupied(mut entry) = map.entry("key") {
    *entry.get_mut() += 50;
  }
  assert_eq!(map.get("key"), Some(&150));

  let mut set = new_hash_set();
  set.insert("alpha");
  assert!(set.contains("alpha"));
}

Streaming checksums

Chunk boundaries do not affect the checksum of the concatenated bytes. The example crosses the internal 64-byte stripe boundary, as the streaming tests do.

use whasher::{StreamHasher, compute_checksum, compute_checksum_with_seed};

fn main() {
  let data = b"The quick brown fox jumps over the lazy dog. A fast streaming hash with buffer.";
  let mut hasher = StreamHasher::new();
  for chunk in data.chunks(7) {
    hasher.write(chunk);
  }
  let checksum = hasher.finish();
  assert_eq!(checksum, compute_checksum(data));
  assert_eq!(hasher.finish(), checksum);
  assert_eq!(hasher.total_bytes_written(), data.len() as u64);

  hasher.write(b"tail");
  let full = [data.as_slice(), b"tail"].concat();
  assert_eq!(hasher.finish(), compute_checksum(&full));

  let mut seeded = StreamHasher::with_seed(42);
  seeded.write(data);
  assert_eq!(seeded.finish(), compute_checksum_with_seed(data, 42));
  seeded.reset();
  assert!(seeded.is_empty());
  assert_eq!(seeded.finish(), compute_checksum_with_seed(b"", 42));
}

compute_checksum(data) matches StreamHasher::new() followed by write(data) and finish(). It is not an alias for fast_hash(data). Use StreamHasher, rather than the re-exported GxHasher, when byte-chunk invariance is required.

Concurrent collections

This scoped-thread example adapts the concurrent map and set test. Scoped threads borrow the collections; detached ownership can instead use Arc, as in the test suite.

use std::thread;

use whasher::{papaya_map_with_capacity, papaya_set_with_capacity};

fn main() {
  let map = papaya_map_with_capacity::<u64, u64>(4000);
  let set = papaya_set_with_capacity::<u64>(4000);
  thread::scope(|scope| {
    for t in 0..4u64 {
      let map = &map;
      let set = &set;
      scope.spawn(move || {
        let map_pin = map.pin();
        let set_pin = set.pin();
        for i in 0..1000u64 {
          let key = t * 1000 + i;
          map_pin.insert(key, key * 2);
          set_pin.insert(key);
        }
      });
    }
  });

  let map_pin = map.pin();
  let set_pin = set.pin();
  assert_eq!(map.len(), 4000);
  assert_eq!(set.len(), 4000);
  assert_eq!(map_pin.get(&42), Some(&84));
  assert!(set_pin.contains(&42));
}

Keep pinned access handles alive while using borrowed entries. Drop them promptly after access so they do not unnecessarily delay memory reclamation. Concurrency and reclamation behavior come from papaya; not every operation is guaranteed to be lock-free.

Features

• AES-backed 64-bit and 128-bit byte hashing, with default or explicit seeds.
• Little-endian integer hashing and generic Hash input support.
• Chunk-invariant streaming checksums with non-destructive finalization and seed-preserving reset.
• Allocation-free streaming state: 128-byte size, 64-byte alignment, and a 64-byte residual buffer.
• Compile-time initialization for the default streaming seed and independent folding lanes for instruction-level parallelism.
• Standard and concurrent maps and sets, including initial-capacity constructors.

No optional crate features are currently defined; the default feature set is empty.

Design

All public interfaces and internal streaming logic reside in src/lib.rs. The implementation separates responsibilities through functions and types rather than submodules.

Direct and generic hashing

fast_hash* and hash128* delegate to gxhash::gxhash64 or gxhash::gxhash128. fast_hash_u64 first converts the integer to little-endian bytes. hash128 combines its seeds with seed_a ^ seed_b.rotate_left(32) before calling the backend.

hash_value* creates a seeded GxHasher, passes it to Hash::hash, then calls Hasher::finish. This path follows the type's Hash implementation, not a canonical byte serialization.

Streaming path

1. new or with_seed initializes 4 folding lanes. The zero-seed state is precomputed; other seeds are mixed with lane-specific salts.
2. write fills any buffered remainder, folds complete 64-byte stripes, and retains the trailing bytes. Stripe number modulo 4 selects the lane, independent of write boundaries.
3. Aligned groups of 4 stripes update independent lanes. Complete input stripes are read directly without copying into the residual buffer.
4. finish merges the lane states in order, then hashes the remainder followed by the total byte count encoded as little-endian u64. It leaves the state unchanged.
5. reset restores the original seed state and clears counters without zeroing the residual buffer.

compute_checksum* constructs this state, writes the full input, and finalizes it. Processing is linear in input length with constant auxiliary memory.

Collection path

Standard collection constructors install DefaultBuildHasher::default(). Concurrent constructors use the papaya builder with GxBuildHasher::default() and an optional initial capacity. The backend defaults randomize collection hashing unless the dependency's deterministic feature is enabled through Cargo feature unification.

Compatibility boundaries

Fixed-seed byte hashing is repeatable for the same algorithm configuration. Persisted or transmitted hashes should record the backend version, seed, and algorithm choice; do not assume compatibility across backend or streaming implementation changes.

Generic Hash input is not guaranteed portable across platforms or compiler versions. Encode persistent keys explicitly before byte hashing. Distinct seed pairs in hash128 can produce the same combined 64-bit seed. The APIs do not reproduce MurmurHash or XxHash output from C# implementations.

Technology stack

Dependency versions above are the requirements declared in the package manifest, not exact version pins.

Directory structure

Paths are relative to the package root.

whasher/
  src/
    lib.rs
  tests/
    main.rs
  readme/
    en.md
    zh.md
  AGENTS.md
  Cargo.toml
  Cargo.lock
  README.mdt
  README.md
  test.sh

tests/main.rs covers collection operations, concurrent insertion, hash distribution, seeded hashing, streaming boundaries, reset, cloning, and long inputs. README.mdt includes both language sources; edit those sources and regenerate the combined README.

API reference

All interfaces below are available at the whasher crate root. Arguments named bytes or data borrow input slices; hashing functions return values directly, not Result.

Hash functions

Explicit u64 seeds are interpreted by the backend as i64 bit patterns; the high bit is preserved. Empty slices are valid input.

StreamHasher

Streaming checksum state with private fields. Implements Clone, Debug, Default, and core::hash::Hasher. Clones preserve the current state and can then be advanced or reset independently. Inherent write and finish methods do not require importing the trait.

Chunk invariance concerns write(&[u8]) calls over the same concatenated bytes. It does not make arbitrary Hash implementations a portable encoding. The byte counter wraps modulo 2^64, so it should not be used to track larger lifetime totals.

Collection types and re-exports

Collection constructors

Constructors impose no key trait bounds. Insertion and lookup require the underlying collections' Hash and Eq bounds; cross-thread sharing also requires the applicable Send and Sync bounds. Capacity is an initial allocation hint, not a size limit.

Validation

Run from the package directory within the repository workspace:

./test.sh
bun x mdt

The test script invokes cargo nextest run --all-features --no-capture and requires Cargo Nextest. The document generator requires Bun. The package inherits workspace lint settings.

whasher : 硬件加速哈希与集合

• 项目介绍
• 使用演示
  • 安装与目标要求
  • 哈希与普通集合
  • 流式校验和
  • 并发集合
• 特性介绍
• 设计思路
  • 直接哈希与泛型哈希
  • 流式路径
  • 集合路径
  • 兼容性边界
• 技术堆栈
• 目录结构
• API 说明
  • 哈希函数
  • StreamHasher
  • 集合类型与重导出
  • 集合构造函数
• 验证

项目介绍

whasher 提供基于 gxhash 的字节与整数哈希、流式校验和与哈希集合。支持 64 位与 128 位输出、显式种子，以及实现 Rust Hash trait 的泛型值。

普通映射与集合搭载 GxBuildHasher。并发映射与集合在同一构建器之上组合 papaya，使各集合类型的哈希选择保持一致。

以上均为非加密哈希。不可用于口令、签名、身份认证或防篡改完整性校验。更宽的输出不保证无碰撞。

使用演示

安装与目标要求

cargo add whasher

使用 Rust 2024，工具链需支持 slice::as_chunks（Rust 1.88 起稳定）。当前后端在 x86_64 需要 AES 与 SSE2，在 aarch64 需要 AES 与 NEON，对不支持的目标没有可移植软件回退。

仓库工作区已启用 target-feature=+aes。工作区之外需显式配置所需 target-feature，或面向本机处理器构建：

RUSTFLAGS="-C target-cpu=native" cargo build

处理器必须支持所需指令集。面向本机的构建可能无法在指令集不同的部署机器上运行；分发二进制时请按部署目标选择参数。

哈希与普通集合

本示例对应哈希与集合测试。

use whasher::{
  Entry, fast_hash, fast_hash_u64, fast_hash_with_seed, fast_hash128,
  hash_value, hash_value_with_seed, hash128, hash128_with_seed,
  new_hash_map, new_hash_set,
};

fn main() {
  let data = b"hello world";
  assert_eq!(fast_hash(data), fast_hash_with_seed(data, 0));
  assert_eq!(fast_hash_u64(42), fast_hash(&42u64.to_le_bytes()));
  assert_eq!(fast_hash128(data), hash128_with_seed(data, 0));
  assert_eq!(hash128(data, 7, 9), hash128_with_seed(data, 7 ^ 9u64.rotate_left(32)));
  assert_eq!(hash_value(&(1u64, 2u64)), hash_value_with_seed(&(1u64, 2u64), 0));

  let mut map = new_hash_map();
  map.insert("key", 100);
  if let Entry::Occupied(mut entry) = map.entry("key") {
    *entry.get_mut() += 50;
  }
  assert_eq!(map.get("key"), Some(&150));

  let mut set = new_hash_set();
  set.insert("alpha");
  assert!(set.contains("alpha"));
}

流式校验和

分块边界不影响拼接后字节的校验和。示例跨越内部 64 字节条带边界，与流式测试一致。

use whasher::{StreamHasher, compute_checksum, compute_checksum_with_seed};

fn main() {
  let data = b"The quick brown fox jumps over the lazy dog. A fast streaming hash with buffer.";
  let mut hasher = StreamHasher::new();
  for chunk in data.chunks(7) {
    hasher.write(chunk);
  }
  let checksum = hasher.finish();
  assert_eq!(checksum, compute_checksum(data));
  assert_eq!(hasher.finish(), checksum);
  assert_eq!(hasher.total_bytes_written(), data.len() as u64);

  hasher.write(b"tail");
  let full = [data.as_slice(), b"tail"].concat();
  assert_eq!(hasher.finish(), compute_checksum(&full));

  let mut seeded = StreamHasher::with_seed(42);
  seeded.write(data);
  assert_eq!(seeded.finish(), compute_checksum_with_seed(data, 42));
  seeded.reset();
  assert!(seeded.is_empty());
  assert_eq!(seeded.finish(), compute_checksum_with_seed(b"", 42));
}

compute_checksum(data) 等价于 StreamHasher::new() 后依次 write(data) 与 finish()，并非 fast_hash(data) 的别名。需要字节分块恒等时应使用 StreamHasher，而非重导出的 GxHasher。

并发集合

本作用域线程示例改写自并发映射与集合测试。作用域线程借用集合；脱离作用域的所有权可改用 Arc，如测试套件所示。

use std::thread;

use whasher::{papaya_map_with_capacity, papaya_set_with_capacity};

fn main() {
  let map = papaya_map_with_capacity::<u64, u64>(4000);
  let set = papaya_set_with_capacity::<u64>(4000);
  thread::scope(|scope| {
    for t in 0..4u64 {
      let map = &map;
      let set = &set;
      scope.spawn(move || {
        let map_pin = map.pin();
        let set_pin = set.pin();
        for i in 0..1000u64 {
          let key = t * 1000 + i;
          map_pin.insert(key, key * 2);
          set_pin.insert(key);
        }
      });
    }
  });

  let map_pin = map.pin();
  let set_pin = set.pin();
  assert_eq!(map.len(), 4000);
  assert_eq!(set.len(), 4000);
  assert_eq!(map_pin.get(&42), Some(&84));
  assert!(set_pin.contains(&42));
}

借用条目期间应保持 pin 访问句柄存活，访问后及时释放，以免不必要地延迟内存回收。并发与回收行为来自 papaya，并非所有操作都保证无锁。

特性介绍

• 基于 AES 的 64 位与 128 位字节哈希，支持默认或显式种子。
• 小端整数哈希与泛型 Hash 输入支持。
• 分块恒等的流式校验和，非破坏性收尾，保留种子的复位。
• 零堆分配的流式状态：128 字节体积、64 字节对齐、64 字节残留缓冲。
• 默认流式种子编译期初始化，独立折叠链提升指令级并行。
• 普通与并发映射、集合，均支持初始容量构造。

当前未定义可选 crate 特性，默认特性集为空。

设计思路

全部公开接口与内部流式逻辑集中在 src/lib.rs，以函数与类型而非子模块划分职责。

直接哈希与泛型哈希

fast_hash* 与 hash128* 委托给 gxhash::gxhash64 或 gxhash::gxhash128。fast_hash_u64 先将整数转为小端字节。hash128 以 seed_a ^ seed_b.rotate_left(32) 合并种子后调用后端。

hash_value* 创建带种子的 GxHasher，经 Hash::hash 写入后调用 Hasher::finish。该路径遵循类型的 Hash 实现，而非规范的字节序列化。

流式路径

1. new 或 with_seed 初始化 4 条折叠链。零种子状态编译期预计算；其他种子与各链盐值混合。
2. write 补满缓冲残留，折叠完整 64 字节条带，并保留尾部字节。条带号对 4 取模决定链归属，与写入边界无关。
3. 对齐的 4 条带组更新独立折叠链。完整输入条带直接读取，不拷贝进残留缓冲。
4. finish 依序汇合各链状态，再对残留字节与按小端编码的累计字节数做末端混合，不改动内部状态。
5. reset 恢复构造时种子对应的初态并清零计数器，不清零残留缓冲。

compute_checksum* 构造该状态、写入全部输入并收尾。处理耗时随输入线性增长，辅助内存恒定。

集合路径

普通集合构造函数安装 DefaultBuildHasher::default()。并发构造函数以 GxBuildHasher::default() 配置 papaya 构建器，可指定初始容量。后端默认随机化集合哈希，除非通过 Cargo 特性统一启用依赖的 deterministic 特性。

兼容性边界

固定种子的字节哈希在相同算法配置下可复现。落盘或传输的哈希应记录后端版本、种子与算法选择；不要假设跨后端或流式实现变更仍兼容。

泛型 Hash 输入不保证跨平台或跨编译器版本可移植。持久化键请先显式编码再做字节哈希。hash128 的不同种子对可能合并出相同的 64 位种子。以上接口不重现 C# 实现的 MurmurHash 或 XxHash 输出。

技术堆栈

目录结构

路径相对包根目录。

whasher/
  src/
    lib.rs
  tests/
    main.rs
  readme/
    en.md
    zh.md
  AGENTS.md
  Cargo.toml
  Cargo.lock
  README.mdt
  README.md
  test.sh

tests/main.rs 覆盖集合操作、并发插入、哈希分布、带种子哈希、流式边界、复位、克隆与超长输入。README.mdt 包含两语言源文档；请编辑源文档后重新生成合并的 README。

API 说明

以下接口均位于 whasher crate 根。参数 bytes 或 data 借用输入切片；哈希函数直接返回数值，不返回 Result。

哈希函数

显式 u64 种子由后端按 i64 位型解释，最高位保留。空切片为合法输入。

StreamHasher

流式校验和状态，字段私有。实现 Clone、Debug、Default 与 core::hash::Hasher。克隆保留当前状态，之后可独立推进或复位。固有方法 write 与 finish 无需导入 trait。

分块恒等针对相同拼接字节上的 write(&[u8]) 调用，并不使任意 Hash 实现成为可移植编码。字节计数按 2^64 取模环绕，不宜用于统计更长的生命周期总量。

集合类型与重导出

集合构造函数

构造函数不施加键 trait 约束。插入与查找要求底层集合的 Hash 与 Eq 约束；跨线程共享还需满足相应的 Send 与 Sync 约束。容量是初始分配提示，并非大小上限。

验证

在仓库工作区内于包目录执行：

./test.sh
bun x mdt

测试脚本调用 cargo nextest run --all-features --no-capture，需要 Cargo Nextest。文档生成需要 Bun。包继承工作区 lint 设置。