Crate memspan 

memspan

SIMD-accelerated byte-class scanning for lexers and parsers.

English | 简体中文

Quick start

let src = b"   hello, world";

// Skip leading whitespace — dispatches to AVX2 / NEON / SIMD128 at runtime.
let n = memspan::skip_whitespace(src);
assert_eq!(n, 3);

// Find the first comma.
let comma = memspan::skip_until(src, b',');
assert_eq!(comma, Some(8));

Overview

memspan provides zero-allocation, no_std-compatible functions to skip, count, and locate bytes in ASCII character classes, dispatching to the best available SIMD backend at runtime:

Architecture	Dispatch order
x86_64	AVX-512BW → AVX2 → SSE4.2 → scalar
x86	SSE4.2 → scalar
aarch64	NEON → scalar
wasm32	SIMD128 → scalar
other	scalar

Installation

[dependencies]
memspan = "0.1"

For no_std without an allocator:

[dependencies]
memspan = { version = "0.1", default-features = false }

Built-in classes

All class functions return the byte length of the longest matching prefix.

Function	Matches
skip_whitespace	, \t, \r, \n
skip_digits	0–9
skip_hex_digits	0–9, a–f, A–F
skip_octal_digits	0–7
skip_binary	0, 1
skip_alpha	a–z, A–Z
skip_alphanumeric	a–z, A–Z, 0–9
skip_ident_start	a–z, A–Z, _
skip_ident	a–z, A–Z, 0–9, _
skip_lower	a–z
skip_upper	A–Z
skip_ascii	0x00–0x7F
skip_non_ascii	0x80–0xFF
skip_ascii_graphic	0x21–0x7E (printable non-space)
skip_ascii_control	0x00–0x1F, 0x7F

Generic operations

skip_while, skip_until, count_matches, and find_last accept any Needles value — a single u8, a fixed-size array [u8; N], or a &[u8] slice:

// Skip while any of several bytes match.
let n = memspan::skip_while(b"  ,\t ok", [b' ', b',', b'\t']);
assert_eq!(n, 5);

// Find the first occurrence of any needle — like memchr but multi-byte.
let pos = memspan::skip_until(b"hello\nworld", b'\n');
assert_eq!(pos, Some(5));

// Count every newline in a buffer.
let lines = memspan::count_matches(b"a\nb\nc\n", b'\n');
assert_eq!(lines, 3);

// Find the rightmost match.
let last = memspan::find_last(b"\"hello\"", b'"');
assert_eq!(last, Some(6));

Custom classes with skip_class!

Define your own byte class and get the same SIMD dispatch as the built-ins:

memspan::skip_class! {
    /// Skip whitespace and commas.
    pub fn skip_ws_and_comma(bytes = [b' ', b'\t', b'\r', b'\n', b',']);
}

memspan::skip_class! {
    /// Skip lowercase ASCII letters.
    pub fn skip_lowercase(ranges = [b'a'..=b'z']);
}

memspan::skip_class! {
    /// Skip alphanumeric plus common punctuation.
    pub fn skip_punct_ident(
        bytes  = [b'_', b'-', b'!', b'?'],
        ranges = [b'a'..=b'z', b'A'..=b'Z', b'0'..=b'9'],
    );
}

assert_eq!(skip_ws_and_comma(b"  , ok"), 4);
assert_eq!(skip_lowercase(b"abcXYZ"), 3);
assert_eq!(skip_punct_ident(b"hello-world! 42"), 12);

Benchmarks

Throughput in GiB/s across input sizes, measured on GitHub Actions runners (2026-04-22, --quick Criterion runs). Environments: aarch64 — macOS-latest (ARM64, NEON); x86_64 — ubuntu-latest (X64, runtime AVX2 detection).

aarch64 — NEON

Function	16 B	32 B	64 B	256 B	4 KiB	64 KiB
skip_binary	1.9	7.0	10.9	23.1	39.5	34.9
skip_octal_digits	2.2	7.3	12.3	27.5	41.0	45.9
skip_digits	2.2	7.3	10.6	26.7	39.9	44.8
skip_hex_digits	1.8	4.1	6.4	14.8	21.8	23.5
skip_alpha	2.3	5.8	10.4	23.1	32.6	37.6
skip_alphanumeric	1.8	4.3	6.8	14.9	19.7	23.1
skip_ident_start	1.5	3.4	6.1	12.5	24.4	23.3
skip_ident	1.3	3.8	5.6	13.2	16.6	17.8
skip_whitespace	1.9	4.5	7.0	15.2	20.2	19.0

aarch64 — scalar fallback

Function	16 B	32 B	64 B	256 B	4 KiB	64 KiB
skip_binary	2.3	2.1	1.9	1.8	2.0	2.0
skip_octal_digits	1.7	2.2	2.0	1.9	2.1	2.1
skip_digits	1.9	2.3	2.2	2.3	2.4	2.2
skip_hex_digits	1.7	1.4	1.5	1.5	1.2	1.3
skip_alpha	2.3	2.5	2.2	1.9	1.9	1.9
skip_alphanumeric	1.8	1.8	1.7	1.5	1.6	1.6
skip_ident_start	1.9	1.9	1.9	1.7	1.9	2.0
skip_ident	1.5	1.6	1.5	1.5	1.6	1.6
skip_whitespace	1.4	1.6	1.8	1.6	1.8	1.8

x86_64 — AVX2

Function	16 B	32 B	64 B	256 B	4 KiB	64 KiB
skip_binary	2.0	2.5	4.7	15.8	60.7	88.4
skip_octal_digits	2.0	2.5	4.7	16.3	62.3	62.8
skip_digits	2.0	2.5	4.7	16.3	63.2	80.9
skip_hex_digits	1.4	1.7	3.1	10.3	33.2	38.3
skip_alpha	2.0	2.5	4.3	14.6	59.9	68.6
skip_alphanumeric	1.5	1.7	3.2	10.6	31.6	37.5
skip_ident_start	1.5	1.7	3.3	11.1	39.7	46.3
skip_ident	1.8	1.6	2.9	9.9	28.4	33.8
skip_whitespace	1.9	2.5	4.4	13.7	38.2	46.5

x86_64 — scalar fallback

Function	16 B	32 B	64 B	256 B	4 KiB	64 KiB
skip_binary	2.3	2.5	2.1	2.7	3.0	3.0
skip_octal_digits	2.2	2.4	2.1	2.6	3.0	3.0
skip_digits	2.1	2.5	2.1	2.7	3.0	3.0
skip_hex_digits	1.4	1.5	1.2	1.4	1.5	1.5
skip_alpha	2.1	1.8	1.8	1.7	1.8	1.8
skip_alphanumeric	1.3	1.4	1.2	1.4	1.5	1.5
skip_ident_start	1.3	1.4	1.2	1.4	1.5	1.5
skip_ident	1.3	1.4	1.3	1.4	1.5	1.5
skip_whitespace	1.8	1.5	1.3	1.9	2.0	2.0

Generic dispatch (skip_until / skip_while)

Function	Backend	16 B	32 B	64 B	256 B	4 KiB	64 KiB
skip_until	aarch64 NEON	0.5	1.1	2.5	7.1	15.5	17.0
skip_until	aarch64 scalar	1.4	1.6	1.5	1.4	1.6	1.6
skip_until	x86_64 AVX2	0.5	0.7	1.3	4.9	25.8	36.9
skip_until	x86_64 scalar	0.9	0.9	0.9	1.0	1.0	1.0
skip_while	aarch64 NEON	0.7	1.5	3.0	8.6	16.3	16.7
skip_while	aarch64 scalar	1.7	1.9	2.0	1.9	2.0	2.1
skip_while	x86_64 AVX2	0.7	0.8	1.4	5.2	26.6	37.1
skip_while	x86_64 scalar	1.1	1.1	1.0	1.1	1.2	1.2

skip_class! macro vs skip_while

	Backend	16 B	32 B	64 B	256 B	4 KiB	64 KiB
skip_class! macro	aarch64 NEON	2.0	4.4	6.9	13.5	16.7	18.2
skip_while (array)	aarch64 NEON	0.7	1.4	3.0	8.8	15.1	17.6
skip_class! macro	x86_64 AVX2	1.7	2.4	4.3	13.6	35.8	41.7
skip_while (array)	x86_64 AVX2	0.7	0.8	1.4	5.2	26.4	37.8

Features

Feature	Default	Description
std	✓	Link against the standard library
alloc		Enable heap allocation without std
(neither)		Pure no_std / no_alloc

License

memspan is dual-licensed under the MIT license and the Apache License (Version 2.0).

See LICENSE-APACHE, LICENSE-MIT for details.

Copyright (c) 2026 Al Liu.

Re-exports

pub use skip::*;

Modules

skip

SIMD-accelerated skipping utilities for lexing.

utils

Utilities for SIMD-accelerated lexing, including CPU feature detection.

Macros

skip_class

Define a custom skip_* function for an ASCII byte class, generating the same scalar fallback + SIMD loop the built-in skip::skip_digits, skip::skip_whitespace, etc. use internally. On x86/x86_64 the generated function dispatches through AVX2 (256-bit) → SSE4.2 (128-bit) → scalar; on aarch64 through NEON; on wasm32 through SIMD128.

Traits

Needles

Needle types for SIMD-accelerated searching.