lattice-embed

Vector embedding generation with SIMD-accelerated operations for semantic search and similarity matching.

Features

• Native Embeddings: Generate embeddings locally using pure Rust inference (no ONNX, no Python)
• SIMD Acceleration: AVX2/AVX-512/NEON optimized vector operations (7x speedup)
• LRU Cache: SHA-256-based caching to avoid recomputation
• Async API: Full async/await support with tokio

Models

TextEmbedding3Small's ID is an OpenAI model id, not a HuggingFace one; EmbeddingModel::is_remote() returns true for it and the native embedding service rejects it (see supported-model reference).

Dimensions and input-token limits vary per model. See the supported-model reference for every variant's limit, pooling strategy, and shipped/partial/scaffold-only availability.

Services

NativeEmbeddingService

Single-model service with lazy initialization. Inference is serialized (one call at a time). Concurrency is handled internally.

use lattice_embed::{EmbeddingService, EmbeddingModel, NativeEmbeddingService};

let service = NativeEmbeddingService::default();
let embedding = service.embed_one("Hello, world!", EmbeddingModel::default()).await?;
assert_eq!(embedding.len(), 384);

CachedEmbeddingService

Wraps any EmbeddingService with LRU caching. Identical texts return cached embeddings.

use lattice_embed::{CachedEmbeddingService, NativeEmbeddingService, EmbeddingService};
use std::sync::Arc;

let inner = Arc::new(NativeEmbeddingService::new());
let cached = CachedEmbeddingService::new(inner, 1000); // 1000 entry cache

let emb1 = cached.embed_one("Hello", EmbeddingModel::default()).await?;
let emb2 = cached.embed_one("Hello", EmbeddingModel::default()).await?; // cache hit

SIMD Vector Operations

Dispatch is resolved per platform. x86_64 uses runtime feature detection with automatic scalar fallback. On aarch64 the f32 kernels always use NEON (mandatory on that architecture, no runtime check), while the int8 kernel runtime-detects SDOT. On wasm32 the SIMD choice is fixed at compile time and covers the f32 kernels only (see below).

wasm32 (SIMD128)

The SIMD128 route exists for the four f32 kernels: dot_product, cosine_similarity, squared Euclidean distance, and normalize. The int8 path (dot_product_i8) has no SIMD128 branch and always dispatches to the scalar kernel on wasm32.

Unlike the x86_64/aarch64 rows above, wasm32 has no runtime CPU-feature detection: a given .wasm binary either was or wasn't compiled with -C target-feature=+simd128, and that choice is fixed for the whole artifact. SimdConfig::simd128_enabled() mirrors this directly (cfg!(target_feature = "simd128")) rather than reading a field, so it is not something a config value can override -- a force-scalar-style SimdConfig cannot re-route a simd128-compiled wasm build back to the scalar kernels. To get scalar dispatch on wasm32, build without the simd128 target feature; there is no runtime override for a build that has it.

Performance (384-dim vectors)

Measured by crates/embed/benches/simd_bench.rs; run cargo bench -p lattice-embed to reproduce on your own hardware — these are illustrative, not a portability guarantee.

Usage

use lattice_embed::utils::{cosine_similarity, dot_product, normalize, euclidean_distance};

let a = vec![1.0, 0.0, 0.0];
let b = vec![1.0, 0.0, 0.0];

let sim = cosine_similarity(&a, &b);  // 1.0 (identical direction)
let dot = dot_product(&a, &b);        // 1.0
let dist = euclidean_distance(&a, &b); // 0.0

let mut v = vec![3.0, 4.0];
normalize(&mut v); // v = [0.6, 0.8], magnitude = 1.0

Int8 Quantization

4x memory reduction with ~99% accuracy:

use lattice_embed::simd::QuantizedVector;

let v = vec![0.1, 0.2, 0.3, /* ... */];
let q = QuantizedVector::from_f32(&v);

// Compare quantized vectors directly
let sim = q.cosine_similarity(&other_quantized);

Cache

SHA-256-based hashing with LRU eviction. Default capacity: 4000 entries (~6MB for 384-dim vectors).

use lattice_embed::{EmbeddingCache, EmbeddingModel, EmbeddingRole, ModelConfig};

let cache = EmbeddingCache::new(1000); // custom capacity; use `with_default_capacity()` for 4000

let key = cache.compute_key(
    "text",
    ModelConfig::new(EmbeddingModel::BgeSmallEnV15),
    EmbeddingRole::Generic,
);
cache.put(key, vec![0.1, 0.2, 0.3]);

let stats = cache.stats();
println!("Hit rate: {:.1}%", stats.hit_rate() * 100.0);

Feature Flags

[dependencies]
lattice-embed = { version = "0.4", default-features = false }  # SIMD only
lattice-embed = { version = "0.4" }  # Full (native + SIMD)

Batch Processing

let texts = vec![
    "First document".to_string(),
    "Second document".to_string(),
    "Third document".to_string(),
];

let embeddings = service.embed(&texts, EmbeddingModel::BgeSmallEnV15).await?;
assert_eq!(embeddings.len(), 3);

Maximum batch size: 1000 texts (to prevent OOM).

API Reference

EmbeddingService Trait

#[async_trait]
pub trait EmbeddingService: Send + Sync {
    async fn embed(&self, texts: &[String], model: EmbeddingModel) -> Result<Vec<Vec<f32>>>;
    async fn embed_one(&self, text: &str, model: EmbeddingModel) -> Result<Vec<f32>>;
    fn supports_model(&self, model: EmbeddingModel) -> bool;
    fn name(&self) -> &'static str;
}

SIMD Config

use lattice_embed::simd_config;

let config = simd_config();
println!("AVX2: {}, NEON: {}", config.avx2_enabled, config.neon_enabled);