inferencelayer

Kortexya's engine-native inference layer: LLM generation and the embedding/encoder family on wgpu (hand-written WGSL kernels, any adapter — Metal / Vulkan / DX12 / GL) with a pure-Rust CPU fallback (gemm + rayon — no torch, no candle, no onnxruntime).

What it runs today:

Capability	Notes
LFM2 / Qwen3 generation	quantized weights (turboquant), grammar-constrained decoding, speculative/MTP paths
Dense embeddings	BERT, XLM-RoBERTa (BioLORD…), Qwen3-embed — mean/CLS/last-token pooling, always L2-normalized
Late interaction	ColBERT per-token embeddings
CrossEncoder scoring	`*ForSequenceClassification` heads (pair inputs + segment ids → raw relevance logit)
Vision-text	SigLIP
Sharded fleet serving	pipeline sharding over TCP/WebSocket/WebRTC (browser peers via wasm), `lfm2-serve` HTTP layer

Parity discipline: every architecture lands with a gate against its reference implementation (sentence-transformers / HF / candle) — see tests/ (pinned goldens, GPU-vs-CPU kernel parity) and the docs below.

Using it

[dependencies]
inferencelayer = { git = "ssh://git@gitlab.com/kortexya/inferencelayer.git", default-features = false, features = ["encoder-cpu", "cli"] }

For local development against a checkout, patch the git source in your project's .cargo/config.toml (git-ignored) instead of editing Cargo.toml:

[patch."ssh://git@gitlab.com/kortexya/inferencelayer.git"]
inferencelayer = { path = "../inferencelayer" }

Embedding quickstart (the path the OSFKB services use):

use inferencelayer::EmbedEngine;
use inferencelayer::encoder_weights::EncBatch;
use inferencelayer::pooling::EmbedOut;

let mut engine = EmbedEngine::auto(&checkpoint_dir, 8192)?;   // GPU if present, else CPU
let batch = EncBatch::from_seqs(token_id_seqs);
let EmbedOut::Pooled(vectors) = engine.encode(&batch)? else { unreachable!() };

Feature flags

Feature	Default	What it gates
`net`	✅	the pipeline-sharding fleet transport (`shard`, `shard_serve`)
`cli`	✅	the `tokenizers` dependency for the CLI binaries
`encoder-cpu`	✅	the CPU encoder runtime (no-GPU fallback) + the GPU parity oracle
`server`	—	the axum/tokio HTTP serving layer (`lfm2-serve`)
`ws`	—	WebSocket fleet transport
`webrtc`	—	native WebRTC data-channel transport (large async stack)

A wasm build (--no-default-features) leaves only the wgpu compute core — the engine core never depends on tokenizers (C onig_sys) or std::net.

Consumer tip: encoder-only builds (--no-default-features --features encoder-cpu,cli) skip the serving/fleet stack entirely and compile much faster.

Binaries

lfm2-serve (HTTP serving, --features server), lfm2-shard-worker / lfm2-shard-run / lfm2-profile-stage (fleet), lfm2-generate (CLI generation), kernel-lab, scoreboard (the perf ledger). Binary names keep the lfm2- prefix from the crate's origin.

Docs

ARCHITECTURE.md — engine + fleet architecture
SERVING.md — the HTTP serving layer
WEBRTC_TRANSPORT.md, WEBRTC_RESUME.md — the P2P transport
BROWSER_35B_DEMO.md — the browser-worker wasm path

Tests

cargo test                        # default features; model-gated suites skip without their env flags
RUN_CE_PARITY=1 cargo test --test cross_encoder_parity -- --nocapture   # needs the HF cache

Some parity suites need real checkpoints in the local HuggingFace cache and are env-gated (RUN_CE_PARITY=1, …); they skip cleanly when unset. The shard tests need the net feature — run the full suite with default features.

Provenance & consumers

Extracted 2026-07-12 from osfkb crates/lfm2-wgpu via git subtree split — the 64-commit crate history is preserved; the crate was renamed lfm2-wgpu → inferencelayer (the scope outgrew LFM2). This repo is the single source of truth for the engine — osfkb's copy was removed the same day (its in-flight delta was ported here first and re-verified).

Consumers, all on the git dependency:

reasoninglayer-service/services/embedding-service-rs — inferencelayer = { git = "…" }.
osfkb (osfkb-application, osfkb-infrastructure, apps/lfm2-browser-worker) — via lfm2-wgpu = { package = "inferencelayer", git = "…" }, so its use lfm2_wgpu:: imports are unchanged.

inferencelayer 0.2.5