# inferencelayer
Kortexya's **engine-native inference layer**: LLM generation and the embedding/encoder family on
**wgpu** (hand-written WGSL kernels, any adapter — Metal / Vulkan / DX12 / GL) with a
**pure-Rust CPU fallback** (`gemm` + rayon — no torch, no candle, no onnxruntime).
What it runs today:
| **LFM2 / Qwen3 generation** | quantized weights (turboquant), grammar-constrained decoding, speculative/MTP paths |
| **Dense embeddings** | BERT, XLM-RoBERTa (BioLORD…), Qwen3-embed — mean/CLS/last-token pooling, always L2-normalized |
| **Late interaction** | ColBERT per-token embeddings |
| **CrossEncoder scoring** | `*ForSequenceClassification` heads (pair inputs + segment ids → raw relevance logit) |
| **Vision-text** | SigLIP |
| **Sharded fleet serving** | pipeline sharding over TCP/WebSocket/WebRTC (browser peers via wasm), `lfm2-serve` HTTP layer |
Parity discipline: every architecture lands with a gate against its reference implementation
(sentence-transformers / HF / candle) — see `tests/` (pinned goldens, GPU-vs-CPU kernel parity)
and the docs below.
## Using it
```toml
[dependencies]
inferencelayer = { git = "ssh://git@gitlab.com/kortexya/inferencelayer.git", default-features = false, features = ["encoder-cpu", "cli"] }
```
For local development against a checkout, patch the git source in your project's
`.cargo/config.toml` (git-ignored) instead of editing `Cargo.toml`:
```toml
[patch."ssh://git@gitlab.com/kortexya/inferencelayer.git"]
inferencelayer = { path = "../inferencelayer" }
```
Embedding quickstart (the path the OSFKB services use):
```rust
use inferencelayer::EmbedEngine;
use inferencelayer::encoder_weights::EncBatch;
use inferencelayer::pooling::EmbedOut;
let mut engine = EmbedEngine::auto(&checkpoint_dir, 8192)?; // GPU if present, else CPU
let batch = EncBatch::from_seqs(token_id_seqs);
let EmbedOut::Pooled(vectors) = engine.encode(&batch)? else { unreachable!() };
```
## Feature flags
| `net` | ✅ | the pipeline-sharding fleet transport (`shard`, `shard_serve`) |
| `cli` | ✅ | the `tokenizers` dependency for the CLI binaries |
| `encoder-cpu` | ✅ | the CPU encoder runtime (no-GPU fallback) + the GPU parity oracle |
| `server` | — | the axum/tokio HTTP serving layer (`lfm2-serve`) |
| `ws` | — | WebSocket fleet transport |
| `webrtc` | — | native WebRTC data-channel transport (large async stack) |
A wasm build (`--no-default-features`) leaves only the wgpu compute core — the engine core never
depends on `tokenizers` (C `onig_sys`) or std::net.
Consumer tip: encoder-only builds (`--no-default-features --features encoder-cpu,cli`) skip the
serving/fleet stack entirely and compile much faster.
## Binaries
`lfm2-serve` (HTTP serving, `--features server`), `lfm2-shard-worker` / `lfm2-shard-run` /
`lfm2-profile-stage` (fleet), `lfm2-generate` (CLI generation), `kernel-lab`, `scoreboard`
(the perf ledger). Binary names keep the `lfm2-` prefix from the crate's origin.
## Docs
- `ARCHITECTURE.md` — engine + fleet architecture
- `SERVING.md` — the HTTP serving layer
- `WEBRTC_TRANSPORT.md`, `WEBRTC_RESUME.md` — the P2P transport
- `BROWSER_35B_DEMO.md` — the browser-worker wasm path
## Tests
```bash
cargo test # default features; model-gated suites skip without their env flags
RUN_CE_PARITY=1 cargo test --test cross_encoder_parity -- --nocapture # needs the HF cache
```
Some parity suites need real checkpoints in the local HuggingFace cache and are env-gated
(`RUN_CE_PARITY=1`, …); they skip cleanly when unset. The shard tests need the `net` feature —
run the full suite with default features.
## Provenance & consumers
Extracted 2026-07-12 from `osfkb` `crates/lfm2-wgpu` via `git subtree split` — the 64-commit
crate history is preserved; the crate was renamed `lfm2-wgpu` → `inferencelayer` (the scope
outgrew LFM2). **This repo is the single source of truth for the engine** — osfkb's copy was
removed the same day (its in-flight delta was ported here first and re-verified).
Consumers, all on the git dependency:
- `reasoninglayer-service/services/embedding-service-rs` — `inferencelayer = { git = "…" }`.
- `osfkb` (`osfkb-application`, `osfkb-infrastructure`, `apps/lfm2-browser-worker`) — via
`lfm2-wgpu = { package = "inferencelayer", git = "…" }`, so its `use lfm2_wgpu::` imports are
unchanged.
## License
Proprietary — © Kortexya. Not for publication to crates.io.