Expand description
An SGLang-inspired LLM serving runtime, implemented from scratch in Rust.
This crate distills the architecture of SGLang (sgl-project/sglang) into a compact, dependency-free reference implementation:
engine— offline batch engine + continuous-batching schedulerradix_cache— RadixAttention-style prefix cache (compute-skip)kv_cache— paged KV cache with reference counting and copy-on-writesampler— temperature / top-k / top-p / penalty samplinggrammar— regex / JSON constrained-decoding logit maskdsl— an SGLang-style frontend language (gen/select/fork)model— model backend trait + deterministic n-gram toy LMtokenizer— tokenizer trait + deterministic byte-level tokenizerserver— minimal dependency-free HTTP serving frontend
The implementation deliberately keeps external dependencies to zero so the
whole runtime compiles and runs entirely offline. See docs/DESIGN.md for
the architecture, the mapping to SGLang, and the parity roadmap.
Re-exports§
pub use apc::ApcCache;pub use dsl::Op;pub use dsl::Program;pub use dsl::StreamResult;pub use engine::CacheBackend;pub use engine::CheckpointId;pub use engine::Engine;pub use engine::EngineConfig;pub use engine::EngineStats;pub use engine::SchedulePolicy;pub use engine::ToyEngine;pub use grammar::mask_logits;pub use grammar::ByteRegex;pub use grammar::Grammar;pub use kv_cache::BlockId;pub use kv_cache::PagedKvCache;pub use model::ModelEngine;pub use model::ModelSession;pub use model::NGramModel;pub use radix_cache::RadixCache;pub use sampler::Sampler;pub use spec::FinishReason;pub use spec::GenerationOutput;pub use spec::RejectReason;pub use spec::SamplingParams;pub use spec::TokenId;pub use spec::WriteRequest;pub use tokenizer::ByteTokenizer;pub use tokenizer::Tokenizer;
Modules§
- apc
- Block-level chained-hash prefix cache, in the style of vLLM’s Automatic Prefix Caching (APC).
- dsl
- An SGLang-style frontend language, expressed as an embeddable builder API.
- engine
- Offline batch engine plus the continuous-batching scheduler.
- grammar
- Dependency-free constrained decoding.
- http_
client - A minimal, dependency-free blocking HTTP/1.1 client.
- json
- A minimal JSON value, parser, and serializer. Self-contained (no
serde) because the runtime must compile offline. - kv_
cache - Paged KV cache with reference counting and copy-on-write.
- model
- Model backend abstraction plus a deterministic n-gram toy language model.
- radix_
cache - Radix-tree prefix cache — the analogue of SGLang’s
RadixAttention. - rng
- A tiny deterministic PRNG (SplitMix64) so sampling is reproducible with no external dependencies.
- sampler
- Logit post-processing and categorical sampling.
- server
- A minimal, dependency-free HTTP serving frontend.
- spec
- Public data model shared across the runtime and the SGLang-style frontend.
- tokenizer
- Tokenizer abstraction plus a deterministic byte-level reference tokenizer.
- triage
- Laya triage: a System-1 gate in front of generation (System 2).